Edward Z. Yang retweeted
Your second problem is equally valid, but we can make a lot of progress on the first problem right now. It's what I've been working on. You *don't* need to be able to read the formal language.
blog.brownplt.org/2026/06/09/pic…
blog.brownplt.org/2026/06/09/pic…
0
1
4
222
Deciding how to help non-expert humans engage with formal verification specifications seems like one of the more important non-verifiable UX problems facing the rise of formal verification. 🧵
20
25
266
31,710
If you have a big pile of vibe coded Python scripts that elaborate a trace of your Python program into Lean, how much should you actually trust this TCB? And if you don't trust this, are you going to write a formal semantics of Python? Will you trust that? (cough Starlark)
3
1
8
1,096
It is not the end of the world if FV is only used for simple verification conditions like "no crashes". But the most interesting theorems are about humans trusting software, and here we essentially have a problem of alignment: the human cannot be removed from this loop. FIN.
1
2
19
1,373
Edward Z. Yang retweeted
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in nd the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in nd the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
95
142
957
162,495
Earlier this year I had a kick where I was like "what if we FLEX EVERYTHING". This was mostly not actually a good idea but Driss made flex gemm actually work! Go check it out.
Gave a talk at PTC on flex_gemm which is a new gemm_epilogue API in pytorch built on Quack. I think its neat, not ground breaking but neat nonetheless :)
github.com/drisspg/transf…
planning to bootstrap this and continue to add fun examples
4
0
25
3,136
Last year we were extremely worried about how to get cudagraphable, dropless mixture of experts. Dist_moe has essentially solved this for us, both internally and in OSS
We’re sharing Dist-MoE, open-source library from Meta SuperIntelligence Labs (MSL @AIatMeta ). Dist-MoE provides fused communication and compute kernels for BF16/MXFP8 training and inference, plus NVFP4 inference.
✅ Fused communication and compute kernels
✅ Full CUDA graph ation packing across layers into a bounded buffer
✅ Dynamic, capacity-driven recompute
✅ Forward and backward kernels for BF16/MXFP8
✅ No shared-expert branch required
Some highlights from our GB300 NVL72 benchmarks:
• Up to 1.40× speedup over Mixture-of-Kittens (MoK) in our imbalanced, ten-layer MXFP8 training workload across 64 GPUs. Dist-MoE packed activation state across all ten layers with zero capacity-induced recompute.
• 3.09 useful PFLOP/s per GPU (3,090 TFLOP/s) for NVFP4 Mega inference across 16 GPUs.
• 1.86 useful PFLOP/s per GPU (1,856 TFLOP/s) for MXFP8 Staged training across 64 GPUs.
• 1.27 useful PFLOP/s per GPU (1,271 TFLOP/s) for BF16 training across 64 GPUs.
• Sub-linear scaling under imbalance: in NVFP4 inference across 16 GPUs, 8× the received rows on the busiest GPU required 4.82× the iteration time; about 40% lower iteration time per received row.
Full configurations, raw results, and measurement definitions are in the blog.
@levendlee and I also put together an interactive visual explainer covering token dispatch and combine, quantization, forward/backward kernels, activation packing, dynamic recompute, and memory planning.
Blog:
meta-pytorch.org/dist_moe/previ…
Repository:
github.com/meta-pytorch/d…
TorchTitan (@PyTorch ) integration:
github.com/pytorch/torcht…
If you work on MoE training, inference, or GPU kernels, take a look. We welcome feedback and contributions.
#PyTorch #MixtureOfExperts #MoE #OpenSource #TorchTitan #DistMoE
3
9
160
8,589
It is an incredibly pleasant feeling when Claude knows about variational types and will happily go implement them for you
2
0
12
2,209
Reminding myself that types are good even for things that are trivially testable cuz you don't have to remember to write those tests!
2
0
19
1,709
Opus 5.5 and Kokoro made a video for my latest (non AI-written) essay "Working the roofline for DeepSeek-V3 on Hopper". I'm really happy with how it turned out!
3
9
122
6,345
The actual essay, with a lot more details and all the formulas laid out for careful reading, can be found here! deepseek-v3.ezyang.com/studies/03-roo…
0
4
17
1,204