Pinned Tweet
My talk on accelerated diffusion and flow models at @BerkeleyLab's DL4Sci School is online!
Check it out for an overview on how to:
- Reduce the number of denoising steps
- Accelerate each step
📺 youtube.com/watch?v=ZnRzw9… (jump to 22:18 to learn more about the video below)
Check it out for an overview on how to:
- Reduce the number of denoising steps
- Accelerate each step
📺 youtube.com/watch?v=ZnRzw9… (jump to 22:18 to learn more about the video below)
3
26
186
9,197
Julius Berner retweeted
(1/7) FastVideo FastH3 Omni goes local on DGX Spark, RTX GPUs, and Apple MLX 🚀
- FastH3 Omni, a 5 s 480p clip from a photo and a prompt takes ~3 min. On two Sparks, ~2 min
- FastH3 T2AV is 52.6 s on one Spark, 30.7 s on two Sparks
Blog: haoailab.com/blogs/fasth3-o…
Cookbook: haoailab.com/FastVideo/cook…
HF 🤗 huggingface.co/collections/Fa…
@NVIDIARTXSpark @NVIDIAAI @MiniMax_AI
Every clip in this video was generated on NVIDIA DGX Spark. 8-step NVFP4 checkpoint.
- FastH3 Omni, a 5 s 480p clip from a photo and a prompt takes ~3 min. On two Sparks, ~2 min
- FastH3 T2AV is 52.6 s on one Spark, 30.7 s on two Sparks
Blog: haoailab.com/blogs/fasth3-o…
Cookbook: haoailab.com/FastVideo/cook…
HF 🤗 huggingface.co/collections/Fa…
@NVIDIARTXSpark @NVIDIAAI @MiniMax_AI
Every clip in this video was generated on NVIDIA DGX Spark. 8-step NVFP4 checkpoint.
7
20
87
10,184
Julius Berner retweeted
MiniMax-H3 🙂 PTD Ref2VA Acceleration LoRAs
-an 8-step and a 4-step accelerator for the Ref2VA
-applied PTD (Parametric Trajectory Distillation) to MiniMax-H3, distilling the Ref2VA (reference-image → video + audio) trajectory into a few-steps
huggingface.co/taobao-content…
-an 8-step and a 4-step accelerator for the Ref2VA
-applied PTD (Parametric Trajectory Distillation) to MiniMax-H3, distilling the Ref2VA (reference-image → video + audio) trajectory into a few-steps
huggingface.co/taobao-content…
2
15
139
9,009
Cool work and very close to our concurrent Parametric Trajectory Distillation (PTD): alan-lanfeng.github.io/PTD/
Some connections to previous works:
- Parallel Decoding Distillation (PDD): while PDD uses a picewise-linear parametrization of the student trajectory (fused during inference), SMD & PTD propose a basis expansion with learned coefficients (discarded at inference).
- Lagrangian Flow-Maps: PDD/PTD/SMD can be viewed as special cases, where the JVP can be computed in closed form, since the terminal time enters only through the student head.
Details on these connections in our papers:
- PTD: arxiv.org/abs/2610.11498
- PDD: arxiv.org/abs/2607.26004
Some connections to previous works:
- Parallel Decoding Distillation (PDD): while PDD uses a picewise-linear parametrization of the student trajectory (fused during inference), SMD & PTD propose a basis expansion with learned coefficients (discarded at inference).
- Lagrangian Flow-Maps: PDD/PTD/SMD can be viewed as special cases, where the JVP can be computed in closed form, since the terminal time enters only through the student head.
Details on these connections in our papers:
- PTD: arxiv.org/abs/2610.11498
- PDD: arxiv.org/abs/2607.26004
Excited to introduce Simplified Matching Distillation (SMD) for few-step diffusion🧵
No auxiliary score model. No adversarial training. No generator modification.
You will get:
• Photorealistic appearance
• High-quality dynamics
• Memory-efficient training
• Mode covering MD works well across images, video, joint audio–video, and 3D gen, validated across model scales up to 33B parameters.
Besides, SMD supports:
- Multi-teacher capability merging (like OPD in LLM)
- Cross-model capability transfer
- Bidirectional teacher to causal distillation
Full release. Paper, code, model and data.
Project Page: zjwfufu.github.io/SMD-Page/
Paper: arxiv.org/abs/2610.18472
Code: github.com/zjwfufu/SMD
Model Card & Data: huggingface.co/zjwfufu/SMD
Demo Video 👇
0
2
36
2,510
Parallel Decoding Distillation (PDD) is taking off 🛫
We’ve only released PDD end of July and it’s already adopted in:
- Alibaba’s MiniMax H3: huggingface.co/alibaba-pai/Mi…
- WorldPlay2’s causal distillation: worldplay2.github.io
- FastVideo’s FastH3 Omni: haoailab.com/blogs/fastvide…
- NVIDIA’s large language continuous diffusion model: arxiv.org/pdf/2610.02665
- Amazon’s generative forecasting model: arxiv.org/abs/2610.22358
- NVIDIA’s & MSR’s extensions to parametric student trajectories: alan-lanfeng.github.io/PTD & arxiv.org/pdf/2610.18472
We’ll be open-sourcing our implementation soon - stay tuned!
We’ve only released PDD end of July and it’s already adopted in:
- Alibaba’s MiniMax H3: huggingface.co/alibaba-pai/Mi…
- WorldPlay2’s causal distillation: worldplay2.github.io
- FastVideo’s FastH3 Omni: haoailab.com/blogs/fastvide…
- NVIDIA’s large language continuous diffusion model: arxiv.org/pdf/2610.02665
- Amazon’s generative forecasting model: arxiv.org/abs/2610.22358
- NVIDIA’s & MSR’s extensions to parametric student trajectories: alan-lanfeng.github.io/PTD & arxiv.org/pdf/2610.18472
We’ll be open-sourcing our implementation soon - stay tuned!
Having worked with many distillation methods for accelerating diffusion & flow models, I’m still blown away by the simplicity of Parallel Decoding Distillation (PDD).
1️⃣ State-of-the-art quality and diversity with variable NFE
2️⃣ First to scale trajectory-distillation to few-PDD is about as cheap and simple as it gets:
🎯single, regression-based MSE loss using 1 student & 1 detached teacher forward
❌NO data required
❌NO adversarial loss or alternating training
❌NO JVP or finite-difference approx.
❌NO additional time-embedding or trainable models (e.g. fake score, discriminator)
❌NO stabilization tricks (e.g. normalization, clipping, warm-up)
2
11
107
5,605
Julius Berner retweeted
Avengers assemble: 1 year BakerLab + NVIDIA GenAIR = 🔥 rfd4-proteina.org
Presenting RFD4-Proteina, our latest protein design model, significantly better and faster than everything we compared against, both in silico and in the lab! 1/30
Presenting RFD4-Proteina, our latest protein design model, significantly better and faster than everything we compared against, both in silico and in the lab! 1/30
27
201
657
63,005
Julius Berner retweeted
🚀 Huge congrats to Hao AI Lab @ UCSD on FastH3 Omni!
Achieving up to 12× lower latency on NVIDIA Blackwell using NVFP4 QAD and FastGen PDD + DMD.
Achieving up to 12× lower latency on NVIDIA Blackwell using NVFP4 QAD and FastGen PDD + DMD.
As promised, FastVideo open-sources @MiniMax_AI FastH3 Omni, training code included! 🚀
- Real-time gen @ 12x faster on NVIDIA Blackwell
- NVFP4/FP8 for Spark + RTX!
- ComfyUI day-0
Collab with @NVIDIAAI , Nuva Lab @nuvalab and Reactor @reactorworld
Open source must win! ♥️
0
3
26
1,882
Julius Berner retweeted
The latest result of our amazing collaboration between FastVideo and FastGen: FastH3 Omni
- 12x faster reference-to-video (image, video, & voice references)
- distilled using our new combination of PDD & DMD
Blog: haoailab.com/blogs/fastvide…
PDD: x.com/julberner/stat…
0
1
4
459
The latest result of our amazing collaboration between FastVideo and FastGen: FastH3 Omni
- 12x faster reference-to-video (image, video, & voice references)
- distilled using our new combination of PDD & DMD
Blog: haoailab.com/blogs/fastvide…
PDD: x.com/julberner/stat…
- 12x faster reference-to-video (image, video, & voice references)
- distilled using our new combination of PDD & DMD
Blog: haoailab.com/blogs/fastvide…
PDD: x.com/julberner/stat…
As promised, FastVideo open-sources @MiniMax_AI FastH3 Omni, training code included! 🚀
- Real-time gen @ 12x faster on NVIDIA Blackwell
- NVFP4/FP8 for Spark + RTX!
- ComfyUI day-0
Collab with @NVIDIAAI , Nuva Lab @nuvalab and Reactor @reactorworld
Open source must win! ♥️
2
4
19
1,711
Julius Berner retweeted
4/7 How: PDD + DMD distillation.
PDD teaches the model to cover several denoising steps per forward pass.
DMD pulls the finished output back toward the base model's quality.
Training code and reference-aware preprocessing are open. Big shoutout to @NVIDIAAI FastGen team!
PDD teaches the model to cover several denoising steps per forward pass.
DMD pulls the finished output back toward the base model's quality.
Training code and reference-aware preprocessing are open. Big shoutout to @NVIDIAAI FastGen team!
1
1
10
4,775
Julius Berner retweeted
As promised, FastVideo open-sources @MiniMax_AI FastH3 Omni, training code included! 🚀
- Real-time gen @ 12x faster on NVIDIA Blackwell
- NVFP4/FP8 for Spark + RTX!
- ComfyUI day-0
Collab with @NVIDIAAI , Nuva Lab @nuvalab and Reactor @reactorworld
Open source must win! ♥️
- Real-time gen @ 12x faster on NVIDIA Blackwell
- NVFP4/FP8 for Spark + RTX!
- ComfyUI day-0
Collab with @NVIDIAAI , Nuva Lab @nuvalab and Reactor @reactorworld
Open source must win! ♥️
35
93
834
143,169
Check out RVM: simple, effective RL fine-tuning for diffusion models ⚡
Roll out (no grads) → compute rewards → re-noise → reward-weighted velocity regression + anchoring
Unifies previous approaches and scales to video models with SOTA performance!
Roll out (no grads) → compute rewards → re-noise → reward-weighted velocity regression + anchoring
Unifies previous approaches and scales to video models with SOTA performance!
Simple, efficient, scalable diffusion RL without likelihood ratios?
Meet RVM: Reward-based Velocity Matching
🎯 Rewards directly in velocity space
⚡ Cheap video fine-tuning
🧩 Unifies recent objectives
🎬 Fixes the static-video problem by dynamic motion reward
How it works 🧵
1
5
25
1,416
Julius Berner retweeted
Exciting to see the community building on FastGen and pushing distillation techniques forward! 🚀
This cool new work builds on ideas from our PDD and PiFlow, exploring novel parameterizations of student heads. Great to see these ideas evolving!
This cool new work builds on ideas from our PDD and PiFlow, exploring novel parameterizations of student heads. Great to see these ideas evolving!
Excited to introduce Simplified Matching Distillation (SMD) for few-step diffusion🧵
No auxiliary score model. No adversarial training. No generator modification.
You will get:
• Photorealistic appearance
• High-quality dynamics
• Memory-efficient training
• Mode covering MD works well across images, video, joint audio–video, and 3D gen, validated across model scales up to 33B parameters.
Besides, SMD supports:
- Multi-teacher capability merging (like OPD in LLM)
- Cross-model capability transfer
- Bidirectional teacher to causal distillation
Full release. Paper, code, model and data.
Project Page: zjwfufu.github.io/SMD-Page/
Paper: arxiv.org/abs/2610.18472
Code: github.com/zjwfufu/SMD
Model Card & Data: huggingface.co/zjwfufu/SMD
Demo Video 👇
0
8
61
4,099
Julius Berner retweeted
(1/6) The all-new FastVideo FastH3-QAD-v2 and the all-new H3 NVFP4 VAE decoders are here! 🚀🚀
This entire 768p video was generated on two DGX Sparks, using QAD-v2 and both new decoders. 8-step video + audio, QAD weights 💾
@NVIDIARTXSpark @NVIDIAAI @MiniMax_AI
This entire 768p video was generated on two DGX Sparks, using QAD-v2 and both new decoders. 8-step video + audio, QAD weights 💾
@NVIDIARTXSpark @NVIDIAAI @MiniMax_AI
5
17
103
18,434
Julius Berner retweeted
▼概要
Qwen-Image-2.1向けの高速化LoRA「Qwen-Image-2.1-Fun-Acc-LoRAs」が公開。Parallel Decoding Distillation（PDD）の適用により、わずか4ステップでの画像生成と画像編集を実現。
▼ポイント
・Qwen-Image-2.1をベースとした4ステップ（4 NFE）生成用の公式LoRA
機能に対応
・40ステップのTeacherモデルや他手法との生成結果の比較デモを提示
▼使い所
Qwen-Image-2.1での画像生成および編集処理を、少ない計算量で効率化・高速化する環境向け。
#QwenImage21 #LoRA
URLはリプ⬇️
Qwen-Image-2.1向けの高速化LoRA「Qwen-Image-2.1-Fun-Acc-LoRAs」が公開。Parallel Decoding Distillation（PDD）の適用により、わずか4ステップでの画像生成と画像編集を実現。
▼ポイント
・Qwen-Image-2.1をベースとした4ステップ（4 NFE）生成用の公式LoRA
機能に対応
・40ステップのTeacherモデルや他手法との生成結果の比較デモを提示
▼使い所
Qwen-Image-2.1での画像生成および編集処理を、少ない計算量で効率化・高速化する環境向け。
#QwenImage21 #LoRA
URLはリプ⬇️
1
5
89
6,971
Julius Berner retweeted
1/ RVM is a new RL technique that does reward-weighted velocity matching without estimating likelihood ratios. They test it on SD3.5, Wan and SkyReels.
Quite similar to diffusionNFT (in fact they show nft is special case of rvm)
Paper: arxiv.org/abs/2608.23664
Quite similar to diffusionNFT (in fact they show nft is special case of rvm)
Paper: arxiv.org/abs/2608.23664
3
11
82
4,039
Julius Berner retweeted
We are right in the middle of this craze for continuous diffusion language models and Sigma is our latest work to date:
- 3B/8B embedding-based continuous diffusion LM
- Strong reasoning
- Initial distillation results, with much more to show for soon!
Releases soon
- 3B/8B embedding-based continuous diffusion LM
- Strong reasoning
- Initial distillation results, with much more to show for soon!
Releases soon
💫 Large Language Continuous Diffusion Models
Presenting Sigma, the first large-scale (3B/8B) continuous diffusion LM (dLM) trained with likelihood.
📈 Comparable with discrete dLMs and AR on math and coding
🎛️ Enables meaningful study of inference-time interventions
Jointly uo01), Jean-Marie Lemercier (@jm_lemercier), Simon Welker (@_cobalamin)
Team: @YongganFu @mmkamani7 Sajad Norouzi @julberner @tomasgeffner @karsten_kreis @YongxinChen1 @MoleiTaoMath @jwthickstun @PavloMolchanov Ante Jukic @ArashVahdat @MardaniMorteza
0
2
12
1,106
Julius Berner retweeted
Diffusion model is not a native likelihood model, why RL finetune it using likelihood based approach from LLMs? We present RVM, a simple recipe to finetune the velocity/score directly without turning to likelihood at all. It is fast, scalable, and stable. Checkout the paper 🔥
Simple, efficient, scalable diffusion RL without likelihood ratios?
Meet RVM: Reward-based Velocity Matching
🎯 Rewards directly in velocity space
⚡ Cheap video fine-tuning
🧩 Unifies recent objectives
🎬 Fixes the static-video problem by dynamic motion reward
How it works 🧵
1
6
70
3,959
Julius Berner retweeted
Excited to share our latest RL method for diffusion models!
The cord idea behind this simple and effective object: collect final samples from PFODE rollout and their rewards, then compute an advantage-weighted pretraining loss to reinforce/forget good/bad generated samples
The cord idea behind this simple and effective object: collect final samples from PFODE rollout and their rewards, then compute an advantage-weighted pretraining loss to reinforce/forget good/bad generated samples
Simple, efficient, scalable diffusion RL without likelihood ratios?
Meet RVM: Reward-based Velocity Matching
🎯 Rewards directly in velocity space
⚡ Cheap video fine-tuning
🧩 Unifies recent objectives
🎬 Fixes the static-video problem by dynamic motion reward
How it works 🧵
0
6
51
2,771
Julius Berner retweeted
Latest result in fine tuning of diffusion model, general method, but demonstrated highly effective even for video generations.
Simple, efficient, scalable diffusion RL without likelihood ratios?
Meet RVM: Reward-based Velocity Matching
🎯 Rewards directly in velocity space
⚡ Cheap video fine-tuning
🧩 Unifies recent objectives
🎬 Fixes the static-video problem by dynamic motion reward
How it works 🧵
0
2
32
2,182
Julius Berner retweeted
Simple, efficient, scalable diffusion RL without likelihood ratios?
Meet RVM: Reward-based Velocity Matching
🎯 Rewards directly in velocity space
⚡ Cheap video fine-tuning
🧩 Unifies recent objectives
🎬 Fixes the static-video problem by dynamic motion reward
How it works 🧵
Meet RVM: Reward-based Velocity Matching
🎯 Rewards directly in velocity space
⚡ Cheap video fine-tuning
🧩 Unifies recent objectives
🎬 Fixes the static-video problem by dynamic motion reward
How it works 🧵
1
18
107
13,723