This is gold for anyone building AI agents.
Agents call the model again and again: plan, write code, check, fix, repeat. Every one of those calls is a wait.
EAGLE-4 speeds up the model itself instead of changing your app:
- A small draft model guesses several words ahead
- The big model checks them all in one pass and keeps what's right
- The output stays the same as the big model on its own, it just arrives faster
EAGLE-4 adjusts how far ahead it guesses based on your hardware and how many requests you're handling.
That matters for agents running at scale. In the paper's test with 512 requests at once, it reported 1.70x the throughput of DFlash 2 and 1.99x DSpark (Qwen 3.8 27B, 4x NVIDIA B300).
It plugs into SGLang, vLLM and TokenSpeed, and the code is on GitHub so you can benchmark it on your own setup.
Save this if you build agents.
Agents call the model again and again: plan, write code, check, fix, repeat. Every one of those calls is a wait.
EAGLE-4 speeds up the model itself instead of changing your app:
- A small draft model guesses several words ahead
- The big model checks them all in one pass and keeps what's right
- The output stays the same as the big model on its own, it just arrives faster
EAGLE-4 adjusts how far ahead it guesses based on your hardware and how many requests you're handling.
That matters for agents running at scale. In the paper's test with 512 requests at once, it reported 1.70x the throughput of DFlash 2 and 1.99x DSpark (Qwen 3.8 27B, 4x NVIDIA B300).
It plugs into SGLang, vLLM and TokenSpeed, and the code is on GitHub so you can benchmark it on your own setup.
Save this if you build agents.
EAGLE-4 is here.
From EAGLE-1 to EAGLE-3, we’ve kept making inference faster.
EAGLE-4 is our fastest yet.
961.6 tokens/s on @Alibaba_Qwen’s Qwen3.8-27B with TokenSpeed (BF16, 4× @nvidia B300, TP4, one request, T=0).
Github: github.com/eagle-hq/EAGLE…
Oct 20, 2026 · 11:02 PM UTC
23
3
45
29,750
@dr_cintas The 512-concurrent-request result is the interesting part, since spec decoding gains usually shrink as batches grow. I'd want to see acceptance rates on real agent traces, where a lot of the output is tool-call JSON.
0
0
0
10
@dr_cintas worth adding for anyone self-hosting: those numbers are from the paper's setup (512 concurrent requests, 4x B300). your speedup depends on how often the draft gets accepted on your own traffic.
replay a sample of real agent requests with it on and off before switching
replay a sample of real agent requests with it on and off before switching
0
0
0
18
@dr_cintas Tokenspeed support is what got me, anyone already tried it with vLLM?
0
0
0
10
@dr_cintas LMAO the 'every one of those calls is a wait' line is so true, I've been staring at my terminal for minutes just waiting on plan-check-fix loops
0
0
0
9
@dr_cintas the draft-model trick is underrated, cutting agent latency at the model level instead of rewriting your loop is huge
0
0
0
11
@dr_cintas Wait so the output matches the big model exactly? That's the part that sells it for me, how much latency does it really shave off on a single request though
0
0
0
7
@dr_cintas will run it on my own setup before i believe any paper number
0
0
1
35
@dr_cintas Speed-ups that don't touch your app code are always the easiest win, I'd test this on my setup before the weekend
0
0
0
9
@dr_cintas Saving this. Been waiting for something like this for my agent setup
0
0
0
7
@dr_cintas Did they test it on smaller models too or just the 27B
0
0
0
6
@dr_cintas yo this is actually wild, the speedup on agent loops would be huge
0
0
0
23
