SGLang cuts time-to-first-token 37% versus vLLM when 80% of a prompt is a shared prefix: 310ms down to 195ms at 50 concurrent requests, on an 84% cache hit rate.
Raw throughput on unique prompts is close: 1,920 tok/s versus vLLM's 1,850, a 4% edge either way.
Push shared-prefix TTFT gap widens to 44%. If your traffic is mostly repeated system prompts, that's the number to plan around, not tok/s.
Raw throughput on unique prompts is close: 1,920 tok/s versus vLLM's 1,850, a 4% edge either way.
Push shared-prefix TTFT gap widens to 44%. If your traffic is mostly repeated system prompts, that's the number to plan around, not tok/s.
2
0
0
544
MLPerf Inference v6.0 results for GPT-OSS 120B, a 117B-parameter model with only 5.1B active per token, per GPU:
- B200: about 7,800 tok/s
- H200: about 3,100 tok/s
- MI355X: about 2,600 tok/s
- H100 (estimated): about 1,300 tok/s
That's B200 at roughly 2.5x H200's throughput 200 also carries more memory bandwidth, 8.0 TB/s and 192 GB versus H200's 4.8 TB/s and 141 GB.
- B200: about 7,800 tok/s
- H200: about 3,100 tok/s
- MI355X: about 2,600 tok/s
- H100 (estimated): about 1,300 tok/s
That's B200 at roughly 2.5x H200's throughput 200 also carries more memory bandwidth, 8.0 TB/s and 192 GB versus H200's 4.8 TB/s and 141 GB.
1
0
0
416
Standard decoding tops out around 1,200 tok/s. Add EAGLE-3 speculative decoding and it jumps to about 3,600. Switch to DFlash and it hits roughly 9,000.
At 0.8 acceptance rate with 5 speculative tokens, that's a 3.7x speedup, and first-token latency drops from 45ms to 8ms.
The depends on the draft model. Instruction-following drafts hit 0.6-0.75 acceptance, self-speculation drops to 0.4-0.6. Gains also fade above batch size 32.
Requires vLLM 0.8.0+ or SGLang 0.4.6.post4-cu124. Detail
At 0.8 acceptance rate with 5 speculative tokens, that's a 3.7x speedup, and first-token latency drops from 45ms to 8ms.
The depends on the draft model. Instruction-following drafts hit 0.6-0.75 acceptance, self-speculation drops to 0.4-0.6. Gains also fade above batch size 32.
Requires vLLM 0.8.0+ or SGLang 0.4.6.post4-cu124. Detail
2
0
1
550
At 80% shared prefix and 50 concurrent requests, SGLang's TTFT lands at 195ms against vLLM's 310ms, a 37% cut.
Push overlap to 95% and the gap widens: 93% cache hit rate, about 44% faster time to first token.
Unique-prompt throughput barely moves (1,920 vs 1,850 tok/s), so the p on prefix-heavy workloads like chat and RAG, not raw generation.
Numbers are from vLLM v0.18.0 and SGLang v0.5.9. Full benchmark
Push overlap to 95% and the gap widens: 93% cache hit rate, about 44% faster time to first token.
Unique-prompt throughput barely moves (1,920 vs 1,850 tok/s), so the p on prefix-heavy workloads like chat and RAG, not raw generation.
Numbers are from vLLM v0.18.0 and SGLang v0.5.9. Full benchmark
3
0
2
691
MI300X's software gaps get most of the attention. Its throughput doesn't need excuses.
At batch 1-4, H100 with TensorRT-LLM holds a 20-30% throughput edge. Push to batch 64-128 and that gap narrows to 5-10%. On Llama 3.1 70B FP16 at batch 32, MI355X runs about 18,000 tokens/sec ghly 19,500, close enough that batch size decides the winner more than the vendor does.
MI450 is expected H2 2026, with cloud availability trailing another 3-6 months behind that.
At batch 1-4, H100 with TensorRT-LLM holds a 20-30% throughput edge. Push to batch 64-128 and that gap narrows to 5-10%. On Llama 3.1 70B FP16 at batch 32, MI355X runs about 18,000 tokens/sec ghly 19,500, close enough that batch size decides the winner more than the vendor does.
MI450 is expected H2 2026, with cloud availability trailing another 3-6 months behind that.
2
0
2
766
Intel's Gaudi 3 skips NVLink entirely: 24x 200GbE RoCE v2 links for 4.8 Tbps aggregate interconnect, against H200's 900 GB/s NVLink and B200's 1.8 TB/s.
Compute is the bigger gap. Gaudi 3 hits 1,835 TFLOPS FP8 dense. H200 reaches 3,958. B200 reaches 4,500, roughly 2.5x Gaudi at 3.3 70B FP8 at batch 128: Gaudi 3 serves about 2,100 tokens/s, H200 about 3,200, B200 about 4,400.
Gaudi 3 draws 900W, between H200's 700W and B200's 1,000W. Ethernet-based scaling is the real pitch here, not raw throughput.
Compute is the bigger gap. Gaudi 3 hits 1,835 TFLOPS FP8 dense. H200 reaches 3,958. B200 reaches 4,500, roughly 2.5x Gaudi at 3.3 70B FP8 at batch 128: Gaudi 3 serves about 2,100 tokens/s, H200 about 3,200, B200 about 4,400.
Gaudi 3 draws 900W, between H200's 700W and B200's 1,000W. Ethernet-based scaling is the real pitch here, not raw throughput.
0
0
0
679
Every data center in Spheron's marketplace is Tier 3+ certified before it's listed. That's the same bar enterprise buyers already require from a hyperscaler, not a discount version of it.
InfiniBand connectivity and full root or bare-metal access come standard, not as an add-on tier.
Backed by a 99.9% uptime SLA, with deploy times as low as 60 seconds from the moment you pick a GPU.
Moving off a hyperscaler shouldn't mean giving up the guarantees that made you trust one in the first place.
InfiniBand connectivity and full root or bare-metal access come standard, not as an add-on tier.
Backed by a 99.9% uptime SLA, with deploy times as low as 60 seconds from the moment you pick a GPU.
Moving off a hyperscaler shouldn't mean giving up the guarantees that made you trust one in the first place.
1
0
0
893
Standard decoding on H200 tops out around 1,700 tokens/sec. Add Eagle-3 speculative decoding and the same card hits 6,200, a 3.5-3.7x speedup that holds across H100, H200 and B200.
The mechanism: a small draft model proposes several tokens ahead, the base model verifies them in ad of generating token by token. Chunked prefill on top of it cuts time-to-first-token from about 38ms to 8ms, a 70-80% drop.
Same hardware, same weights, just a different decoding loop.
The mechanism: a small draft model proposes several tokens ahead, the base model verifies them in ad of generating token by token. Chunked prefill on top of it cuts time-to-first-token from about 38ms to 8ms, a 70-80% drop.
Same hardware, same weights, just a different decoding loop.
0
0
0
787
At 1 concurrent request, TensorRT-LLM beats vLLM by 8%: 130 tokens/s against 120. Push to 100 concurrent requests and the gap widens to 16%, 2,780 versus 2,400 tokens/s.
TensorRT-LLM also wins on tail latency: 1,280ms p95 time-to-first-token at 100 requests, against vLLM's one-time compile step of about 28 minutes. vLLM cold-starts in roughly 62 seconds with none.
Tested on Llama 3.3 70B Instruct, FP8, single H100 SXM5. If your traffic is bursty and low-volume, that compile tax may not be worth paying.
TensorRT-LLM also wins on tail latency: 1,280ms p95 time-to-first-token at 100 requests, against vLLM's one-time compile step of about 28 minutes. vLLM cold-starts in roughly 62 seconds with none.
Tested on Llama 3.3 70B Instruct, FP8, single H100 SXM5. If your traffic is bursty and low-volume, that compile tax may not be worth paying.
3
0
1
1,183
HBM4 isn't broadly available until H2 2026. Your inference speedup is available today, it's called quantization.
On the same H200 card, dropping from BF16 to FP8 roughly doubles throughput, 34 to 69 tokens/sec at batch 1 on a 70B model. FP4 nearly quadruples it, to 137 gger jump than the memory generation you're waiting on will give you.
Rent the card you can get and requantize before you wait for hardware that isn't shipping yet.
On the same H200 card, dropping from BF16 to FP8 roughly doubles throughput, 34 to 69 tokens/sec at batch 1 on a 70B model. FP4 nearly quadruples it, to 137 gger jump than the memory generation you're waiting on will give you.
Rent the card you can get and requantize before you wait for hardware that isn't shipping yet.
2
0
0
1,141
CPU to GPU and GPU to GPU are not the same bus, and the gap is bigger than people plan for.
PCIe Gen5 x16 moves data CPU-to-GPU at about 64 GB/s. NVLink moves it GPU-to-GPU at 900 GB/s, 14x faster. Multi-GPU training that leans on PCIe for anything beyond the initial load is ndwidth on the table.
Know which link your framework actually uses for gradient sync before you blame the GPU for a slow step.
PCIe Gen5 x16 moves data CPU-to-GPU at about 64 GB/s. NVLink moves it GPU-to-GPU at 900 GB/s, 14x faster. Multi-GPU training that leans on PCIe for anything beyond the initial load is ndwidth on the table.
Know which link your framework actually uses for gradient sync before you blame the GPU for a slow step.
1
0
0
1,068