Pinned Tweet
Last week we released s1 - our simple recipe for sample-efficient reasoning & test-time scaling.
We’re releasing 𝐬𝟏.𝟏 trained on the 𝐬𝐚𝐦𝐞 𝟏𝐊 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 but performing much better by using r1 instead of Gemini traces. 60% on AIME25 I.
Details in 🧵1/9
We’re releasing 𝐬𝟏.𝟏 trained on the 𝐬𝐚𝐦𝐞 𝟏𝐊 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 but performing much better by using r1 instead of Gemini traces. 60% on AIME25 I.
Details in 🧵1/9
DeepSeek r1 is exciting but misses OpenAI’s test-time scaling plot and needs lots of data.
We introduce s1 reproducing o1-preview scaling & performance with just 1K samples & a simple test-time intervention.
📜arxiv.org/abs/2501.19393
23
116
762
167,196
my talk @MIT NLP Seminar on Infinite test-time scaling & prefix sliding :)
we seem close to models that can work autonomously for weeks to solve very hard tasks
maybe the final boss: replace compaction
youtube.com/watch?v=CKdlVo…
we seem close to models that can work autonomously for weeks to solve very hard tasks
maybe the final boss: replace compaction
youtube.com/watch?v=CKdlVo…
4
16
210
19,325
@MITe i think compaction can get us there as is, itd be very inefficient
e.g. assume:
- 32 dense layers
- 4096 hidden dim
- bf16
- 500K context
- 5K compaction window
- 1 token ~= 4bytes
kv cache: 32 * 4096 * 2bytes * 500K * 2 (k & v) = 262GB
compaction summarizes that into 5000 * 4brash 262GB to keep 20KB!
e.g. assume:
- 32 dense layers
- 4096 hidden dim
- bf16
- 500K context
- 5K compaction window
- 1 token ~= 4bytes
kv cache: 32 * 4096 * 2bytes * 500K * 2 (k & v) = 262GB
compaction summarizes that into 5000 * 4brash 262GB to keep 20KB!
1
1
16
1,364
@MIT thanks to @shannonzshen @linluqiu @jacobli99 @HuLillian39250 for hosting; as well as to @ChengLuo_lc & team of the @COLM_conf reasoning workshop for also having me present on the topic! :)
0
1
4
848
interesting work! sophisticated ways to remove past LLM context ("kv-cache eviction") seem to mostly come down to how much of the prefix they preserve
it's a cleaner motivation for prefix sliding than ours in arxiv.org/abs/2608.26070
it's a cleaner motivation for prefix sliding than ours in arxiv.org/abs/2608.26070
Every KV cache evictor for efficient long reasoning scores cached tokens to decide which ones to keep. We find the score buys almost nothing: keep the prompt, evict at random per head, and you match the strongest evictor at 32–43% higher throughput.
Paper: arxiv.org/abs/2609.03430
Page: arthur-heng.github.io/Random-Attenti…
3
6
60
8,752
how clean/simple a method is makes for a good metric i think! a rough measure is # of hyperparams it adds:
compaction: summary max len, summary prompt; summary trigger, summary model, summary placement
prefix sliding: window size
compaction: summary max len, summary prompt; summary trigger, summary model, summary placement
prefix sliding: window size
Incredible that this works! Really clean idea, I would’ve thought a more complicated compaction scheme would be needed.
1
0
40
4,656
new paper: Prefix Sliding for efficient test-time scaling
vanilla full attention OOMs on long tasks & compaction loses important details -- prefix sliding is a simple & fast alternative that can outperform both
📜arxiv.org/abs/2608.26070
vanilla full attention OOMs on long tasks & compaction loses important details -- prefix sliding is a simple & fast alternative that can outperform both
📜arxiv.org/abs/2608.26070
18
42
309
60,547
there’s much scaling up to do & extensions to try so put some ideas in the paper. very excited about the potential of this so would love to hear from anybody experimenting w/ prefix sliding!
📜arxiv.org/abs/2608.26070
💻github.com/Muennighoff/pr…
📜arxiv.org/abs/2608.26070
💻github.com/Muennighoff/pr…
1
1
11
1,063
grateful for support from @LaudeInstitute w/ @bradenjhancock @andykonwinski & team; the NVIDIA Academic Grant Program & @PrimeIntellect
thanks to great coauthors @zpwang20 @ZeCh452318 @WeijiaShi2 @huybery@huyberya@jyangballinJiang @mikasenghaas @Grad62304977 @johannes_hage @samsja19 @lschmidt3 @percyliang @_jasonwei @AndrewYNg @LukeZettlemoyer @YejinChoinka @ml_perception
thanks to great coauthors @zpwang20 @ZeCh452318 @WeijiaShi2 @huybery@huyberya@jyangballinJiang @mikasenghaas @Grad62304977 @johannes_hage @samsja19 @lschmidt3 @percyliang @_jasonwei @AndrewYNg @LukeZettlemoyer @YejinChoinka @ml_perception
2
0
14
1,302
can LLMs replace embedding models? in a new work "embedder's dilemma" we find LLMs now beat embedding models -- but at much higher cost. when to choose which?...
📜 arxiv.org/abs/2608.12875
📜 arxiv.org/abs/2608.12875
Did LLMs finally subsume embedding models?
We compared 10 LLMs vs. 26 embedding models across 37 MTEB tasks.
LLMs scored 77.6 while embedding models scored 77.2
But the costs are very different. 🧵
19
77
563
102,249
for retrieval tasks, especially if reasoning-heavy, the extra cost can be worth it
for classification/sts tasks usually best to stick with embeddings. Clustering a bit mixed but as embedding is usually cheaper, LLM may not be worth it.
for classification/sts tasks usually best to stick with embeddings. Clustering a bit mixed but as embedding is usually cheaper, LLM may not be worth it.
1
3
22
3,000
great work by @AdnanElAssadi on this; we've pondered this question during prior work like arxiv.org/abs/2510.10062 so he quickly knocked it out. also great opportunity to work with the one & only @leejnhk!
thanks to @LaudeInstitute for their support!
thanks to @LaudeInstitute for their support!
1
1
21
1,454
