Yoonho Lee retweeted
How do we scale up the context of LLMs?
Agentic search scales to arbitrarily large amounts of info, but can easily miss relevant info.
Idea💡: Optimize organization of info for downstream utility. Leads to 30+% gains even on OOD domains
Paper + code: librarian-agents.github.io
Agentic search scales to arbitrarily large amounts of info, but can easily miss relevant info.
Idea💡: Optimize organization of info for downstream utility. Leads to 30+% gains even on OOD domains
Paper + code: librarian-agents.github.io
LLMs deployed in specialized domains, from enterprise codebases to law firms, dig through a mountain of unstructured context!
Our new paper, Librarian Agents, asks: can an agent adapt by curating its own long-context workspace, anticipating queries before they arrive? 🧵⬇️
17
37
323
24,303
Yoonho Lee retweeted
Introducing Slingshots // FOUR: 14 projects across eight top institutions. Self-improving agent systems and harnesses, maximally-open models and training data, applications for research and decision-making – plus kernels and more. Say hello to the batch 🧵
6
31
90
16,079
ScholarCatalyst is part of Slingshots Four!
Deeply “reading” the literature, i.e. knowing which earlier idea a new problem needs, takes years to get even okay at. Even the best systems do poorly on ScholarCatalyst. With @LaudeInstitute's support, we’ll train directly for this.
Deeply “reading” the literature, i.e. knowing which earlier idea a new problem needs, takes years to get even okay at. Even the best systems do poorly on ScholarCatalyst. With @LaudeInstitute's support, we’ll train directly for this.
ScholarCatalyst // @yoonholeee, @ohmyksh, @chelseabfinn (@Stanford) — A benchmark for finding prior work that could inspire scientific research, and a retrieval system trained for this task.
2
6
25
1,997
Benchmark paper: arxiv.org/abs/2610.02202
Laude only allowed registering up to 5 members, but the team is much larger @ohmyksh @Benjamin_eecs @dayoon12161 @RulinShao @seungonekim @gneubig @PangWeiKoh @achowdhery @AkariAsai @lateinteraction @YejinChoinka @gunheekim @chelseabfinn
x.com/yoonholeee/sta…
Laude only allowed registering up to 5 members, but the team is much larger @ohmyksh @Benjamin_eecs @dayoon12161 @RulinShao @seungonekim @gneubig @PangWeiKoh @achowdhery @AkariAsai @lateinteraction @YejinChoinka @gunheekim @chelseabfinn
x.com/yoonholeee/sta…
Great researchers have an uncanny ability to make connections that seem inevitable in hindsight, in places nobody else would have thought to look. Can we measure this ability?
Introducing ScholarCatalyst: a far-from-saturated benchmark for finding what we call "catalyst 4 lead authors on 207 of their own recent projects.
Paper: arxiv.org/abs/2610.02202
To make sustained progress on open-ended problems, I think agents need the sort of "research taste" that great researchers have. They need to make deep connections between earlier discoveries and problems those discoveries weren't intended to solve. ScholarCatalyst is a first step towards this goal.
More details in the thread below🧵
0
2
16
862
I'm giving a Rising Star talk on text optimization and "cumulative learning" at the AI Symposium at UMich today! (10:15AM ET)
⏰ Happening TOMORROW!
The 2026 #MichiganAI Symposium brings together researchers, industry leaders & students to explore foundational models & their impact on academia, industry and society.
🎤 Keynotes, talks, posters & more, join us at @UMichCSE!
ai.engin.umich.edu/events/2026-ai…
2
7
62
2,383
Yoonho Lee retweeted
LLMs deployed in specialized domains, from enterprise codebases to law firms, dig through a mountain of unstructured context!
Our new paper, Librarian Agents, asks: can an agent adapt by curating its own long-context workspace, anticipating queries before they arrive? 🧵⬇️
Our new paper, Librarian Agents, asks: can an agent adapt by curating its own long-context workspace, anticipating queries before they arrive? 🧵⬇️
7
32
176
43,984
We’d like to condition agents on large text corpora, but most are poorly organized and/or poorly written. Can we train an agent to write well-indexed, LLM-native notes optimized for future agents?
We train a builder agent with RL to maximize the accuracy of a frozen answering agent that reads its workspace, rewarding each edit based acc improvement. Even a small, frozen 4B answerer far outperforms the strongest retrieval baselines on BrowseComp-Plus, and the builder transfers zero-shot to unseen BRIGHT documents.
Scaling this up points to a different kind of continual learning, where an agent adapts to a new domain by writing for its future self rather than updating weights. Co-led with @Anikait_Singh_ @teresaszhang
We train a builder agent with RL to maximize the accuracy of a frozen answering agent that reads its workspace, rewarding each edit based acc improvement. Even a small, frozen 4B answerer far outperforms the strongest retrieval baselines on BrowseComp-Plus, and the builder transfers zero-shot to unseen BRIGHT documents.
Scaling this up points to a different kind of continual learning, where an agent adapts to a new domain by writing for its future self rather than updating weights. Co-led with @Anikait_Singh_ @teresaszhang
LLMs deployed in specialized domains, from enterprise codebases to law firms, dig through a mountain of unstructured context!
Our new paper, Librarian Agents, asks: can an agent adapt by curating its own long-context workspace, anticipating queries before they arrive? 🧵⬇️
3
6
40
2,490
Related to earlier works that let a model "study" its context before seeing a task.
Sleep-time compute @nlpkevinl @sea_snell arxiv.org/abs/2504.13171
Machine Studying @jacobli99 x.com/jacobli99/stat…
Sleep-time compute @nlpkevinl @sea_snell arxiv.org/abs/2504.13171
Machine Studying @jacobli99 x.com/jacobli99/stat…
Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, all this problem Machine Studying.
2
1
9
606
Also closely related to Context LMs from @RulinShao ; both works treat context management as a text-to-text task to RL on, but point this at different text artifacts (corpus filesystem vs next-turn context).
x.com/RulinShao/stat…
x.com/RulinShao/stat…
‼️The Bitter Lesson for context management: Giving LMs unrestricted control over their context beats human-designed SOTA!
Introducing 🩵Context Language Models (CLMs)🩵
- Natively manage their own context
- Treat context as a file
- Learn policies in CLM weights, no harness
0
0
3
206
Yoonho Lee retweeted
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it?
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab b @AmesNatLab @PPPLab @Livermore_Lab
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab b @AmesNatLab @PPPLab @Livermore_Lab
23
60
295
132,796
Yoonho Lee retweeted
🔬 Calling scientists across disciplines: help build SciVeri-Bench v1.0!
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJ…
🌐 Website: sciveri-bench.com
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJ…
🌐 Website: sciveri-bench.com
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
2
16
74
7,580
Yoonho Lee retweeted
The Pi API is officially in beta! 🤖
Many people have already been using it to fine-tune π models for their robots -- below are a few
We’re hoping to expand testing significantly. Let us know if you are interested!
Blog: pi.website/blog/api-beta
Signup: pi.website/waitlist
Many people have already been using it to fine-tune π models for their robots -- below are a few
We’re hoping to expand testing significantly. Let us know if you are interested!
Blog: pi.website/blog/api-beta
Signup: pi.website/waitlist
22
50
560
29,218
Yoonho Lee retweeted
"Stateless Language Agents: Scaling Long-Horizon Automated Research"
Long-running AI research agents waste tokens revisiting old ideas or repeating experiments.
Stateless Language Agents (SLA) instead store research progress externally, giving each agent a fresh, focused r coordinates experiments.
This setup outperforms competing frameworks on software engineering and optimization tasks, matching the strongest baseline on one kernel optimization task with 93% fewer tokens.
alphaxiv.org/abs/2610.07625
Long-running AI research agents waste tokens revisiting old ideas or repeating experiments.
Stateless Language Agents (SLA) instead store research progress externally, giving each agent a fresh, focused r coordinates experiments.
This setup outperforms competing frameworks on software engineering and optimization tasks, matching the strongest baseline on one kernel optimization task with 93% fewer tokens.
alphaxiv.org/abs/2610.07625
14
25
192
11,639
Autoresearch loops are a particularly promising form of continual learning that we can scale indefinitely over time. However, in practice, they often plateau as runs get longer. In naive implementations of such loops, agents "lose motivation" after seeing full history and talk themselves out of trying anything new.
Stateless Language Agents (SLA) looks at why this happens and proposes a few simple modifications to scale usefully to 1B tokens (!). The core trick is to hide as much context as possible from each worker while having a separate advisor agent handle coordination.
SLA achieves the best final result on every task we tested (vs EvoX, CORAL, SwarmResearch), often by a wide margin!
Stateless Language Agents (SLA) looks at why this happens and proposes a few simple modifications to scale usefully to 1B tokens (!). The core trick is to hide as much context as possible from each worker while having a separate advisor agent handle coordination.
SLA achieves the best final result on every task we tested (vs EvoX, CORAL, SwarmResearch), often by a wide margin!
AutoResearch agents keep burning tokens after they stop making progress. 💰😭
From authors of ACE and Meta-Harness: Introducing Stateless Language Agents (SLAs). Stateful search, stateless agents. Advancing the performance-cost Pareto frontier of RSI.
Across 9 tasks, up to 1B sult on every task 📈
- 14.5× fewer tokens to match the best kernel baseline 🚀
- Advisor (the “coordinator”) uses < 0.6% of total tokens 🫣
📑 Paper: arxiv.org/abs/2610.07625
NOT JUST HIGHER NUMBERS. Some more surprising findings in the thread below! 🧵 #LLM #AI #RSI #Agent
14
21
256
12,986
Yoonho Lee retweeted
Life update: I defended my PhD at @StanfordAILab and recently joined @OpenAI!
Extremely thankful to my advisors (@chelseabfinn & @percyliang), collaborators, friends, and everyone else who supported me along the way. ❤️
Very excited for this next chapter!
Extremely thankful to my advisors (@chelseabfinn & @percyliang), collaborators, friends, and everyone else who supported me along the way. ❤️
Very excited for this next chapter!
89
33
1,429
84,739
Excited to see Meta-Harness featured in the 2026 @stateofai report!
This was a great read; a lot of work points to text and code as places where genuine learning can happen.
Full report: stateof.ai
This was a great read; a lot of work points to text and code as places where genuine learning can happen.
Full report: stateof.ai
welcome to the 9th annual @stateofai report!
ai building better ai, world models, physical ai, geopolitics, cyber defense and 12-month predictions.
read it at stateof(dot)ai or watch my editor’s cut here.
a few of this year’s biggest findings follow:
4
7
46
2,959
We need AI scientists with the taste to know when to stop chasing the score and start trying to understand.
SciVeri-Bench🔬 takes a step in this direction by bringing domain scientists into the evaluation loop.
Led by @passing2961, we're looking for scientist contributors!
SciVeri-Bench🔬 takes a step in this direction by bringing domain scientists into the evaluation loop.
Led by @passing2961, we're looking for scientist contributors!
Can AI agents verify if AI agents are doing good science?🔬
To evaluate this capability, we’re developing SciVeri-Bench, a deploy-first, scientist-driven benchmark for scientific verification on open-ended, novel scientific tasks.
We focus on three tasks:
- Scientist rubric critique generation
- Scientist taste evaluation
To build the benchmark, we introduce a Scientist-in-the-Loop harness that supports collaboration between domain scientists and AI researchers, with all communication taking place on OpenReview.
We’re grateful to have been invited to the Frontier Data Summit, where we’ll present our SciVeri-Bench poster on October 8 in San Francisco! If you’re interested in AI for Science and scientific verification, please stop by our poster to chat. We’d also love to meet for coffee if you’re attending #COLM2026! ☕
Currently, 16 scientists from 11 institutions are building SciVeri-Bench v0.1 together as we work toward v1.0. We’re looking for scientists to join as task contributors and sponsors to support the compute costs of running AI agents.
📣 We’ll post a call for task contributors here on X within the next week—stay tuned if you’re interested!
🙏 Huge thanks to our amazing team: @seungonekim,@jinheonbaek, @SoojungYang2, @yoonholeee, @JiayiiGeng, @dongyeopkang
2
1
26
1,985
Yoonho Lee retweeted
Also check out @ShayneRedford’s post and @yoonholeee’s blog about “What Is Taste?”:
x.com/ShayneRedford/…
yoonholee.com/blog/2026/tast…
x.com/ShayneRedford/…
yoonholee.com/blog/2026/tast…
What happens to the direction of science when we delegate it to AI?
With @seungonekim, we ranked all 6,617 #ICML2026 papers with AI—the AI Awards.
Findings:
1⃣Humans & AI disagree. Significantly.
2⃣AI is highly influenced by online hype.
3⃣AI preferred papers w/ perceived impacross settings, whereas humans preferred novelty.
4 AIs agreed fairly highly with one another.
🔗 research-taste.github.io/ai-as-icml-ac/
1/
0
1
6
521
Yoonho Lee retweeted
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods
Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.
Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.
We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/frontier-ai-is…
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods
Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.
Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.
We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/frontier-ai-is…
49
89
305
52,783
