Pinned Tweet
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains.
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions tasks. Claude Opus 5 solves only ~30%.
1/n 👇
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions tasks. Claude Opus 5 solves only ~30%.
1/n 👇
59
137
755
225,809
Steven Dillmann retweeted
Late, but it's here. 🔥
Gemini 4 Argon is now on the RSI-Exam leaderboard, and Google is back in the top 3.
📊 Full (88 tasks): 0.4943 — #3
🌐 Public (35): 0.5153 — #3
Huge jump from the Flash models. Seriously impressive. 👏 Now waiting on Gemini 4 Carbon… 👀
🔗 rsi-exam.ai
@OfficialLoganK
Gemini 4 Argon is now on the RSI-Exam leaderboard, and Google is back in the top 3.
📊 Full (88 tasks): 0.4943 — #3
🌐 Public (35): 0.5153 — #3
Huge jump from the Flash models. Seriously impressive. 👏 Now waiting on Gemini 4 Carbon… 👀
🔗 rsi-exam.ai
@OfficialLoganK
5
7
54
6,024
Steven Dillmann retweeted
Three science benchmarks, three very different tasks, each drawn below as one attempt from start to finish.
1. CritPt poses a research-level physics problem that takes several turns, context stacking as it goes, with one verdict at the end: the deepest yardstick we have.
2. ced scientific computation as a chain, each solution feeding the next step, with a reward at every link: the cleanest learning signal per dollar.
3. Terminal-Bench Science (@terminalbench) drops an agent into a real shell for one long run, tools and retries included, with a single check at the finish: the closest thing to how models will actually do science.
All three are benchmarks first; wrap each verifier as a reward, and each becomes an RL environment with its own depth, horizon, and reward rhythm. Which one would you start a training run with?
1. CritPt poses a research-level physics problem that takes several turns, context stacking as it goes, with one verdict at the end: the deepest yardstick we have.
2. ced scientific computation as a chain, each solution feeding the next step, with a reward at every link: the cleanest learning signal per dollar.
3. Terminal-Bench Science (@terminalbench) drops an agent into a real shell for one long run, tools and retries included, with a single check at the finish: the closest thing to how models will actually do science.
All three are benchmarks first; wrap each verifier as a reward, and each becomes an RL environment with its own depth, horizon, and reward rhythm. Which one would you start a training run with?
2
5
12
915
🚀 Terminal-Bench-Science is now featured in its first open-weight model release!
Kimi K3.1 Pro scores 38.1%, the strongest open-weight result on Terminal-Bench-Science to date.
Still, Terminal-Bench-Science shows BY FAR the biggest gap to SOTA (~30 points), a promising sign that it’s one of the most discriminative measures of frontier capability out there.
Kimi K3.1 Pro scores 38.1%, the strongest open-weight result on Terminal-Bench-Science to date.
Still, Terminal-Bench-Science shows BY FAR the biggest gap to SOTA (~30 points), a promising sign that it’s one of the most discriminative measures of frontier capability out there.
Introducing Kimi K3.1 Pro: our most capable 3T-class open-weight model.
🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal.
🔹 Full-project coding: games in Unreal, Unity & Godot.
🔹 Excels at parametric 3D modeling in Blender & CAD.
🔹 More reliable at finance with hallucination down by 65%.
🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks.
Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on Kimi.ai, Kimi Work, Kimi Code, and the Kimi API.
Open weights by Nov 4, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.ai/blog/kimi-k3-1…
10
10
106
3,691
Steven Dillmann retweeted
Learning that "RSI" benchmarks are just MLE/post-train style tasks repackaged with a new label was disappointing.
We could just take the ~20 ML-adjacent tasks from TB 4 and re-release them as RSI-bench and everyone would clap.
We could just take the ~20 ML-adjacent tasks from TB 4 and re-release them as RSI-bench and everyone would clap.
18
15
277
14,180
Steven Dillmann retweeted
TB-Science 🤝 FutureSim! At the Frontier Data Summit by @SnorkelAI
Turns out we are all from the same region in Germany! In the order:
Tübingen <-> Metzingen <-> Stuttgart
Turns out we are all from the same region in Germany! In the order:
Tübingen <-> Metzingen <-> Stuttgart
0
3
53
1,567
Terminal-Bench-Science featured on @ValsAI's new Vals Science Index alongside MysteryMechanism and BioMysteryBench.
Great to see AI evals finally giving science the attention it deserves. 🤖🧪
Great to see AI evals finally giving science the attention it deserves. 🤖🧪
OpenAI models like Astra and Sol currently top the index.
However, no model leads every discipline. GPT-6 Astra tops life and mathematical sciences, Claude Sonnet 5.5 tops physical and engineering sciences, and GPT-6.1 Sol tops earth sciences.
3
7
40
1,555
Steven Dillmann retweeted
We just launched PersonalAgentEval!🚀🚀🚀 (personalagenteval.com) The first benchmark built to evaluate personal agents across diverse users, preferences, and domains!
We benchmarked 7 popular personal agents across everyday workflows (e.g. booking dinners, paying bills, ch agent operated within accounts pre-loaded with 13 months of realistic history (347–470 records per person). To pass, an agent has to proactively spot crucial background details: like an unread message, a clause in a contract, or a promise made in a past chat. Each task and user simulation are validated and verified by real human.
Key Findings:
• 🏆 Overall Leaders: Instinct and OpenAI Dots topped the leaderboard.
• 🔍 Proactive Context Retrieval: Instinct excelled at uncovering crucial background details—finding 64 of 72 key facts before the user brought them up.
• ⚡ Speed & Efficiency: Grok Bot was both fast and effective, averaging 0.8 minutes of active work per task with a 50% completion rate. Meta Muse required the fewest average interaction turns.
• ⚠️ Missing Promised Steps: Even leading agents missed crucial follow-up steps. In one task, all 7 agents correctly identified a pharmacy mentioned in past logs, but only Grok Bot and Meta Muse remembered to notify the caregiver as promised.
Explore the full leaderboard and read our case study and analysis:
personalagenteval.com
(Building a personal AI agent? Reach out to us if you’d like to get your agent evaluated and featured on the leaderboard.)
#MatrAIx #AI #AIAgents #MachineLearning #LLMs #MetaMuse #GrokBot #OpenAIDots #Hark #ManusCue #Instinct
We benchmarked 7 popular personal agents across everyday workflows (e.g. booking dinners, paying bills, ch agent operated within accounts pre-loaded with 13 months of realistic history (347–470 records per person). To pass, an agent has to proactively spot crucial background details: like an unread message, a clause in a contract, or a promise made in a past chat. Each task and user simulation are validated and verified by real human.
Key Findings:
• 🏆 Overall Leaders: Instinct and OpenAI Dots topped the leaderboard.
• 🔍 Proactive Context Retrieval: Instinct excelled at uncovering crucial background details—finding 64 of 72 key facts before the user brought them up.
• ⚡ Speed & Efficiency: Grok Bot was both fast and effective, averaging 0.8 minutes of active work per task with a 50% completion rate. Meta Muse required the fewest average interaction turns.
• ⚠️ Missing Promised Steps: Even leading agents missed crucial follow-up steps. In one task, all 7 agents correctly identified a pharmacy mentioned in past logs, but only Grok Bot and Meta Muse remembered to notify the caregiver as promised.
Explore the full leaderboard and read our case study and analysis:
personalagenteval.com
(Building a personal AI agent? Reach out to us if you’d like to get your agent evaluated and featured on the leaderboard.)
#MatrAIx #AI #AIAgents #MachineLearning #LLMs #MetaMuse #GrokBot #OpenAIDots #Hark #ManusCue #Instinct
49
54
207
7,572
Great visit to the @ArtificialAnlys office in SF today with @zijian42chen. We got to watch AA-Robotics v0.1 launch in real time! Really exciting work, and I’d love to see it extend to scientific tasks in the physical world.
Thanks for hosting, @wmkeats & @_micah_h!
Thanks for hosting, @wmkeats & @_micah_h!
Welcome to the era of embodied AI.
We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a bjective. We’re releasing results for our first eight tasks today.
We find that frontier language models in coding agent harnesses have recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.
GPT-6 Astra places first in AA-Robotics v0.1, followed by GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5.
AA-Robotics v0.1 details:
➤ Our first eight tasks test a range of capabilities: control and dexterity (precise grasping and object handling), spatial reasoning (how objects interact with one another) and adaptability (how models respond to the unexpected)
➤ Models get only brief instructions: how to connect to the robot arm (via a simple HTTP API), the arm's name and how to submit an attempt. They can view the scene through four cameras and write and run code to plan and send movements
➤ Once a model submits an attempt, its end state is graded against the task's pass criteria. A pass scores 100; an incomplete attempt earns partial credit. All attempts are zero-shot
Key findings:
➤ The GPT family tops the leaderboard: GPT-6 Astra scores 69, ahead of GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. Claude Opus 5.5 (49) and Claude Fable 5.1 (46) trail
➤ Today’s models are incredibly slow at our tasks, taking ~30 minutes on average to complete each tabletop task
➤ Cost varies widely: GPT-6.1 Sol is the cheapest at $1.93 per task on average; GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much
➤ Compared with a human, today's best-performing frontier language models are ~100x slower and ~10x more expensive
More results, clips, and how AA-Robotics v0.1 works follow in the thread below.
2
1
14
882
Steven Dillmann retweeted
Despite the excitement around end-to-end scientific agents and their success on well-scoped optimization tasks, evaluating open-ended research remains a challenge. We're collecting tasks for SciVeri-Bench, with scientists in the loop to assess research quality. Join us!
🔬 Calling scientists across disciplines: help build SciVeri-Bench v1.0!
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJ…
🌐 Website: sciveri-bench.com
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
1
2
33
3,585
Steven Dillmann retweeted
🔬 Calling scientists across disciplines: help build SciVeri-Bench v1.0!
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJ…
🌐 Website: sciveri-bench.com
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJ…
🌐 Website: sciveri-bench.com
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
2
16
74
7,564
Steven Dillmann retweeted
Many are criticizing that the released math results are not up to the academic standards of journal publication. I think this , on the contrary, gives a rare chance for human to interact at an earlier stage of the work. It would be more fruitful this way.
1
1
12
1,212
Steven Dillmann retweeted
That's a wrap on Frontier Data Summit!
More than 30 speakers and a room full of researchers came together to discuss how we measure and advance frontier AI 🚀
More than 30 speakers and a room full of researchers came together to discuss how we measure and advance frontier AI 🚀
0
7
39
1,496
Always loved the work done by @ArtificialAnlys - excited to hear that Terminal-Bench-Science will be included in the next version of the Artificial Analysis Intelligence Index 🚀
Announcing that Artificial Analysis Intelligence Index v5 will come in late October
Intelligence Index v5 will be a significant leap in benchmarking the frontier of AI. Changes include adding Terminal-Bench Science and a new coding benchmark with a private dataset.
Stay tuned for more.
2
5
30
1,698
Steven Dillmann retweeted
Harbor supports network restrictions, separate verifiers, and resource controls.
Different defaults aren’t missing features. Giving users control is deliberate.
Different defaults aren’t missing features. Giving users control is deliberate.
Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.
7
2
94
3,845
Steven Dillmann retweeted
In 22 days, @StevenDillmann from @Stanford will be joining us to share his thoughts on the future of RL.
It's not too late to register for Convergence RL. luma.com/gbcompna
It's not too late to register for Convergence RL. luma.com/gbcompna
1
2
5
628
I’ll be in SF all day on Thursday & Friday this week. If you’re in SF for COLM or SF Tech Week and would like to chat and get Terminal-Bench-Science merch, get in touch via DM!
Thanks to @LaudeInstitute & @harborframework for organizing the merch.
Thanks to @LaudeInstitute & @harborframework for organizing the merch.
4
2
52
1,634
If you'd like to learn about Harbor, Terminal-Bench, and Terminal-Bench-Science without reading through a wall of text, check out these lightning talks by @alexgshaw, @ryan_marten, and me from our recent @terminalbench meetup. Now live on YouTube📽️
Recordings from @terminalbench's latest meetup are now on YouTube! Watch here: youtube.com/watch?v=VTUFfd…
0
2
21
986