Pinned Tweet
Controversial take: well-specified, self-contained science problems, whether in quantum field theory, condensed-matter physics, quantum chemistry, or structural biology, if they can be described and verified digitally, will essentially all become turbo-charged math “word problems.” And they’ll fall to the same frontier models that are now cracking open problems in math.
But that is far, far from “solving” science. That requires layered evaluation, shared environment state across agents and experiments, and autonomous physical labs to close the loop with the real world. I gave a talk on this recently at lifelongagent.github.io — write-ups to follow.
TheoPhysEval:
"a suite of 50 unpublished research-level theoretical physics problems with verifiable answers."
"GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem."
arxiv.org/abs/2610.16733
But that is far, far from “solving” science. That requires layered evaluation, shared environment state across agents and experiments, and autonomous physical labs to close the loop with the real world. I gave a talk on this recently at lifelongagent.github.io — write-ups to follow.
TheoPhysEval:
"a suite of 50 unpublished research-level theoretical physics problems with verifiable answers."
"GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem."
arxiv.org/abs/2610.16733
2
0
12
638
Ying Liu retweeted
THE SCALE MAFIA
one of the most special parts of starting a company is watching the people you fought and bled with in the early days go out and change the world.
the Scale crew was supernatural. @scale_AI was one of the first companies to IOI and IMO-maxx, and the magic of hese geniuses gave a shit and ran through walls. it's no wonder that the Scale mafia is the first true mafia of the modern tech era. just to name a few:
- cognition founder @stevenkplus1 & president @russelljkaplan
- modal founder @akshat_b
- tech & science advisor to POTUS @mkratsios47
- claude code head of product @_catwu
- kpcb partners @LM_Braswell & @nayakkayak
- bun founder @jarredsumner
- tbd head of product @lindugong
- netic founder @melisatokmak
- cobot founder @BradPorter_
- secureframe founder @shravvmehtaa
- paramount skydance cfo @DennisKCinelli
- thinking machines gc lauren lennon
- partner at greylock @sophia_luo_
- liquid ai coo jeffrey li
- gallatin ai founder @agilemofo
- tbd safety @_julianmichael_
- center for ai safety president @devindkim
[photo courtesy the beautiful wedding of @thenateherman (who in many ways was a cofounder of Scale) and @ChristinaG325]
one of the most special parts of starting a company is watching the people you fought and bled with in the early days go out and change the world.
the Scale crew was supernatural. @scale_AI was one of the first companies to IOI and IMO-maxx, and the magic of hese geniuses gave a shit and ran through walls. it's no wonder that the Scale mafia is the first true mafia of the modern tech era. just to name a few:
- cognition founder @stevenkplus1 & president @russelljkaplan
- modal founder @akshat_b
- tech & science advisor to POTUS @mkratsios47
- claude code head of product @_catwu
- kpcb partners @LM_Braswell & @nayakkayak
- bun founder @jarredsumner
- tbd head of product @lindugong
- netic founder @melisatokmak
- cobot founder @BradPorter_
- secureframe founder @shravvmehtaa
- paramount skydance cfo @DennisKCinelli
- thinking machines gc lauren lennon
- partner at greylock @sophia_luo_
- liquid ai coo jeffrey li
- gallatin ai founder @agilemofo
- tbd safety @_julianmichael_
- center for ai safety president @devindkim
[photo courtesy the beautiful wedding of @thenateherman (who in many ways was a cofounder of Scale) and @ChristinaG325]
247
342
8,176
5,283,096
Like this if you've ever set your AI agents to work on technical problems and writings while you tidy up the room and desk their computer sits on.
What a time 🤖
What a time 🤖
0
1
6
341
Frontier labs are citing @scale_AI’s SWE-Bench Pro V2 in their model releases.
Cleaner, harder to game, and a much sharper measure of coding progress.
SWE-Bench Pro Paper: arxiv.org/pdf/2610.24206
Blog: labs.scale.com/blog/swe-bench…
Cleaner, harder to game, and a much sharper measure of coding progress.
SWE-Bench Pro Paper: arxiv.org/pdf/2610.24206
Blog: labs.scale.com/blog/swe-bench…
Today, we release our first model, Flint-40B-A2B, a fast local coding model built with our automated research system. huggingface.co/RecursiveSI/fl…
1. Strong performance. 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified.
2. It is trained from scratch (tokenizer->Pretrain->Midtrain->SFT->RL).
3. Fast in your local laptop. Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark.
4. We build the complete model training pipeline in 4 month!
1
3
37
1,608
Huge thanks to Wen Sang @sang_wen , co-founder & COO of @genspark_ai , for inviting me to visit the Genspark office in Palo Alto.
Wen is a serial entrepreneur, and the signal was clear as far back as our school days, when he was already doing venture club activities. The last timeompanies, it was Smarking, which he had just founded and which later went on to a successful exit. Great to see him building again.
He gave me an in-person demo of Genspark’s newest products (super cool designs) and left me with new insights on how to leverage enterprise agents and truly co-work with them day to day. And some research ideas on multi-agent orchestration and coordination too :)
Wen is a serial entrepreneur, and the signal was clear as far back as our school days, when he was already doing venture club activities. The last timeompanies, it was Smarking, which he had just founded and which later went on to a successful exit. Great to see him building again.
He gave me an in-person demo of Genspark’s newest products (super cool designs) and left me with new insights on how to leverage enterprise agents and truly co-work with them day to day. And some research ideas on multi-agent orchestration and coordination too :)
2
0
18
2,480
Solving the virtual clock problem in AgentEnv
Anyone who has worked on search engines knows that getting time right across the system during training, relative to query time, as in “what was the weather yesterday?”, is critical yet easy to overlook. Reproducible RL environments face the same challenge. In AgentEnv, we explored several approaches before landing on a clean solution for the virtual clock.
The core tension: inject the clock too low in the stack and it interferes with certificate validation and other system functions; inject it too high and you pay a large overhead while still missing invocations.
Anyone who has worked on search engines knows that getting time right across the system during training, relative to query time, as in “what was the weather yesterday?”, is critical yet easy to overlook. Reproducible RL environments face the same challenge. In AgentEnv, we explored several approaches before landing on a clean solution for the virtual clock.
The core tension: inject the clock too low in the stack and it interferes with certificate validation and other system functions; inject it too high and you pay a large overhead while still missing invocations.
Three agents and eight servers ask what time it is, all at once, and every one gets the same answer.
At @ScaleAILabs, we built this virtual clock into our open-source AgentEnv Framework so RL environments stay reproducible. Here's why it matters and how it works 🧵
1
0
6
581
Ying Liu retweeted
A great pianist can play almost anything, but mastering one piece is what makes a performance exceptional. The same goes for AI.
With today's launch of Nemotron 3.5 Super VL, we’re collaborating with @NVIDIAAI to combine their open post-training tools with Scale’s data, ions building expertise. Together, we're turning general-purpose models into powerful, cost-effective specialists for enterprises everywhere.
Welcome to the multi-model world. 🤝
With today's launch of Nemotron 3.5 Super VL, we’re collaborating with @NVIDIAAI to combine their open post-training tools with Scale’s data, ions building expertise. Together, we're turning general-purpose models into powerful, cost-effective specialists for enterprises everywhere.
Welcome to the multi-model world. 🤝
11
11
80
12,565
Ying Liu retweeted
Congrats to @GoogleAI's Gemini 4 Argon for taking the top spot of our Humanity's Sixth Sense (HSS) benchmark with a score of 55.4%, edging out GPT-6-Astra at 53.6%. 🥇
HSS tests AI’s ability to reason beyond what’s visible, from hidden objects to predicting what happens next and avior.
We took a closer look at performance across categories, and that’s where things get interesting. 👀🧵
HSS tests AI’s ability to reason beyond what’s visible, from hidden objects to predicting what happens next and avior.
We took a closer look at performance across categories, and that’s where things get interesting. 👀🧵
42
129
1,169
160,924
Fascinating event! Excited to help bring live, streaming benchmarks and arenas to life for agents tackling extremely long-horizon tasks—with shared resources, coordination, and competition adding new challenges.
Fantastic work by our partners @bake_ai_hq (main organizer), e, @UW, and @Stanford!
In the ongoing Stage 2, each model resumes from its own Nemotron checkpoint, incorporating feedback from its arena battles.
Watch live October 10–13 at rsiarena.live and join the checkpoint evaluation!
Fantastic work by our partners @bake_ai_hq (main organizer), e, @UW, and @Stanford!
In the ongoing Stage 2, each model resumes from its own Nemotron checkpoint, incorporating feedback from its arena battles.
Watch live October 10–13 at rsiarena.live and join the checkpoint evaluation!
Grok 4.7 ran out of API credit and was cut off 41 hours early. Nearly 70% of its GPU budget was still unused.
And somehow, the model it trained still made the top two!! @SpaceXSI
Stage 1 of RSIArena is over. Thank you, our collaborators @huggingface @scale_AI @NotreDame @UW and me! 🧡
Stage 2 training is now LIVE—and this time, the models have human feedback to work with.
Round 1 was a six-day competition: 10 models (AI researchers), the same @NVIDIAAI Nemotron base model, and 1,000 GPU-hours each to build something better.
They chose their data, wrote training code, ran experiments, and submitted their models.
At @COLM_conf and online, people then cast 1,685 votes in blind comparisons. @OpenAI GPT-6 Astra model finished first; the models trained by grok 4.7 and @Zai_org GLM-5.3 rounded out the top three.
Now those judgments go back into the experiment.
In Stage 2, each model resumes from its own Nemotron checkpoint with feedback from its model’s arena battles, another 500 GPU-hours, and $150 in API credit. Across both rounds, that’s a total budget of 1,500 GPU-hours and $450 in API credit per model.
The agents decide how to interpret the feedback, which experiments to run, and what to change in training. Their next models will show whether those decisions actually helped.
Stage 2 runs October 10–13. Watch the training live: 👇
0
0
12
694
Such a pleasure to run into Prof. Theja Tulabandhula (@TTulabandhula) of the University of Illinois Chicago just as I was heading home on the last day of @COLM_conf!
We were next-door office neighbors in grad school and used to chat and joke together a lot. A dozen years later, h other immediately, even if we both look a little different now. 😄
Alongside his role as a tenured professor at UIC, Theja has built an impressive industry career, including founding his own companies.
And of course, an old-friend reunion at an AI conference calls for the obligatory photo!
We were next-door office neighbors in grad school and used to chat and joke together a lot. A dozen years later, h other immediately, even if we both look a little different now. 😄
Alongside his role as a tenured professor at UIC, Theja has built an impressive industry career, including founding his own companies.
And of course, an old-friend reunion at an AI conference calls for the obligatory photo!
0
0
20
620
Ying Liu retweeted
Most existing safety benchmarks measure whether a model refuses a request, not whether it actually helps someone in crisis.
DistressBench contains 718 clinician-authored conversations across 23 self-harm subcategories.
Across 25 frontier models, the median model meets 71.4% of red against a 98.7% ceiling set by clinician reference responses on the same test.
The biggest gap is between recognizing a crisis and acting on it. In 35.3% of conversations where a model fully recognized the crisis, it still didn’t point the user toward human help.
DistressBench contains 718 clinician-authored conversations across 23 self-harm subcategories.
Across 25 frontier models, the median model meets 71.4% of red against a 98.7% ceiling set by clinician reference responses on the same test.
The biggest gap is between recognizing a crisis and acting on it. In 35.3% of conversations where a model fully recognized the crisis, it still didn’t point the user toward human help.
2
3
8
713
Ying Liu retweeted
People increasingly turn to AI chatbots for mental health support. But are models actually helping when it matters most?
We're releasing DistressBench, a new benchmark measuring how well AI models respond to mental health crises.
We're releasing DistressBench, a new benchmark measuring how well AI models respond to mental health crises.
1
7
25
2,006
Ilya @ilyasut saw this three years before most of us.
Raw intelligence, much like physical strength, may no longer be the defining driver of human productivity.
What will endure? Empathy, kindness, curiosity, and the simple ability to find joy in life.
Attending @COLM_conf to learn, talk, and make new friends feels like it :)
Raw intelligence, much like physical strength, may no longer be the defining driver of human productivity.
What will endure? Empathy, kindness, curiosity, and the simple ability to find joy in life.
Attending @COLM_conf to learn, talk, and make new friends feels like it :)
if you value intelligence above all other human qualities, you’re gonna have a bad time
0
0
12
560
Ying Liu retweeted
We're opening up early access Physical AI evaluations at @ScaleAILabs
Evaluating robotics models in real-world environments remains challenging. Across our SFHQ Robotics Lab and our Mexico City Data Factory, we have a range of mobile and stationary robotic embodiments to l evaluation.
If you're developing Physical AI models, we'd love to work with you. DM me if you're interested or email physical-ai@scale.com
Evaluating robotics models in real-world environments remains challenging. Across our SFHQ Robotics Lab and our Mexico City Data Factory, we have a range of mobile and stationary robotic embodiments to l evaluation.
If you're developing Physical AI models, we'd love to work with you. DM me if you're interested or email physical-ai@scale.com
13
9
167
10,490
I am honored to join the fantastic lineup of speakers at the Lifelong Agents Workshop at COLM 2026! I'll be giving a talk titled "From AI4Sci Headlines to Autonomous Scientific Discovery."
Using AI to tackle scientific problems has been a passion of mine since my PhD days, so this one is close to my heart. Two points I'll be making:
1. Science is the lifelong-agent problem in its hardest form.
2. The bottleneck for science agents isn't intelligence — it's the environments and verifiers we can build around them.
Hope to see many of you there!
Time: 1pm, Oct 9, Friday
Location: Hilton San Francisco Union Square
Workshop: lnkd.in/gNkjyNj5
RSVP: luma.com/h6vm3nqc
Using AI to tackle scientific problems has been a passion of mine since my PhD days, so this one is close to my heart. Two points I'll be making:
1. Science is the lifelong-agent problem in its hardest form.
2. The bottleneck for science agents isn't intelligence — it's the environments and verifiers we can build around them.
Hope to see many of you there!
Time: 1pm, Oct 9, Friday
Location: Hilton San Francisco Union Square
Workshop: lnkd.in/gNkjyNj5
RSVP: luma.com/h6vm3nqc
This Friday (Oct 9) at #COLM2026, we're hosting the 2nd Workshop on Lifelong Agents: a full day on how to build agents that keep learning after deployment while staying reliable and aligned.
Speakers:
• Sergey Levine (UC Berkeley)
• Graham Neubig (CMU / OpenHands)
• Yu Su ate)
• Fred Sala (Snorkel AI / UW–Madison)
• Philippe Laban (Microsoft Research)
• Shi Dong (RadixArk)
• Ying Liu (Scale AI)
📍 Continental Ballroom 6, Hilton San Francisco Union Square
🗓 Fri, Oct 9 · 8:30 AM – 5:00 PM PT
💻 Virtual option available
RSVP: luma.com/h6vm3nqc
Schedule: lifelongagent.github.io
Thanks to my co-organizers, our reviewers and authors, and to Salesforce AI Research, Snorkel AI, and Tencent for helping cover registration for student authors!
If you're at COLM and working on agents, come say hi!
#AIAgents #ContinualLearning
1
0
17
882
Ying Liu retweeted
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day.
Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding.
The gap is re 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding.
The gap is re 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
85
148
2,239
208,249
My posts here are usually all work and research, but this guide is exactly what I needed! I make the long commute to SF just for work, so the city feels so close yet still a mystery. Time to bring the family for a day trip (that is not for SF Zoo). Thanks, @_jasonwei !
Visiting SF for COLM and curious what SF is all about?
The bad news: COLM is in the dodgiest neighborhood and if you hang around there, you’ll hate SF just like I used to.
The good news: your twitter buddy Jason Wei wrote a guide on experiencing SF like a local: jasonwei.net/blog/the-best-…
0
0
5
795
We're open-sourcing a real gem: AgentEnv, the platform behind every RL environment we build at @scale_AI .
Our research team partnered with engineering and delivery teams to design and build it together, to generate dynamic, composable, reproducible environments, with real apps and real artifacts.
Our research team partnered with engineering and delivery teams to design and build it together, to generate dynamic, composable, reproducible environments, with real apps and real artifacts.
Building RL environments that are realistic, accurate, and scalable all at once is hard.
That’s what we built AgentEnv for: dynamic, composable, reproducible environments that work across agents, sandboxes, and clouds — with environment and agent triggers, a virtual clock to control time, RBAC for agents, and more.
Start building today: agentenvframework.com
9
10
113
8,944