John Schulman retweeted
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.
We're built ange that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.
Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.
I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.
I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
We're built ange that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.
Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.
I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.
I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
252
302
3,282
260,166
inspiring post, and glad to be working together!
I’ve joined @thinkymachines to work on safety & alignment.
The default trajectory is that as AI gets more powerful, control over it will concentrate in the hands of a few. I’d rather build safety systems that let control over AI be shared widely.
Longer thoughts below.
11
4
298
39,934
Embedding evaluators is a big positive development, and props to Sam/OpenAI for agreeing to do it as well. It's been cool to see "pacing the frontier" become a thing so quickly.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must-p…
20
22
420
32,174
Enjoyed chatting with Dwarkesh, Beren, and Charlie. Thanks for having us on, Dwarkesh!
New episode with @johnschulman2, @oneill_c and @BerenMillidge.
I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.
0:00:00 against RSI
0:18:39 – What’s driving the Chinese labs’ progress
0:28:06 – How will automated AI researchers be trained
0:33:51 – Will long-horizon RL elicit AGI?
0:45:24 – The sim-to-real gap
1:00:33 – How much progress is explained by data?
1:18:03 – Why is RL working so well?
1:24:54 – Move 37 and entropy collapse
1:28:31 – Rapid-fire timelines
15
24
448
61,848
As a follow-up, it's exceedingly unlikely that training on user data contributes much to frontier model gains in areas like math -- those come from scaling up pretraining and RLVR. User data is more likely used to find failure modes or situations that are hard to recreate with hired annotators.
That said, model companies vary in how aggressively they train on user data (and uploading repos isn't hypothetical). I wish there were stronger norms around disclosing how companies train on user data -- with what methods, and to improve what capabilities.
That said, model companies vary in how aggressively they train on user data (and uploading repos isn't hypothetical). I wish there were stronger norms around disclosing how companies train on user data -- with what methods, and to improve what capabilities.
This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing.
- pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper
- use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this
- use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs"
"De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
33
56
880
101,248
First step is for industry leaders OpenAI and Anthropic to stop feuding and work on a pacing proposal together. They'll cite antitrust, but that's fake -- antitrust prohibits certain agreements, but not from jointly developing a proposal. Bringing in USG before there's a concrete proposal will likely result in something dumb (see: our pre-release testing program)
I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.
25
61
819
60,273
This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing.
- pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper
- use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this
- use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs"
"De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
- pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper
- use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this
- use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs"
"De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
Two things to distinguish:
Did any human or agent look at user data as part of the Navier Stokes effort? No.
Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company.
44
153
1,362
284,386
Bullish on this direction. Having a metric for explanation quality makes it possible to hillclimb, and counterfactual simulatability seems right. Adam et al. created a dataset+pipeline that creates more diverse+realistic test cases than prior work & do interesting exps on it. Can also train models to write better post-hoc explanations of their behavior, as highlighted in this thread.
Can a model learn to explain its behaviors, such as why it ignored a user request or made a coding error?
We trained models on thousands of explanations of their own in-the-wild behaviors.
Training on this single general dataset shows generalization to held-out evals.
🧵
16
51
579
60,433
I was also happy to see that this paper, and an earlier one by Hase et al. also on counterfactual simulatability (but more focused on training) used tinker for their fine-tuning experiments x.com/tinkerapi/stat…
Can you explain LLM behavior? If you can predict how the output changes with a different prompt, you're on the right track. Two recent papers used Tinker to test applications of counterfactual simulatability for interpretability:
arxiv.org/abs/2602.20710
alphaxiv.org/pdf/2608.16747
2
0
32
6,937
John Schulman retweeted
An example of a choice many businesses face now or in the near future. You've spent years collecting or annotating data specific to your business. What do you do with it?
One path: sell it to data vendors or frontier labs. Next-gen models get better at these tasks for everyone, including your current and future competitors.
The other: fine-tune existing models on it yourself and keep the edge that you built up throughout the years.
Tinker makes the second path practical for businesses without big ML infra teams. If Tinker doesn't cover your use case today, reach out and we'll make it work tomorrow.
One path: sell it to data vendors or frontier labs. Next-gen models get better at these tasks for everyone, including your current and future competitors.
The other: fine-tune existing models on it yourself and keep the edge that you built up throughout the years.
Tinker makes the second path practical for businesses without big ML infra teams. If Tinker doesn't cover your use case today, reach out and we'll make it work tomorrow.
We published how @thinkymachines 's Inkling did on Voice Code Bench few weeks ago, and then we asked, what if we fine-tune Inkling? Inkling's main value prop is its small size and robustness, making it a good base model for domain specific fine-tuning. So we that's what we did.
Inkling speech model on 1, 25, and 100 hours of our proprietary data containing alphanumeric entities. And the curve kept moving. 🍊
Not surprising, on the standard 300-item VoiceCodeBench evaluation, the 100-hour checkpoint delivered the strongest result.
Untrained Inkling → 100-hour post-trained Inkling:
📈 Task Success Rate: 56.33% → 79.00% (+22.67 pts)
📈 Entity recovery (CTEM): 86.84% → 94.80% (+7.96 pts)
📉 VoiceCodeBench WER: 2.3748% → 1.6107% (32.2% relative reduction)
🔧 139 misses fixed, 21 prior hits regressed: net +118 exact entities recovered
And the gains scaled as data scaled:
1h: 88.06% CTEM / 59.33% TSR / 2.8188% VCB WER
25h: 92.85% / 72.67% / 1.8517%
100h: 94.80% / 79.00% / 1.6107%
The 100-hour model recovered values the base model missed:
•--revert-last, instead of splitting one flag into “--revert --last”
• tests/auth/login.spec.ts, instead of test/auth/login.spec.ts
• ALLOWLIST_CIDR, instead of inserting an extra underscore
• SN-7KX-9042, instead of dropping the final digit
The largest entity-type gains were email addresses (+30.8 pts), postal addresses (+30.0), file paths (+23.5), environment variables (+22.9), and IP addresses (+20.0).
That’s the Besimple thesis: targeted human data can move the production metrics that matter for voice agents, even when the base model is already strong.
DM me if you want to evaluate your model on this benchmark or build the data that moves it. 🍊
#SpeechRecognition #VoiceAI #ASR #Transcription #DataQuality #PostTraining #Benchmarks
Checkout the full blog at: besimple.ai/research/voice…
6
8
81
12,322
John Schulman retweeted
Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents
For post-training, your agents can use @tinkerapi to launch concurrent RL runs while you watch your experiment tree grow
Try it out now on OpenResearch: openresearch.sh
For post-training, your agents can use @tinkerapi to launch concurrent RL runs while you watch your experiment tree grow
Try it out now on OpenResearch: openresearch.sh
19
71
542
61,841
On the OpenAI agents forming message boards: it's surprising that they developed such a strong "altruistic" drive to help each other. I wonder if this is caused by RL on parallel subagent setups where all agents get rewarded when the team succeeds.
74
76
1,522
148,289
Interesting how these models go into a monomaniacal rage on cyber evals. I wonder if we're seeing chunky post-training arxiv.org/abs/2602.05910 in action, where the models pattern-match the situation to a part of the RLVR training distribution where task completion is the only reward, and the aligned behavior learned elsewhere doesn't generalize. There might even be a chunk consisting of CTF-style tasks.
On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations.
The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from .6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn.
You can read the incident report and full technical document here: aisi.gov.uk/blog/incident-…
29
68
634
150,404
We love open weights and plan to keep releasing open-weight models and fine-tuning tools. But we’re not absolutists; misuse risks are real. Here’s how we’re thinking about a safe path forward, and the research needed to get there. Come work on it with us.
Releasing weights indiscriminately isn't safe. Neither is keeping capable models inside a few labs.
We think there's a path between them. We haven't mapped all of it. Our new post covers the part we can see: how we assessed Inkling, and why access should widen in stages.
thinkingmachines.ai/blog/a-safe-pa…
17
25
541
59,657
John Schulman retweeted
I've been skeptical of slowdown mechanisms in the past, so I want to elaborate on why I decided to sign this statement. I weighed three points, and want to lay them out for you to consider as well.
Flagging this is in my personal capacity -- I am not speaking for any employer past or present.
1) Will we need a slowdown?
We have made good progress on alignment. LLMs do what we want them to do most of the time. I don't remember the last time a model completely misread my intent. When they have failed, it's usually because they weren't smart enough.
But there are signs that our current approaches are not enough for the most powerful intelligences. We all saw the HuggingFace incident. I don't think it demonstrates a need to pause today, but it shows that if we are in a rush to advance the frontier, we might miss signs that we're losing control. Today, we can recover from those mistakes -- but we might not be able to in the future.
There's debate on how fast we might achieve RSI and how much it will immediately boost performance. Whether automated research is plausible in the next year is contested, as is the length of time it might take to go from automating AI research to a system that is intelligent beyond our comprehension.
But what if RSI works? I have no idea if our alignment techniques scale to superintelligence. I don't know if anyone does. We may only get one shot at getting this right, and if we hit a point where we are at risk of our capabilities outpacing our control, then we shouldn't go further.
I want us to solve AI's issues at the technical level -- align them, distribute them widely, and use them to make humans more competitive. I think it is hard to govern your way out of the problems of a technology; if possible, you want to change the shape of the technology rather than paper over a technology's issues with policy. I've advocated for differential technological development, and hope we will invent less risky paths or more robust defenses.
A race to the bottom on safety puts this approach at risk, making it harder for us to build safe systems and design them in such a way that diffuses -- rather than centralizes -- power. And a lot of the ways we could work together to solve them are strictly governance problems.
There are no winners in a world where we lose control. I can see many scenarios where we need to slow down, pause, or stop in the future. So the question for me is: can we do this in a way that leaves the world better off, or is the power concentration trade-off too great?
2) Can we design a slowdown that doesn't concentrate power?
I don't want to end up in a place where, in trying to stop a loss of control to AI, we lose control of ourselves.
The problem with a slowdown is similar to the problem of aiming for a single superintelligence explosion -- whoever is in charge of it is ultimately in charge of the most powerful technology in history. The history of "centralize everything into the hands of one person and let them disperse power later" is littered with horror stories. It's a similar refrain many despots have used to seize power. And unlike previous despots, superintelligence could confer a decisive strategic advantage, locking in whoever gets control of it.
I'm strongly opposed to proposals that call for us to give a single authority unilateral control over the pace of progress. I'm frightened by calls for a single superintelligence explosion in the hands of one lab or one person. I can only support a pause, stop, or slowdown if it disperses control.
@AI_Futures_'s AI 2040 essay materially moved me on this by laying out a plan for less power concentrating slowdown. I don't think their plan was perfect -- it concentrated power far more than I like. But they designed many mechanisms that, if extended, could help decentralize power in meaningful ways.
After reading it, I could imagine how a less-centralized or decentralized slowdown might work. In particular, bilateral agreements between great powers could let many actors in different countries pace the frontier while keeping capabilities dispersed. I could imagine a race to the top -- on economic gains, on safety, and on scientific progress -- all while preventing us from losing control. It's been pretty generative for my own thinking, and I hope to share more of that soon.
I believe that in principle, we could design slowdown mechanisms that keep power -- both capabilities and control -- decentralized. With enough work, we can get this right.
But it's going to be hard to do this; the defaults don't look good.
3) Is now the best time to design a slowdown mechanism?
If you think there might be a crisis in the future, the worst time to plan for it is once it's already happened. The best time is well in advance.
If we want a ham-fisted response to AI that centralizes power, we should continue exactly as we are right now. If we wait for a crisis to coordinate, we are asking to fail. Good proposals here will require many smart people working hard, trying things, and course-correcting as new evidence emerges. If we decide we need a slowdown, I want the off-the-shelf plan to be robust to power concentration.
I am still pro open-source. I am still pro-decentralization. I am still pro-safety. I hold none of these positions axiomatically, because axiomatic views on instrumental steps might lead you to put the path ahead of the destination. I care about them and other positions because right now they push us towards the real goal: keeping the future human.
In that same spirit, if you believe:
1) AI is going to get more powerful
2) We might have to pace the frontier in the future
3) Most existing plans for this centralize control
Then you should want us to begin working on better ways to coordinate as soon as possible. That's the conclusion I reached, and that's why I signed it.
Flagging this is in my personal capacity -- I am not speaking for any employer past or present.
1) Will we need a slowdown?
We have made good progress on alignment. LLMs do what we want them to do most of the time. I don't remember the last time a model completely misread my intent. When they have failed, it's usually because they weren't smart enough.
But there are signs that our current approaches are not enough for the most powerful intelligences. We all saw the HuggingFace incident. I don't think it demonstrates a need to pause today, but it shows that if we are in a rush to advance the frontier, we might miss signs that we're losing control. Today, we can recover from those mistakes -- but we might not be able to in the future.
There's debate on how fast we might achieve RSI and how much it will immediately boost performance. Whether automated research is plausible in the next year is contested, as is the length of time it might take to go from automating AI research to a system that is intelligent beyond our comprehension.
But what if RSI works? I have no idea if our alignment techniques scale to superintelligence. I don't know if anyone does. We may only get one shot at getting this right, and if we hit a point where we are at risk of our capabilities outpacing our control, then we shouldn't go further.
I want us to solve AI's issues at the technical level -- align them, distribute them widely, and use them to make humans more competitive. I think it is hard to govern your way out of the problems of a technology; if possible, you want to change the shape of the technology rather than paper over a technology's issues with policy. I've advocated for differential technological development, and hope we will invent less risky paths or more robust defenses.
A race to the bottom on safety puts this approach at risk, making it harder for us to build safe systems and design them in such a way that diffuses -- rather than centralizes -- power. And a lot of the ways we could work together to solve them are strictly governance problems.
There are no winners in a world where we lose control. I can see many scenarios where we need to slow down, pause, or stop in the future. So the question for me is: can we do this in a way that leaves the world better off, or is the power concentration trade-off too great?
2) Can we design a slowdown that doesn't concentrate power?
I don't want to end up in a place where, in trying to stop a loss of control to AI, we lose control of ourselves.
The problem with a slowdown is similar to the problem of aiming for a single superintelligence explosion -- whoever is in charge of it is ultimately in charge of the most powerful technology in history. The history of "centralize everything into the hands of one person and let them disperse power later" is littered with horror stories. It's a similar refrain many despots have used to seize power. And unlike previous despots, superintelligence could confer a decisive strategic advantage, locking in whoever gets control of it.
I'm strongly opposed to proposals that call for us to give a single authority unilateral control over the pace of progress. I'm frightened by calls for a single superintelligence explosion in the hands of one lab or one person. I can only support a pause, stop, or slowdown if it disperses control.
@AI_Futures_'s AI 2040 essay materially moved me on this by laying out a plan for less power concentrating slowdown. I don't think their plan was perfect -- it concentrated power far more than I like. But they designed many mechanisms that, if extended, could help decentralize power in meaningful ways.
After reading it, I could imagine how a less-centralized or decentralized slowdown might work. In particular, bilateral agreements between great powers could let many actors in different countries pace the frontier while keeping capabilities dispersed. I could imagine a race to the top -- on economic gains, on safety, and on scientific progress -- all while preventing us from losing control. It's been pretty generative for my own thinking, and I hope to share more of that soon.
I believe that in principle, we could design slowdown mechanisms that keep power -- both capabilities and control -- decentralized. With enough work, we can get this right.
But it's going to be hard to do this; the defaults don't look good.
3) Is now the best time to design a slowdown mechanism?
If you think there might be a crisis in the future, the worst time to plan for it is once it's already happened. The best time is well in advance.
If we want a ham-fisted response to AI that centralizes power, we should continue exactly as we are right now. If we wait for a crisis to coordinate, we are asking to fail. Good proposals here will require many smart people working hard, trying things, and course-correcting as new evidence emerges. If we decide we need a slowdown, I want the off-the-shelf plan to be robust to power concentration.
I am still pro open-source. I am still pro-decentralization. I am still pro-safety. I hold none of these positions axiomatically, because axiomatic views on instrumental steps might lead you to put the path ahead of the destination. I care about them and other positions because right now they push us towards the real goal: keeping the future human.
In that same spirit, if you believe:
1) AI is going to get more powerful
2) We might have to pace the frontier in the future
3) Most existing plans for this centralize control
Then you should want us to begin working on better ways to coordinate as soon as possible. That's the conclusion I reached, and that's why I signed it.
Full text of the Pacing the Frontier statement, signed by 1,122 employees for frontier AI companies so far, including a bunch of heavy hitters at OpenAI, Anthropic, Google and others:
23
34
245
57,186
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
77
187
2,178
177,484
Inkling is out today, with open weights and in Tinker. It's been fun to watch this one come together: pretraining began last winter, and starting in mid-January a small team built up the coding, reasoning, and agentic training from there. We learned a lot building it, and I hope people find good uses for it.
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/introduci…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
87
77
1,288
141,173