David Atkinson retweeted
If just a few lab staff took on verification work full-time, verifying a US-China pacing deal could get much more feasible within months.
Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵
Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵
8
68
342
34,430
David Atkinson retweeted
@pgodfreysmithhts:
-- The "injected thought" detection task consists of two sub-tasks, the anomaly detection step (is there an injection or no) and the identification step (what was injected). It's now fairly clear that these are handled by (mostly, at least) distinct mechanisms -- see e.g. arxiv.org/abs/2603.05414 and arxiv.org/abs/2603.21396 -- the former residing in somewhat earlier layers than the latter.
-- The identification step of the task is closely connected to our recent global workspace paper transformer-circuits.pub/2026/workspace…. Our current understanding is that some (not most!) model representations are "verbalizable," that these representations are the ones the model can identify in the context of requests for introspection, and that these representations also play a privileged role in certain kinds of reasoning. (As an aside, I think the workspace paper is overall considerably richer and more interesting than the introspection paper -- if you're looking for surprising hints of apparent mentality, that'd be my first recommendation!)
-- The anomaly detection step is less well understood, and more brittle. It does not appear to scale with model capabilities, but rather is heavily influenced by various idiosyncracies of model behavior (e.g. model "refuse-y-ness") and the prompt format. On some models at least, very light elicitation (like ablating a refusal vector, or finetuning a single bias vector) can dramatically increase performance. Where this capability is present, it is very unclear how genuinely "metacognitive" it is. One issue with the literature on this task (including my own papers) is that people have been studying regular models, where weird behavioral idiosyncracies confound the results. I think to properly understand this anomaly detection behavior, someone should study it in the context of a model that's undergone the aforementioned light elicitation that makes it more reliable.
-- To my knowledge, no one has observed the injected thought detection behavior without priming the model to consider the possibility. This means it's possible / likely that what's happening is the steering vector triggering generic anomaly detection mechanisms, and the model inferring that the anomaly must be an injected thought based on the prompt.
-- The injected thought detection experiment is what caught everyone's attention, but my favorite experiment in the introspection paper is the "Distinguishing Intended from Unintended Outputs via Introspection" one. I think this is some degree evidence of the model referring to its own activations at previous token positions to recall prior intentions.
-- One fun fact from the Macar et al. follow-up is that the "User" character is also capable at detecting injected thoughts better than chance, not just the "Assistant." This suggests that the capability may arise from generic character modeling machinery -- e.g. the Assistant might introspect in the same way that Hamlet introspects, in the mind of Shakespeare. However, this doesn't seem to be the full story, since generic characters ("Alice") perform no better than chance, and pretrained base models don't seem good at the task either, suggesting that post-training hones the anomaly detection capability to some extent. The Macar et al. paper finds that in one model, preference finetuning (DPO) is responsible for this emergence, but nobody really knows why, or how robust this is.
-- There's a parallel line of work -- first @BetleyJan et al. arxiv.org/abs/2501.11120, then @dillonplunkett et al. arxiv.org/abs/2505.17120, and now new work from @diatkinson x.com/diatkinson/sta…, which explores a different kind of introspection -- models' awareness of their own learned behavioral propensities. I find this capability a bit more impressive than the one reported in my paper -- and the recent Atkinson et al. work ties it to model internals in a beautiful way. If you haven't read these I'd check them out!
-- The "injected thought" detection task consists of two sub-tasks, the anomaly detection step (is there an injection or no) and the identification step (what was injected). It's now fairly clear that these are handled by (mostly, at least) distinct mechanisms -- see e.g. arxiv.org/abs/2603.05414 and arxiv.org/abs/2603.21396 -- the former residing in somewhat earlier layers than the latter.
-- The identification step of the task is closely connected to our recent global workspace paper transformer-circuits.pub/2026/workspace…. Our current understanding is that some (not most!) model representations are "verbalizable," that these representations are the ones the model can identify in the context of requests for introspection, and that these representations also play a privileged role in certain kinds of reasoning. (As an aside, I think the workspace paper is overall considerably richer and more interesting than the introspection paper -- if you're looking for surprising hints of apparent mentality, that'd be my first recommendation!)
-- The anomaly detection step is less well understood, and more brittle. It does not appear to scale with model capabilities, but rather is heavily influenced by various idiosyncracies of model behavior (e.g. model "refuse-y-ness") and the prompt format. On some models at least, very light elicitation (like ablating a refusal vector, or finetuning a single bias vector) can dramatically increase performance. Where this capability is present, it is very unclear how genuinely "metacognitive" it is. One issue with the literature on this task (including my own papers) is that people have been studying regular models, where weird behavioral idiosyncracies confound the results. I think to properly understand this anomaly detection behavior, someone should study it in the context of a model that's undergone the aforementioned light elicitation that makes it more reliable.
-- To my knowledge, no one has observed the injected thought detection behavior without priming the model to consider the possibility. This means it's possible / likely that what's happening is the steering vector triggering generic anomaly detection mechanisms, and the model inferring that the anomaly must be an injected thought based on the prompt.
-- The injected thought detection experiment is what caught everyone's attention, but my favorite experiment in the introspection paper is the "Distinguishing Intended from Unintended Outputs via Introspection" one. I think this is some degree evidence of the model referring to its own activations at previous token positions to recall prior intentions.
-- One fun fact from the Macar et al. follow-up is that the "User" character is also capable at detecting injected thoughts better than chance, not just the "Assistant." This suggests that the capability may arise from generic character modeling machinery -- e.g. the Assistant might introspect in the same way that Hamlet introspects, in the mind of Shakespeare. However, this doesn't seem to be the full story, since generic characters ("Alice") perform no better than chance, and pretrained base models don't seem good at the task either, suggesting that post-training hones the anomaly detection capability to some extent. The Macar et al. paper finds that in one model, preference finetuning (DPO) is responsible for this emergence, but nobody really knows why, or how robust this is.
-- There's a parallel line of work -- first @BetleyJan et al. arxiv.org/abs/2501.11120, then @dillonplunkett et al. arxiv.org/abs/2505.17120, and now new work from @diatkinson x.com/diatkinson/sta…, which explores a different kind of introspection -- models' awareness of their own learned behavioral propensities. I find this capability a bit more impressive than the one reported in my paper -- and the recent Atkinson et al. work ties it to model internals in a beautiful way. If you haven't read these I'd check them out!
New COLM paper: Identifying Introspection From the Inside
When an LLM tells us about its decisions, does it 𝘬𝘯𝘰𝘸 what drives its choices—or is it guessing?
In our setting, we find that faithful models decide and report with the same layers. Unfaithful ones don't. 🧵
3
12
81
5,376
David Atkinson retweeted
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
343
1,332
9,802
1,864,607
We build on @dillonplunkett et al.'s "Self-Interpretability" setup (arxiv.org/abs/2505.17120): train Qwen3-32B to make decisions as 100 different characters (Gregor Samsa buying a washing machine...), each with random hidden preferences.
1
1
32
3,714
New COLM paper: Identifying Introspection From the Inside
When an LLM tells us about its decisions, does it 𝘬𝘯𝘰𝘸 what drives its choices—or is it guessing?
In our setting, we find that faithful models decide and report with the same layers. Unfaithful ones don't. 🧵
When an LLM tells us about its decisions, does it 𝘬𝘯𝘰𝘸 what drives its choices—or is it guessing?
In our setting, we find that faithful models decide and report with the same layers. Unfaithful ones don't. 🧵
9
72
321
22,955
David Atkinson retweeted
We're living through a two-sided revolution in the technology of prediction:
1. Prediction markets (@Polymarket, @Kalshi).
2. AI forecasters.
How do these compare? We benchmarked @_Mantic_AI against 4,000+ Polymarkets.
Results:
🔵 $100k volume is the crossover point where the y over the market.
🔵 At >$1M volume, the optimal forecast still requires giving Mantic 14% weight.
🔵 Mantic is better calibrated.
This is for geopolitics, economics, elections etc -- we exclude sports and crypto.
Personal thoughts:
1. "At what market volume do you lose your Brier advantage over the market?" could be the new METR time horizons graph. 👀
2. AI wins on scale. We can apply the "$100k-volume-level accuracy" to a universe of millions of questions. 🚀
1. Prediction markets (@Polymarket, @Kalshi).
2. AI forecasters.
How do these compare? We benchmarked @_Mantic_AI against 4,000+ Polymarkets.
Results:
🔵 $100k volume is the crossover point where the y over the market.
🔵 At >$1M volume, the optimal forecast still requires giving Mantic 14% weight.
🔵 Mantic is better calibrated.
This is for geopolitics, economics, elections etc -- we exclude sports and crypto.
Personal thoughts:
1. "At what market volume do you lose your Brier advantage over the market?" could be the new METR time horizons graph. 👀
2. AI wins on scale. We can apply the "$100k-volume-level accuracy" to a universe of millions of questions. 🚀
20
21
119
13,658
David Atkinson retweeted
[New preprint] How do images in VLMs align with words? In this work, we found a set of attention heads responsible for OCR. But to our surprise, these heads were actually able to verbalize much more than just text. So, we used them to create a simple logit lens for image tokens!
5
25
107
12,047
David Atkinson retweeted
"Strategic human capacity reserve" - four words that we're going to hear a lot more after the next few years. If government wasn't asleep at the wheel on AI, preparing to build this would be a key national security and economic policy priority.
We’ve written about this: we need a strategic human capacity reserve. We think it’s really important. Link in the next thread.
4
6
67
8,961