we must know, we will know
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com
7
9
101
12,220
Dan Balsam retweeted
Excited to share this new result on EVEE, interp generated hypothesis actually reclassifying VUS. Want to share a bit about how this happened 🧵 (1/n)
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it ed as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
3
14
45
2,729
advancing science through interpretability, more to come soon. very proud of the team
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it ed as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
0
0
31
778
really interesting work
New paper: we ran a classic rat experiment on LLMs. We steer models positively or negatively as they read about one of two meaningless "zones." Despite identical tokens, they seek out positively steered zones and avoid negative ones. And given a lever, it shuts off bad states. 🧵
0
0
7
763
Dan Balsam retweeted
The craziest experiment in my lab is a new setup by @diatkinson where he has found a way induce introspection on open LMs using @dillonplunkett's protocol, that lets him look inside introspection. It's nuts.
Here's how the experiment works... 🧵 →
Here's how the experiment works... 🧵 →
6
24
227
13,321
Dan Balsam retweeted
@maksym_andr ...a judge with a rubric associated with it. Said another way, somebody's reward model is another person's monitor (and vice versa) already! What we need to figure out is dynamics of obfuscation and how to avoid them--not just brush the problem under the rug.
1
2
20
524
proud of the team for shipping here. many of the best benefits from AI will come from advances in the life sciences, but also some of the most important risks to mitigate. this is a step forward in allowing legitimate biological research to develop unimpeded while blocking potentially hazardous work
Biosecurity is the next frontier of AI security. We built SOTA monitors so agents can do more biology, safely.
Our monitors outperform frontier model safeguards with fewer refusals on dual-use tasks. They’re fast, real-time, and robust to adversarial attacks. 🧵
0
0
16
775
solving alignment is a major challenge which will require the world’s brightest minds, but i believe we can and must succeed
Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there. x.com/i/article/2105…
0
1
17
800
models know when they’re reward hacking, we can use interp to detect this
Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied!
We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵
0
0
28
827
this type of access is very important. we need independent technical organizations to evaluate models, and to make sure the best possible monitoring and safeguard systems are deployed where they need to be.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must-p…
2
1
19
1,553
interp is ultimately a verifiable task. one important thing many people are missing is how effective agents w/ interp tools will become
"Why am I optimistic about interpretability? It's partly because I think we're starting to have really good traction - but also because I can imagine this incredible speedup."
Catch our Chief Scientist @banburismus_ on the latest episode of @MLStreetTalk!
1
2
48
4,248
relatedly more powerful open models need great inference time safeguards to prevent both misuse and rogue agents. at Goodfire we've trained activation monitors for risks like offensive cyber, CBRNe, and reward hacking on trillion parameter models with nearly no inference overhead. we are working on partnering with inference providers to strengthen the open ecosystems security
Neoclouds have limited cybersecurity. Next time agents successfully go rouge, they'll try taking over a neocloud to run more copies. This is bad.
Thus: neoclouds should greatly strengthen their cybersecurity and every company with strong cyber models should help with that.
4
6
24
3,160
more from us there soon.
in the mean time more on the efficiency and effectiveness of activation monitors here: anthropic.com/research/next-…
in the mean time more on the efficiency and effectiveness of activation monitors here: anthropic.com/research/next-…
1
0
3
244
and more on our thoughts about this and related problems: x.com/DanJBalsam/sta…
Often I meet seasoned engineers who want to pivot into AI safety *research*. I think this is a mistake: safety is now a production problem and the world needs great engineers more than ever.
I wrote a short post expanding on this:
0
0
3
199
Often I meet seasoned engineers who want to pivot into AI safety *research*. I think this is a mistake: safety is now a production problem and the world needs great engineers more than ever.
I wrote a short post expanding on this:
I wrote a short post expanding on this:
people on here are thinkers so they assume that reasoning about alignment is the hard part and making sure millions(1) of task types, environments, and their respective virtual machines are configured correctly is the easy part but it’s essentially the opposite. you need to have worked in a large technology organization to appreciate the complexity and unreliability of this intuitively
(1) made up order of magnitude
9
18
186
29,737
Link: goodfire.com/blog/ai-safety…
If you’re a mid-career engineer getting into AI safety, you should strongly consider applying to engineering jobs working in AI safety, including at @GoodfireAI.
If you’re a mid-career engineer getting into AI safety, you should strongly consider applying to engineering jobs working in AI safety, including at @GoodfireAI.
4
0
36
2,697
@GoodfireAI The title of this essay is a reference to Andy Jones’ post from 2021: lesswrong.com/posts/YDF7XhMT…
I think it’s even more true today. Safety needs to scale.
I think it’s even more true today. Safety needs to scale.
0
0
18
1,228