Daniel Murfet retweeted
Norbert Wiener, a forefather of control theory and arguably the first alignment researcher, warned in 1960 that we had better be sure “the purpose put into the machine is the purpose which we really desire.” New post: patterning as control (in the Toy Model of Superposition).
1
18
74
3,062
Daniel Murfet retweeted
A bit of advertising: I recently completed revisions for a tutorial paper titled "Geometric Functional Inequalities for Markov Processes: An Invitation via the Langevin Diffusion", which can now be found online at arxiv.org/abs/2611.07785.
7
107
755
27,075
Daniel Murfet retweeted
Why are many orgs aggressively hiring/scaling up, yet it's extremely hard to find a full time position?
Why are there calls to expand the number of people doing research in AI safety, yet our academic institutions are crumbling under the weight of the sheer number of papers and horted to do research slowly and deeply, yet sped along on crunch time, with the hope that scale will clarify our confusions?
In this new blogpost (done in my personal time), I argue that these are rooted in a tension between a frame of "war mode" and "civilisation mode". This tension, in my view, creates uneasiness and despair, as well as makes it difficult to see clearly.
Check it out on LW: lesswrong.com/posts/7e38jwHh…
or on Substack: davidafrica.substack.com/p/near-horizon…
Why are there calls to expand the number of people doing research in AI safety, yet our academic institutions are crumbling under the weight of the sheer number of papers and horted to do research slowly and deeply, yet sped along on crunch time, with the hope that scale will clarify our confusions?
In this new blogpost (done in my personal time), I argue that these are rooted in a tension between a frame of "war mode" and "civilisation mode". This tension, in my view, creates uneasiness and despair, as well as makes it difficult to see clearly.
Check it out on LW: lesswrong.com/posts/7e38jwHh…
or on Substack: davidafrica.substack.com/p/near-horizon…
11
26
261
25,493
Daniel Murfet retweeted
New paper: we generate Lean-grounded synthetic data to train verifiers that can find mistakes in natural language math proofs.
Broadly, I hope we can use Lean and formal-to-informal generalization as part of the solution to supervising superhuman models in the future.
Broadly, I hope we can use Lean and formal-to-informal generalization as part of the solution to supervising superhuman models in the future.
8
23
167
5,913
Daniel Murfet retweeted
I've tweeted before about how architecture improvements usually improve the constant factor in loss scaling laws, keeping the same exponent. The scaling ladders in the Ripple-65B release from @RicursiveAI are a nice illustration of this at scale! 🧵
5
10
89
5,602
Daniel Murfet retweeted
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gu…
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gu…
66
136
1,573
185,076
Daniel Murfet retweeted
"Assessing AI Consciousness" is now available as a book: hutter1.net/ai/bayesconsc.… It develops a Bayesian framework separating theoretical credence from empirical evidence, providing a precise, scalable method for calibrated assessments (with online tool and code): ai-cognition.org/cacophony-tool/ Happy Halloween.
13
84
379
18,616
Daniel Murfet retweeted
I’m really excited to read this! I’m a bit biased because I think @BuchananBen is a gem of a human and I really enjoyed working with him back at Georgetown/CSET. I also want to get to the bottom of something that’s been bugging me since I read @kevinroose's 'The AGI Chronicles', Struggle' covers it.
There are a few passages in 'The AGI Chronicles' that suggest Dario Amodei and Jack Clark played a role in shaping how the Commerce Dept thought about export controls (am attaching the pages here so folks know what I’m referring to).
In a nutshell, Roose suggests that Amodei and Clark sent a memo to Buchanan on export controls, and then Buchanan drafted his own proposal on export controls, which eventually became policy. The causation is fuzzy, and I think Roose is probably trying to avoid over-claiming how much influence Amodei and Clark had.
It matters a lot how policies get developed, and I think it would be great to hear more accounts from folks who were in the room about what went down. (Bonus if @jackclarkSF wants to write a book...)
There are a few passages in 'The AGI Chronicles' that suggest Dario Amodei and Jack Clark played a role in shaping how the Commerce Dept thought about export controls (am attaching the pages here so folks know what I’m referring to).
In a nutshell, Roose suggests that Amodei and Clark sent a memo to Buchanan on export controls, and then Buchanan drafted his own proposal on export controls, which eventually became policy. The causation is fuzzy, and I think Roose is probably trying to avoid over-claiming how much influence Amodei and Clark had.
It matters a lot how policies get developed, and I think it would be great to hear more accounts from folks who were in the room about what went down. (Bonus if @jackclarkSF wants to write a book...)
I led AI policy in the Biden administration, and my colleague @tantumscollins was Director for Technology on the National Security Council. Our book on AI and national security -- THE BITTER STRUGGLE -- comes out in 11 days. @tylercowen said it “will become the standard source” hina chip bans and Biden-era AI policy. @crmiller1 said it is “essential reading for understanding how AI will impact geopolitics and how the great powers are shaping AI.”
We wrote this book because the decisions shaping AI -- and the future for all of us -- are too important to only be debated behind closed doors. If you're planning on reading it, pre-ordering is one of the best ways to help us bring this conversation to a broader audience.
penguinrandomhouse.com/books/821154/t…
1
1
5
595
Daniel Murfet retweeted
This is so awesome and exciting!!! I'm really grateful to @sethlazar and the others involved. It's hard to create new infrastructure in academia (especially academic philosophy), and many hours of labour go into building and maintaining journals. Thank you for this!
0
3
27
1,559
Daniel Murfet retweeted
I agree with Geoffrey that even bringing AI up to the standards of more rigorous industries might not be enough. His examples are from engineering, but I'd say the same about governance.
Many proposals for more serious AI governance take finance as their model, calling for independent audits, bank-style supervision or an AI version of FINRA. I worked in AI in finance for a few years, and I understand the appeal. These approaches are certainly better than the status quo at AI companies. But we shouldn't treat them as sufficient.
The financial sector has had decades to build these controls, and they still fail. We've seen big failures, and there are many, many small ones that never reach the headlines, so the public record makes these mechanisms look more reliable than they are.
The risks in finance are also better understood and more predictable than the risks posed by AI systems that are new and change quickly, let alone systems that are, as Geoffrey says, good at being adversarial. These governance mechanisms would keep us safer than we currently are, but probably not safe.
Many proposals for more serious AI governance take finance as their model, calling for independent audits, bank-style supervision or an AI version of FINRA. I worked in AI in finance for a few years, and I understand the appeal. These approaches are certainly better than the status quo at AI companies. But we shouldn't treat them as sufficient.
The financial sector has had decades to build these controls, and they still fail. We've seen big failures, and there are many, many small ones that never reach the headlines, so the public record makes these mechanisms look more reliable than they are.
The risks in finance are also better understood and more predictable than the risks posed by AI systems that are new and change quickly, let alone systems that are, as Geoffrey says, good at being adversarial. These governance mechanisms would keep us safer than we currently are, but probably not safe.
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
4
3
18
1,455
Daniel Murfet retweeted
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
9
41
290
30,032
Daniel Murfet retweeted
Scalable oversight at Res is hiring! Come work alongside our excellent new TCS team, and newest SO team member @siddarthv66
I am happy to announce Resolution's theoretical computer science (TCS) leadership!
1. Omer Reingold is joining as TCS Technical Lead, alongside his work at Stanford. ❤️
2. Tal Rabin is joining as Board Advisor, and helping with team building and leadership within TCS. ❤️
6
4
40
2,430
Daniel Murfet retweeted
I am happy to announce Resolution's theoretical computer science (TCS) leadership!
1. Omer Reingold is joining as TCS Technical Lead, alongside his work at Stanford. ❤️
2. Tal Rabin is joining as Board Advisor, and helping with team building and leadership within TCS. ❤️
1. Omer Reingold is joining as TCS Technical Lead, alongside his work at Stanford. ❤️
2. Tal Rabin is joining as Board Advisor, and helping with team building and leadership within TCS. ❤️
2
12
151
10,465
Daniel Murfet retweeted
Thank you, @siddarthv66! I am very excited to work with you. ❤️
The coming wave of ex-capabilities switching to alignment and safety broadly will be great!
The coming wave of ex-capabilities switching to alignment and safety broadly will be great!
I've decided to start working on technical alignment research, specifically related to RL and how to fix it (or get rid of it if we must). I will be joining the brilliant folks @resolution_org in Berkeley, a very exciting new alignment bet by @geoffreyirving and others from @AISecurityInst and @TimaeusResearch, as part of the scalable oversight team. I intend to write more about why I'm switching to work on alignment, and why I think more capabilities researchers should really do this. But long story short:
1. We are not on track to solve alignment of superintelligence
2. We will likely have a version of RSI soon, accompanied soon by superintelligence
3. The capabilities recipe works for verifiable, unhackable signal, which the frontier is increasingly diverging from in the pursuit of growing capabilities. It is unclear what RSI means unless we find a scalable recipe to train with proxy rewards.
1
1
74
5,073
Daniel Murfet retweeted
Resolution is giving $1M to the Mathlib Initiative to strengthen human workflows and tooling for the arriving wave of AI-formalized mathematics. This grant is not tied to any Resolution-specific formalization project: the goal is general community health (for us and others!). 🧵
5
10
105
6,082
Daniel Murfet retweeted
Today we published the first in what will likely be a regular series of standalone alignment reports on our models’ behaviors. Beyond increasing transparency, we hope they’ll convey how we think about alignment and situate the behaviors we observe in that broader context.
We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports.
Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping.
All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September.
Read the full report: anthropic.com/research/inves…
8
11
161
7,310