Pinned Tweet
Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
“Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
“Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
51
116
879
172,989
I didn't initially notice this paragraph from OpenAI's September post on "Building standards for the next phase of AI."
Its a good sentiment, even though "Fully autonomous RSI" seems like too high a bar - heightened loss of control risks would happen well before that point. vocation of democratic principles has been much more frequent than its follow through on operationalization of those principles, but I like them at least saying that "informed democratic choices" should be a prerequisite to proceeding with certain AI development paths.
A very good start for OpenAI and other companies to operationalize this oversight would be share their current compute allocations being spent on internal AI R&D vs external inference vs security hardening etc.
Its a good sentiment, even though "Fully autonomous RSI" seems like too high a bar - heightened loss of control risks would happen well before that point. vocation of democratic principles has been much more frequent than its follow through on operationalization of those principles, but I like them at least saying that "informed democratic choices" should be a prerequisite to proceeding with certain AI development paths.
A very good start for OpenAI and other companies to operationalize this oversight would be share their current compute allocations being spent on internal AI R&D vs external inference vs security hardening etc.
1
6
38
3,974
Nathan Calvin retweeted
We obviously haven’t had the major jobs wipeout yet. But you read stuff like this. And you read what economics students themselves are saying about working with AI (ht @odavis_h) and it’s hard not to feel like the seismic shift is on its way.
He goes so far as to say that if he were to run the White House CEA today he'd do it with a lot of reliance on Claude and friends.
138
150
2,486
400,190
Was trying to get Astra to explain to me how this would work and how crazy it is
"It's a little like two prisoners communicating through a wall by tapping, except instead of audible taps, they're using nanosecond-scale differences in processor performance."
"It's a little like two prisoners communicating through a wall by tapping, except instead of audible taps, they're using nanosecond-scale differences in processor performance."
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
5
19
121
9,225
Nathan Calvin retweeted
I think a lot of people who have become worried about AI risk in the last few months are still really sleeping on BlueDot as a place to get up to speed fast. Got this nice message a few days ago
4
10
189
4,765
Nathan Calvin retweeted
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing…
229
322
4,386
476,417
Google forcing AI researchers at GDM to sign non-disparagement clauses (that last in perpetuity!) is truly appalling.
I hope that Google immediately discontinues this practice and releases prior employees who have signed from those obligations.
I hope that Google immediately discontinues this practice and releases prior employees who have signed from those obligations.
Important, and I'm glad this is finally public. Google DeepMind researchers are under in perpetuity non-disparagement clauses forbidding them from ever speaking critically of the company. I'd already confirmed this with multiple people.
Yes, some ex-employees have spoken out nonetheless. But it has also discouraged others from doing so. And yes, it most certainly has a chilling effect with regards to discussions on safety and risk.
Others have written compellingly about how important the ability to speak freely and critically is, especially on topics like safety and risk. And what a testament to the research employee community's coordination and hard-won bargaining power it has been to get there at other companies.
GDM burn that commons by having this. And everyone who signs burns that commons too. Let's get it changed.
4
9
111
3,149
Nathan Calvin retweeted
Geoffrey’s posts are always worth reading, but this one is a must-read. It lays out a really clear high-level hierarchy of safety levels, separated by which standard of rigor they adhere to, from “slapdash empirics” to “unconditional proofs”. The AI industry is unfortunately currently operating at the bottom level.
I think it is really important for the public to understand that it is within society’s power demand the higher levels! Not everything needs formal. verification and mathematical proofs, but
1. It is reasonable to demand that for the riskiest technologies, and
2. There are intermediate standards of rigor that are way better than what we have now.
I think it is really important for the public to understand that it is within society’s power demand the higher levels! Not everything needs formal. verification and mathematical proofs, but
1. It is reasonable to demand that for the riskiest technologies, and
2. There are intermediate standards of rigor that are way better than what we have now.
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
1
10
101
7,201
PSA, AI phishing scam on X
This account (@Futana6) messaged me saying they work at Tech Crunch and asking if I want to be interviewed with a personalized message (I assume to send over an interview schedule link that is actually a phishing link).
The account has 3,900 tweet about Tech Crunch which makes it more sophisticated than some other versions of this scam I saw before, but when I Google their name I find no evidence they work there and Pangram classifies their message as 100% AI.
Please ban them and do a better job cracking down on these sorts of scams @Support
This account (@Futana6) messaged me saying they work at Tech Crunch and asking if I want to be interviewed with a personalized message (I assume to send over an interview schedule link that is actually a phishing link).
The account has 3,900 tweet about Tech Crunch which makes it more sophisticated than some other versions of this scam I saw before, but when I Google their name I find no evidence they work there and Pangram classifies their message as 100% AI.
Please ban them and do a better job cracking down on these sorts of scams @Support
6
3
33
2,067
Nathan Calvin retweeted
70% of the internet was vulnerable yesterday to very bad security flaws inside a coding language. Turns out, *one volunteer* is mostly responsible for maintaining that code & helping push updates. Yesterday Ondřej Surý saved the internet. So I called him up. He rules. theatlantic.com/technology/202…
10
316
1,139
67,249
yeah as others have said, two different trends around AI + cyber are happening at the same time and its confusing:
AI helping cybersecurity situation:
- 5.6 sol (yay daybreak) discovering this PHP vulnerability that was already there so people can patch it (of course can be worse in short term while lots of systems go unpatched, and also long tail of legacy unpatched systems)
AI not helping cybersecurity situation:
- lots of AI powered automated cyberattacks (e.g. railway and opencode hacks)
Vulnerability discovery probably on average favors defenders in the medium term. Automating cyber operations will help attackers a lot though, and probably enough to overwhelm positive effects on the former (every random community bank or old person is not gonna have AI swarms protecting them against AI swarm attacks). Of course the cycle of patching vulnerabilities is becoming a lot harder because of how fast automated cyberattacks are.
Also an aside on AI spear fishing in particular -
Literally today I had one of those fake tech crunch reporters reach out over DMs and the fake was pretty convincing (thousands of followers, tweets and photos about tech crunch, but when I googled became clear person didn't exist).
People who are not used to the idea that people can lie in convincing ways on the internet are going to get exploited over and over again and its gonna be bad (and also generally make it way harder to have good interactions with humans you don't know online).
AI helping cybersecurity situation:
- 5.6 sol (yay daybreak) discovering this PHP vulnerability that was already there so people can patch it (of course can be worse in short term while lots of systems go unpatched, and also long tail of legacy unpatched systems)
AI not helping cybersecurity situation:
- lots of AI powered automated cyberattacks (e.g. railway and opencode hacks)
Vulnerability discovery probably on average favors defenders in the medium term. Automating cyber operations will help attackers a lot though, and probably enough to overwhelm positive effects on the former (every random community bank or old person is not gonna have AI swarms protecting them against AI swarm attacks). Of course the cycle of patching vulnerabilities is becoming a lot harder because of how fast automated cyberattacks are.
Also an aside on AI spear fishing in particular -
Literally today I had one of those fake tech crunch reporters reach out over DMs and the fake was pretty convincing (thousands of followers, tweets and photos about tech crunch, but when I googled became clear person didn't exist).
People who are not used to the idea that people can lie in convincing ways on the internet are going to get exploited over and over again and its gonna be bad (and also generally make it way harder to have good interactions with humans you don't know online).
A day in which a vulnerability is discovered by security researchers using defensive AI is certainly not the "worst day" by any means.
But I admit I have huge uncertainty as to how the near future will unfold, and am worried, not so much from novel vulnerability discovery as by attack automation.
3
5
39
11,047
(1) A lot of the current AI cybersecurity risks probably were gonna happen no matter what Anthropic did and many of Anthropic's prior warnings do look prescient there (though Anthropic having poor protections against distillation despite being warned did accelerate the capabilities on some of the Chinese open-weight models that are being used in current attacks)
(2) This sort of sentiment does not apply to RSI risks, where Anthropic really is intensifying the race and is counterfactually contributing to risks much more (and where saying "it was always going to be this way" is much less credible)
(3) Regardless of the merits, as a comms approach, I do not predict that this angle (it was inevitable, its not our fault) is going to resonate with the general public as people blame AI companies for AI related harms
(2) This sort of sentiment does not apply to RSI risks, where Anthropic really is intensifying the race and is counterfactually contributing to risks much more (and where saying "it was always going to be this way" is much less credible)
(3) Regardless of the merits, as a comms approach, I do not predict that this angle (it was inevitable, its not our fault) is going to resonate with the general public as people blame AI companies for AI related harms
quite the statement from Anthropic's head of security
5
17
129
12,519
It is striking the extent to which after reading this WSJ article - an article that OpenAI commented on repeatedly and from three reporters who are deeply sourced within OpenAI - I still don't really know what evidence OpenAI has that merited these firings.
Put yourself in the rking on AI safety at OpenAI.
After reading this article would you have a clear sense of what lines these employees crossed that merited their firing?
Or not knowing the exact lines, would you generally pull back from interactions with third parties that could later be cited as evidence of disloyalty?
Put yourself in the rking on AI safety at OpenAI.
After reading this article would you have a clear sense of what lines these employees crossed that merited their firing?
Or not knowing the exact lines, would you generally pull back from interactions with third parties that could later be cited as evidence of disloyalty?
Grateful to @ZeffMax, @keachhagey and @berber_jin1 for this careful reporting.
@tomekkorbak, @balesni and @j_asminewang pushed OpenAI to keep AI’s reasoning visible. OpenAI’s reasons for firing them are no clearer after this article than they were at 2pm on the day it fired them. But the issues they were raising are more important than ever. Proud @psst_org has been able to support them.
8
15
124
6,607
Nathan Calvin retweeted
This was one of the third-party engagements I worked on at OpenAI. I'm glad the report is out! I'm disappointed that METR wasn't allowed to comment on the scale of redactions made by OpenAI and that there is no explanation why the publication was delayed for 2.5 months.
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
11
28
471
17,430
The WSJ says OpenAI is searching employees personal phones and looking through their Signal messages
The firings of three OpenAI safety researchers came in the throes of a battle over how to keep AI systems under human control—and who gets to see the secret designs that could determine the winners of the biggest commercial race in history. on.wsj.com/48bYPG8
10
14
178
5,734
Uh, what happens in 18 months? Is Anthropic's view that one way or another, we aren't going to be worrying about cybersecurity anymore 😅
as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)
14
7
167
14,685
RT @deanwball: Everyone should be looking for data points to advance the longstanding debate in AI policy between “diffuse dual-use capabil…
0
19
0
31
"OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted""
concerning
concerning
I feel like this isn't getting enough attention:
- there were some pretty big holes found by the red-teamers, e.g. agents could just keep acting and not let the user do stuff and this would prevent monitors from being run (also misalignment monitors never ran on actions that were compacted away)
- The terms of this were "we will red-team for you" not "we will inform the public", so OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted" (or "everything would be fine if you saw what was redacted")
- The version of the report shared with OpenAI was 61 pages, and the version shared with METR/Redwood leadership after redaction was 38 pages, so 23 pages got redacted.
x.com/METR_Evals/sta…
5
11
121
5,066