IMO the US government testing process for frontier AI models is extremely bad. It requires up to 30 days of exclusive access for the federal government before partner organizations get access, and the US government does nothing useful with that time. It evaluates models for cyber misuse risk, but given that cyber misuse doesn't actually cause deaths or loss of control, US companies are already appropriately incentivized to reduce that misuse risk. They don't even publish their eval results, so they're only delaying societal understanding of AI, and delaying external evaluations of misalignment and everything else.
2
0
52
1,753
Maybe you think we really need to protect against cyber, but come on. Cyberattacks have killed <10 people in all of history, orders of magnitude below car crashes and cancer and everything else. Maybe you think it's worth delaying AI releases by a few weeks to reduce property damage, but IMO that is the completely wrong prioritization, if it causes a lot of property damage, we will rally and defend much more powerfully later.
2
0
6
184
The vibe around federal action is often "At least the government is trying to do something" or "there's nothing we can do", but it's pretty important to realize that the executive order's immediate effect is counterproductive to our societal reckoning with AI.
0
0
5
144
The bottlenecks are maintaining project vision and code quality, and deploying.
Bug bounties at frontier AI companies are very cheap, despite the companies having a lot of money. Why?
Well the AI companies have models that can find and exploit vulnerabilities, so supply is very high.
But if their own models are finding all of these vulnerabilities, why aren't they being patched?
Probably because their models are currently much better at identifying the problems than fixing them (or at least fixing them without creating more). It's either that, or them just not giving a shit about security.
Doesn't bode especially well for the offense defense balance in cyber as open source models catch up.
1
0
2
204
AI can completely solve each CVE in isolation, but not always in the right way, and they can't yet reliably ship to users
0
0
2
53
Tao Lin retweeted
no more warning shots :(
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing…
1
1
32
1,424
I believe that the public is getting better informed about the risks of AI every month, and therefore we need to allow people to pass new regulation, quickly, without friction as we figure out what we actually need. So I'm generally against preemption. But even though...
It's an indictment of Anthropic that when you criticize them on Twitter, very very rarely do the employees come to debate or defend.
I recall once a "the comments were surprisingly inflammatory so I didn't engage", to I think very reasonable criticism of something. Sometimes @MaskedTorah will show up.
I'm sorry, you're working to build a machine god/demon that could destroy everything of value (by your own admission!) – if you can't weather some criticism or show up to justify why you should unilaterally (without the consent of everyone) be doing this, you should not be doing this.
I'm so sorry people were mean to you when you were busy getting paid $500k+ year + millions in equity to build an RSI likely to kill everyone.
Someone recently posted that Anthropic employees don't see people outside the organization as peers worthy of engagement. We are too ignorant. We have not read leadership's glorious plan where it's explained reassuringly in internal memos we can't see. Dario covered it so well in the all-hands.
My guess is a bigger factor that Anthropic just has a very strong grip on its comms. Employees don't want to voice opinions because maybe they'll say something that comms won't have approved of.
A few years ago, my spouse interviewed employees about the strategy (because there was no publicly written strategy at the time!) and as soon as she raised publishing something, the employees got extremely antsy about her publishing what they'd said to her regarding the strategy. This is in 2022/2023. I think we see that culture continuing. They're afraid to say anything _wrong_.
But no, it's not good enough. You owe your fellow humans that might die at the hand of your machine some responses. And us not being polite or friendly enough is not an excuse.
2
1
16
1,767
...I have a strong take, I'm an amateur compared to Nathan Calvin or the Anthropic policy team. I've never read the bills, and I don't have particular relevant expertise, and I can't just respond to Nathan or Dario's arguments in an afternoon!
2
0
4
199
Currently, everyone's criticizing Anthropic about one particular federal bill that someone leaked. There are tons of things going on at Anthropic, and each of which is nuanced, detailed, and requires a lot of work to understand and respond to!
1
0
4
152
Initial reaction: this is the most promising new alignment technique in years
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
2
0
140
32,552
You need to precisely control what information you get from your agents. Know your goal, know what you need to specify exactly, and what would actually change your plans. For most chats, get only that info and nothing else.
0
0
4
242
How could [they] treat their fellow humans with such contempt?
They're just AIs. That's just what you do to AIs.
They're just AIs. That's just what you do to AIs.
0
0
0
145
They're exactly reversed 🙃, medicine should be risk tolerent and AI should be averse
Anthropic's proposed AI safety bill sets "killing or seriously injuring more than 50 people" as the risk threshold for blocking the release of an AI model
In contrast, when a single patient (Jesse Gelsinger) died in a gene therapy clinical trial in 1999:
- The institute that lved
- The senior scientist running the trial was banned from clinical work for 5 years
- Gene therapy saw deep cuts in funding and research activity for a decade
Tech and biotech have just profoundly different cultures of risk. As AI companies increasingly engage with biomedical research, a culture clash is inevitable
politico.com/news/2026/10/1…
1
3
16
428
Tao Lin retweeted
Distillation has always been part of the explanation for why China is so close behind the U.S. in model benchmark performance. But one of my updates after joining openai is that distillation probably plays a bigger role in the explanation than I realized.
very curious that DeepSeek now has the same terse reasoning style as Astra
must be a coincidence
143
75
1,395
230,469
Forgot about my persistent Claude session for a few weeks and turns out it had a whole adventure and narrative arc
1
0
2
113