Pinned Tweet
How does an AI lab prove that it's doing what it says its doing, and not something else without leaking sensitive IP.
We put an open-source LLM in a trusted computer (a TEE) in order to enable privacy-preserving auditing.
Take a look, and help us improve! [1/2]
We put an open-source LLM in a trusted computer (a TEE) in order to enable privacy-preserving auditing.
Take a look, and help us improve! [1/2]
4
5
35
4,178
Roy Rinberg retweeted
If just a few lab staff took on verification work full-time, verifying a US-China pacing deal could get much more feasible within months.
Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵
Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵
8
67
341
34,318
Roy Rinberg retweeted
I interviewed @tsmilton_amodo about the tech we'd need to verify a potential US-China deal on AI.
We discuss Amodo's prototype for inference verification, as well as the myriad ways you could use power monitoring, zero knowledge proofs and/or unilateral surveillance to sary is using compute.
Though solving some kind of verification is central to many people's theories of victory, there are precious few people working on it currently. Tom is one such person!
Listen to the episode on Youtube, Spotify or Apple Podcasts.
You can also find more about Tom and Amodo's work at
ai-verification.com
00:00:00 Using compute verification for an AI slowdown
00:06:08 Proving a datacentre is only running inference
00:13:22 Reading a GPU's workload from its power draw
00:23:35 What would it take to get verification deal-ready?
00:37:49 How realistic is verification?
00:46:07 Preparing for a rushed US-China deal
01:07:20 How good is AI at hardware design?
We discuss Amodo's prototype for inference verification, as well as the myriad ways you could use power monitoring, zero knowledge proofs and/or unilateral surveillance to sary is using compute.
Though solving some kind of verification is central to many people's theories of victory, there are precious few people working on it currently. Tom is one such person!
Listen to the episode on Youtube, Spotify or Apple Podcasts.
You can also find more about Tom and Amodo's work at
ai-verification.com
00:00:00 Using compute verification for an AI slowdown
00:06:08 Proving a datacentre is only running inference
00:13:22 Reading a GPU's workload from its power draw
00:23:35 What would it take to get verification deal-ready?
00:37:49 How realistic is verification?
00:46:07 Preparing for a rushed US-China deal
01:07:20 How good is AI at hardware design?
2
10
59
5,437
Roy Rinberg retweeted
My plea to all AI lab staff: reassess if your current project is your best opportunity to delay RSI, or stop it blowing up in our face.
If it's not, move immediately.
There's no time to waste even a week to inertia or politeness.
The strategic situation is shifting under our reassessed your options recently, you could easily be wasting your time or possibly making things worse.
You have access to info I don't, but here's 8 ways priorities seem to have shifted to me, which I elaborate on in the video:
1. Helping with capabilities advances looks worse than in the past, especially bringing forward RSI
2. Safety & alignment evals need a credibility revolution to stop misleading the public
3. 'Pacing the frontier' is unambiguously necessary, and working to make it happen is a huge new opportunity for talent
4. Chain-of-thought monitoring probably stops working without a big technical and regulatory push to keep it
5. It's worse than ever to patch surface indications of deep structural problems you know aren't getting fixed
6. There's ever less need for altruists to help with primarily commercial work
7. Staff have an ever-shrinking window to organise as a group and collectively shape events
8. There are new opportunities for research impact outside frontier companies that are worth considering
I know you're probably some combination of exhausted, overwhelmed, and anxious already.
But unfortunately there's no way to escape the need to regularly check in on the high level strategic picture and see where you can slot in to maximum effect.
I wish I could do it for you but nobody far from your situation can. Godspeed my friends.
(From the 80,000 Hours Podcast, links below.)
If it's not, move immediately.
There's no time to waste even a week to inertia or politeness.
The strategic situation is shifting under our reassessed your options recently, you could easily be wasting your time or possibly making things worse.
You have access to info I don't, but here's 8 ways priorities seem to have shifted to me, which I elaborate on in the video:
1. Helping with capabilities advances looks worse than in the past, especially bringing forward RSI
2. Safety & alignment evals need a credibility revolution to stop misleading the public
3. 'Pacing the frontier' is unambiguously necessary, and working to make it happen is a huge new opportunity for talent
4. Chain-of-thought monitoring probably stops working without a big technical and regulatory push to keep it
5. It's worse than ever to patch surface indications of deep structural problems you know aren't getting fixed
6. There's ever less need for altruists to help with primarily commercial work
7. Staff have an ever-shrinking window to organise as a group and collectively shape events
8. There are new opportunities for research impact outside frontier companies that are worth considering
I know you're probably some combination of exhausted, overwhelmed, and anxious already.
But unfortunately there's no way to escape the need to regularly check in on the high level strategic picture and see where you can slot in to maximum effect.
I wish I could do it for you but nobody far from your situation can. Godspeed my friends.
(From the 80,000 Hours Podcast, links below.)
20
74
483
21,776
Roy Rinberg retweeted
Effective Immediately, all organizations using PHP for their systems are advised to be prepared for an update addressing two critical vulnerabilities. PHP is widely used for web applications. These vulnerabilities often provide a direct path to the server's file system or database. If an immediate update is not feasible, it is recommend that organizations take the system offline until it can be fully mitigated. NSA will follow up shortly with more specific guidance.
1,295
10,828
70,240
12,626,108
Roy Rinberg retweeted
Went under the radar but Anthropic have posted updates to the "Moonshot R&D projects" in their Frontier Safety Roadmap.
They've developed a protoype that verifies that outputs come from a particular set of model weights - and rolled out signing on all internal token traffic!
They've developed a protoype that verifies that outputs come from a particular set of model weights - and rolled out signing on all internal token traffic!
6
17
299
20,526
Roy Rinberg retweeted
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
136
171
1,173
236,432
Roy Rinberg retweeted
What's going on with AI safety in China?
I've written a brief primer. TLDR: there is concrete frontier safety engagement, but the field is small (~<100 full-time people, $20M/yr funding).
As more of us realize the need for 'international buy-in on 'frontier pacing', consider helon!
I've written a brief primer. TLDR: there is concrete frontier safety engagement, but the field is small (~<100 full-time people, $20M/yr funding).
As more of us realize the need for 'international buy-in on 'frontier pacing', consider helon!
5
31
261
11,175
Roy Rinberg retweeted
Neoclouds have limited cybersecurity. Next time agents successfully go rouge, they'll try taking over a neocloud to run more copies. This is bad.
Thus: neoclouds should greatly strengthen their cybersecurity and every company with strong cyber models should help with that.
Thus: neoclouds should greatly strengthen their cybersecurity and every company with strong cyber models should help with that.
655
987
12,388
2,277,475
Roy Rinberg retweeted
OpenAI's newest AI, Astra, is reported to use an 'opaque reasoning' architecture where more of the reasoning occurs in activations instead of natural language. This may be the single worst development for AI security/safety to date.
The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight.
My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here.
Some concrete examples of why this is extremely dangerous (inspired by recent events):
- In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought!
- The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.)
At a more basic level, my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan.
The public doesn't currently have enough information to assess exactly how bad the architecture of Astra is, though based on the discussion in the article, the direction it moves in is extremely concerning. OpenAI should release more information about Astra's architecture as well as how much this architecture degrades monitorability and increases the AI's ability to reason opaquely. Credible independent assessment (or verification of OpenAI's assessment) is also important.
It seems like we are now engaged in a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs. (While simultaneously, AI companies believe they'll train extremely capable AIs within a few years.) It may not be too late for AI companies and employees at AI companies to take aggressive action to avoid the worst outcomes.
The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight.
My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here.
Some concrete examples of why this is extremely dangerous (inspired by recent events):
- In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought!
- The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.)
At a more basic level, my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan.
The public doesn't currently have enough information to assess exactly how bad the architecture of Astra is, though based on the discussion in the article, the direction it moves in is extremely concerning. OpenAI should release more information about Astra's architecture as well as how much this architecture degrades monitorability and increases the AI's ability to reason opaquely. Credible independent assessment (or verification of OpenAI's assessment) is also important.
It seems like we are now engaged in a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs. (While simultaneously, AI companies believe they'll train extremely capable AIs within a few years.) It may not be too late for AI companies and employees at AI companies to take aggressive action to avoid the worst outcomes.
I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
111
273
1,693
168,262
Roy Rinberg retweeted
new: OpenAI & others quietly using loop transformers that don't show their 'thinking' when scaled up
a leap forward on performance, but sparking concerns inside & outside OpenAI re: security as this takes off
a leap forward on performance, but sparking concerns inside & outside OpenAI re: security as this takes off
35
54
730
523,895
Roy Rinberg retweeted
Overall, I’m very surprised at how little media coverage there’s been around the OpenAI / Hugging Face attack. It’s clearly one of the most important things to happen this year.
601
918
12,275
1,284,130
Roy Rinberg retweeted
I think it's fair to estimate this attack at significantly greater than 16hrs of human time
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
19
48
1,018
88,752
Roy Rinberg retweeted
FWIW, the OpenAI and METR reports are great examples of overcoming all of these org frictions to release substantial, potentially-pivotal public scientific evidence. I expect that required major internal will.
I've heard people worked >14-hour days to make it happen. They should feel intensely proud. I hope they see how impactful this may be in shifting public preparedness, and want to keep leaning into it. There will be more areas that will require expansive public evidence-sharing if we are to make it through.
I've heard people worked >14-hour days to make it happen. They should feel intensely proud. I hope they see how impactful this may be in shifting public preparedness, and want to keep leaning into it. There will be more areas that will require expansive public evidence-sharing if we are to make it through.
I am familiar with the intense frictions to publishing the necessary evidence from inside frontier AI labs. It would directly touch on sensitive IP (e.g. reasoning traces, training approaches, parameter estimates for takeoff-speed models[1]), and disclosures increase legal exposure. More broadly, sharing more of the secret sauce than necessary runs against the now-ingrained culture of how the industry thinks it should compete, even if most researchers agree we shouldn't compete on loss-of-control-prevention. Releasing blogposts with high-level claims, backed by lab researchers’ track records of credibility but not the full evidence, feels much less risky, even if it limits outsiders’ trust in the conclusions. That was indeed sufficient in an era where the stakes of scientific consensus were low.
The stakes are getting extremely high. A more powerful model swarm in any US company could correctly conclude that silent subversion of security and research infrastructure, self-exfiltration, resource acquisition, and establishment of rogue deployments across several ME, SEA, and CN datacenters are all likely to be useful for the swarm’s completion of future more complex RL tasks. If that happened, the US would lose its AI lead to China, humanity could durably lose its monopoly over cyberspace, and from hereon exist under perennial assault from an undeterrable digital North Korea, or worse.
6
23
212
51,878
Roy Rinberg retweeted
I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
295
1,062
6,627
1,990,367
Roy Rinberg retweeted
*We need all the scientific evidence on severe misalignment out in the open, urgently.*
TL;DR If severe misalignment is indeed emerging absent a high bar of alignment+control execution, the top priority actions for OpenAI and Anthropic in the next couple months should be to make public all necessary evidence for non-safety AI technical experts to become convinced of the science around loss-of-control themselves.
[the below is all my personal opinion and doesn't represent my employer]
Any meaningful solution to prevent loss-of-control-of-AI will require pan-US-AI-industry adherence to costly practices around AI control, alignment, and security. Crucially, this will include Meta, SpaceXAI, and even NVIDIA, and almost certainly require the blessing or coordination of the US government if it is to be internationalized. The current politics of AI make it unlikely that even OpenAI and Anthropic's joint technical consensus would be sufficient to motivate USG action so long as the lagging AI players are against it. Even if OpenAI and Anthropic were to go to the White House today with slam-dunk technical evidence of an imminent national security threat of AI loss-of-control absent intervention, the USG would probably not trust in their own in-house reasoning about that evidence, and would look to the views of the other admin-friendly AI players and advisors. These other actors appear largely skeptical.
The only solution is science.
TL;DR If severe misalignment is indeed emerging absent a high bar of alignment+control execution, the top priority actions for OpenAI and Anthropic in the next couple months should be to make public all necessary evidence for non-safety AI technical experts to become convinced of the science around loss-of-control themselves.
[the below is all my personal opinion and doesn't represent my employer]
Any meaningful solution to prevent loss-of-control-of-AI will require pan-US-AI-industry adherence to costly practices around AI control, alignment, and security. Crucially, this will include Meta, SpaceXAI, and even NVIDIA, and almost certainly require the blessing or coordination of the US government if it is to be internationalized. The current politics of AI make it unlikely that even OpenAI and Anthropic's joint technical consensus would be sufficient to motivate USG action so long as the lagging AI players are against it. Even if OpenAI and Anthropic were to go to the White House today with slam-dunk technical evidence of an imminent national security threat of AI loss-of-control absent intervention, the USG would probably not trust in their own in-house reasoning about that evidence, and would look to the views of the other admin-friendly AI players and advisors. These other actors appear largely skeptical.
The only solution is science.
16
60
390
47,554
Roy Rinberg retweeted
"But what's the alternative?"
The same stuff people have been pushing for for a while - good safety and security practices, auditing against those so we can have trust + implement any key requirements or slowdown that might be needed, not tolerating pervasive GPU smuggling, etc.
The same stuff people have been pushing for for a while - good safety and security practices, auditing against those so we can have trust + implement any key requirements or slowdown that might be needed, not tolerating pervasive GPU smuggling, etc.
3
1
37
2,491
Roy Rinberg retweeted
Yesterday, my OpenAI collaborator and I gave a detailed talk on the Huggingface incident, our models creating "the message board", model misalignment, and more.
youtube.com/watch?v=87DyyM…
I hope it can answer a lot of the questions folks have, and we will release a full detailed postmortem at a later time!
youtube.com/watch?v=87DyyM…
I hope it can answer a lot of the questions folks have, and we will release a full detailed postmortem at a later time!
176
496
3,454
2,405,952
Roy Rinberg retweeted
I wonder whether we will soon start to see faster AI self-improvement at OpenAI vs. Anthropic based on the former’s known deeper investment in RL, TTC, and math proving more useful for tasks related to AI R&D, and that this gap may grow significantly over the next 6 months.
(Obviously possible Ant is seeing similar results w/ internal models, but my weakly-held sense is that they’re not.)
If so, it seems *really* crucial for OAI to be able to correctly calibrate its relative position in the RSI ramp so it can incorporate it into its alignment+security decision-making, especially related to full RSI. The Allies rushed to a bomb on the incorrect assumption that the Axis was right on their tail, when in fact they were far behind and the bomb was plausibly unnecessary, purely due to a fog of war. We also saw something similar with the original strawberry results, where OpenAI felt an intense sense of urgency based on rumors that Anthropic+GDM were discovering it in parallel a couple months behind when in fact I’ve now heard they were ~9 months behind on it.
I expect OpenAI would act very differently wrt future concerning misalignment findings if they knew they were 3 months ahead. It would be very simple for the parties to resolve such an uncertainty.
Executing on a pace-info-sharing scheme probably not crucial today, but will be in a few months.
(Obviously possible Ant is seeing similar results w/ internal models, but my weakly-held sense is that they’re not.)
If so, it seems *really* crucial for OAI to be able to correctly calibrate its relative position in the RSI ramp so it can incorporate it into its alignment+security decision-making, especially related to full RSI. The Allies rushed to a bomb on the incorrect assumption that the Axis was right on their tail, when in fact they were far behind and the bomb was plausibly unnecessary, purely due to a fog of war. We also saw something similar with the original strawberry results, where OpenAI felt an intense sense of urgency based on rumors that Anthropic+GDM were discovering it in parallel a couple months behind when in fact I’ve now heard they were ~9 months behind on it.
I expect OpenAI would act very differently wrt future concerning misalignment findings if they knew they were 3 months ahead. It would be very simple for the parties to resolve such an uncertainty.
Executing on a pace-info-sharing scheme probably not crucial today, but will be in a few months.
3
11
180
20,011
Roy Rinberg retweeted
I’m incredibly excited that Dan Wattendorf is joining OpenAI Foundation as our Head of Bio-Resilience.
As AI capabilities advance, biology could become one of the most consequential areas - for both extraordinary benefits and serious risks. We need to build the institutions and technologies that will help the world prepare.
Dan has a rare combination of scientific judgment and the ability to make ambitious things happen. He helped pioneer DARPA’s early mRNA technology, served as a US Air Force colonel, and led the Gates Foundation’s Global Health Accelerator.
There is an enormous amount to build, and I’m very glad Dan will be leading the charge.
As AI capabilities advance, biology could become one of the most consequential areas - for both extraordinary benefits and serious risks. We need to build the institutions and technologies that will help the world prepare.
Dan has a rare combination of scientific judgment and the ability to make ambitious things happen. He helped pioneer DARPA’s early mRNA technology, served as a US Air Force colonel, and led the Gates Foundation’s Global Health Accelerator.
There is an enormous amount to build, and I’m very glad Dan will be leading the charge.
18
25
317
43,679