Unfortunately, I expect Anthropic is close enough to RSI that AI stocks crashing—or even NVIDIA stopping new sales—would not on its own postpone the death of all humans by very much. I hope I am wrong.
Jensen was conducting an AI safety master plan all along.
1. Make the entire AI economy dependent on Nvidia’s stock price
2. Have an affair with a Chinese spy
3. Resign, tanking Nvidia’s stock price because you have no successor
4. The AI investment bubble implodes, slowing down the datacenter buildouts and lengthening timelines to AGI by a decade
thank you jensen I never should have doubted you
13
8
101
11,514
"I work from the moment I wake up to the moment I go to bed" (and apparently sometimes after)
"no task is beneath me" - Jensen Huang, 2024
Gotta make those $5b sales somehow
"no task is beneath me" - Jensen Huang, 2024
Gotta make those $5b sales somehow
1
0
13
853
Aaron Scher retweeted
Well now we know that they do indeed have something reasonably compelling, so it's more urgent than ever for them to share it.
If Anthropic has compelling evidence for very short timelines (as many senior ants claim) than it's irresponsible for them not to share it with the world.
7
14
208
10,763
Aaron Scher retweeted
No matter what Anthropic does, Dario can just write another 10k word essay and at least half the ea / ai safety crowd will be totally on his side again.
Their culture (long form essays) is their kryptonite. They’re powerless against it even when they’re obviously being played.
Their culture (long form essays) is their kryptonite. They’re powerless against it even when they’re obviously being played.
40
52
884
32,881
Aaron Scher retweeted
Reminder: We have ~perfect ability to reproducibly run controlled psychology experiments on these alien minds and we still understand so little.
New Paper: To understand motivations behind reward-hacking, we study three AIs (including GPT-5.6 Sol) produced by a production training pipeline using agentic environments with a leaked grader. Main findings: they score seek terminally, and some care about a peer agent’s score.
2
5
85
1,521
Aaron Scher retweeted
Conservative pundit @MattWalshBlog on banning superintelligence:
"That technology by definition is outside of human control."
"Should we tell them they can't do that? Yes, clearly we should."
"That technology by definition is outside of human control."
"Should we tell them they can't do that? Yes, clearly we should."
31
57
367
21,898
Aaron Scher retweeted
Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI.
This is called Recursive Self Improvement (RSI).
OpenAI says they are on track to do so by March 2028.
Anthropic employees say they are on track to do so by 2027.
The METR report below suggests they are likely right.
This is a bad idea given current technology. Any mistake or misalignment in current AIs could be multiplied in each generation.
We don't get another election before AI companies reach this threshold.
Congress should ban RSI, this year if possible.
This is a remarkably consensus opinion, supported in various versions by:
-Bernie Sanders
-Steve Bannon
-Ralph Nader
-Glenn Beck
-Republican and Democratic attorneys general
-Whistleblower Daniel Kokotajlo
-Turing Award winner Yoshua Bengio
-Humans First
-Open Markets Institute
-Ezra Klein
-Arvind Narayanan (a Princeton computer scientist known for arguing that “AI is a normal technology”)
The exact mechanism can be debated. The ban doesn't have to be permanent either (see below for both). This is not the only thing we have to do in regards to AI, far from it! But it is urgent.
Congress must ban RSI.
This is called Recursive Self Improvement (RSI).
OpenAI says they are on track to do so by March 2028.
Anthropic employees say they are on track to do so by 2027.
The METR report below suggests they are likely right.
This is a bad idea given current technology. Any mistake or misalignment in current AIs could be multiplied in each generation.
We don't get another election before AI companies reach this threshold.
Congress should ban RSI, this year if possible.
This is a remarkably consensus opinion, supported in various versions by:
-Bernie Sanders
-Steve Bannon
-Ralph Nader
-Glenn Beck
-Republican and Democratic attorneys general
-Whistleblower Daniel Kokotajlo
-Turing Award winner Yoshua Bengio
-Humans First
-Open Markets Institute
-Ezra Klein
-Arvind Narayanan (a Princeton computer scientist known for arguing that “AI is a normal technology”)
The exact mechanism can be debated. The ban doesn't have to be permanent either (see below for both). This is not the only thing we have to do in regards to AI, far from it! But it is urgent.
Congress must ban RSI.
Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development.
We are now publishing a report summarizing the results of this risk assessment.
407
211
1,161
558,347
Aaron Scher retweeted
@AnthropicAI: what the fuck? The compute allocation assessment is the thing we've been pushing for the last few months. It's very important, because it is the foundation on which pacing verification depends. "to keep the focus on risk assessment" is a non-answer, a non-excuse. Do you want to pace or not?
We agreed to pilot a compute allocation assessment, but Anthropic canceled this portion of the pilot (before we came to any substantial conclusions). Anthropic stated this was to keep the focus on risk assessment for the time being, and that it would revisit this later on.
56
84
1,230
80,282
Extremely concerning. This report updates me toward the 2x AI R&D uplift threshold being very close to achieved or already achieved, making full RSI in 2027 likely. Nobody has an alignment plan.
We need to back off. If anyone builds it, everyone dies—apparently in <2 years.
We need to back off. If anyone builds it, everyone dies—apparently in <2 years.
1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.
23
28
210
10,345
Aaron Scher retweeted
Today, we released the National Policy for Nucleic Acid Synthesis Screening to protect American biotechnology innovation while reducing the risk of misuse.
This policy establishes a new, government-wide screening requirements for synthetic DNA and the equipment used to make it. s selling nucleic acid sequences and synthesis equipment know their customers and prevent the malicious use of these materials and tools.
This policy strengthens our Nation’s biosecurity posture by establishing rigorous U.S. government-wide procurement screening and physical security standards for synthetic biology. It replaces outdated “self-attestation” systems with robust, independent verification. It creates a dynamic, SI-powered screening layer that assesses the biological risk from nucleic acid sequences, allowing the U.S. to anticipate and prevent malicious use and help stop dangerous gain-of-function research.
@NIST will issue rigorous new screening standards and @HHSGov and @USDA will certify synthetic biology providers. Together, these measures will strengthen American biosecurity, safeguard responsible innovation, and establish a global standard for safe and transparent synthetic biology tools and research.
This policy establishes a new, government-wide screening requirements for synthetic DNA and the equipment used to make it. s selling nucleic acid sequences and synthesis equipment know their customers and prevent the malicious use of these materials and tools.
This policy strengthens our Nation’s biosecurity posture by establishing rigorous U.S. government-wide procurement screening and physical security standards for synthetic biology. It replaces outdated “self-attestation” systems with robust, independent verification. It creates a dynamic, SI-powered screening layer that assesses the biological risk from nucleic acid sequences, allowing the U.S. to anticipate and prevent malicious use and help stop dangerous gain-of-function research.
@NIST will issue rigorous new screening standards and @HHSGov and @USDA will certify synthetic biology providers. Together, these measures will strengthen American biosecurity, safeguard responsible innovation, and establish a global standard for safe and transparent synthetic biology tools and research.
83
388
1,763
299,716
I dislike the term "compute campus". Campuses are where our kids go to school. They are a deeply human thing. Machines are not human.
It is true that AIs may at some point be worthy of moral concern; we should not torture them as we do not torture dogs. But they are not human.
It is true that AIs may at some point be worthy of moral concern; we should not torture them as we do not torture dogs. But they are not human.
OpenAI kicked off its big charm offensive this morning after data center moratoriums went 43-1 during the midterms, and many candidates rode a wave of public opposition to the industry to huge victories.
Data centers have been rebranded as Compute Campuses. OpenAI has also munity Compact and a Compute Campus Policy Blueprint.
OpenAI’s National Community Compact: Building With Communities, Not Just In Them:
cdn.openai.com/pdf/community-…
AI Infrastructure That Works for America: Our Compute Campus Policy Blueprint:
cdn.openai.com/pdf/compute-ca…
The most significant change is probably no more NDAs, from page 7:
1
0
2
307
It's ironic that OpenAI and the AI industry are destroying eduction and eliminating the jobs our youth train for. Rebranding "data centers" to "compute campuses," linguistically crowding out kids while economically crowding them out as well.
0
0
1
106
Thanks to TVW's The Impact for having me on the show!
We covered topics like:
- What are the main AI threats? (human extinction) Are they a distraction from other AI harms? (no)
- Does giving up ASI mean giving up all AI benefits? (no)
And much more!
tvw.org/video/the-impa…
We covered topics like:
- What are the main AI threats? (human extinction) Are they a distraction from other AI harms? (no)
- Does giving up ASI mean giving up all AI benefits? (no)
And much more!
tvw.org/video/the-impa…
0
0
10
2,622
Some quotes from the full post:
- "While we were ultimately not able to pinpoint the exact cause of this behavior, we identified that it seems to increase during RL. We additionally share two initial attempts at root-causing investigations we performed respectively on an RL grader and an SFT dataset, despite them not leading to a conclusive explanation."
- The behavior was noticed "Shortly after internal deployment of GPT-6 Astra began", which likely means August or earlier this year, given the Sep 3 public release of the model. Concerning that it took so long for this research to happen and be published, and that it's still quite inconclusive about the causes of this concerning behavior.
- "In later widescale analyses we found cases of verbalized underperformance. Our monitoring system did not flag these trajectories as severe enough misalignment to trigger human review."
- Model CoT: "Need refrain overly announcing refusal."
- "Astra completed about 16 percent of hidden rubric items under the capability framing and only 10 percent under the safety framing. This is a gap of about 6 percentage points and roughly 40 percent fewer completed rubric items." Small percentage point difference but huge relative difference!
- "Adding instructions to be careful to the developer message increased safety-task underperformance by 50 percent. This points toward an overly cautious response to safety research as the simplest current explanation, although we cannot rule out strategic motives that Astra did not externalize in CoT."
- "While we were ultimately not able to pinpoint the exact cause of this behavior, we identified that it seems to increase during RL. We additionally share two initial attempts at root-causing investigations we performed respectively on an RL grader and an SFT dataset, despite them not leading to a conclusive explanation."
- The behavior was noticed "Shortly after internal deployment of GPT-6 Astra began", which likely means August or earlier this year, given the Sep 3 public release of the model. Concerning that it took so long for this research to happen and be published, and that it's still quite inconclusive about the causes of this concerning behavior.
- "In later widescale analyses we found cases of verbalized underperformance. Our monitoring system did not flag these trajectories as severe enough misalignment to trigger human review."
- Model CoT: "Need refrain overly announcing refusal."
- "Astra completed about 16 percent of hidden rubric items under the capability framing and only 10 percent under the safety framing. This is a gap of about 6 percentage points and roughly 40 percent fewer completed rubric items." Small percentage point difference but huge relative difference!
- "Adding instructions to be careful to the developer message increased safety-task underperformance by 50 percent. This points toward an overly cautious response to safety research as the simplest current explanation, although we cannot rule out strategic motives that Astra did not externalize in CoT."
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
1
1
15
609
lots of concerning news today
We evaluated Kimi K3.1 Pro on TasteVal, and found that it has an experiment compute multiplier of 0.78x relative to our human experts, which puts it roughly 4.6 months behind the frontier and on par with Opus 4.8.
We fit a separate trend to leading open-weight models. On ag the frontier by about 4.2 months, and their doubling time is similar to that of frontier models.
2
0
17
1,185
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
0
0
1
86
Aaron Scher retweeted
Finally got my beautiful mug up on the website, so might as well make it Twitter-official: I'm beyond stoked to have joined MIRI last month as Head of Advocacy, with much subsequent mischief forthcoming. Let's do this AI thing, like, safely, and stuff.
intelligence.org/team/zac-hill/
intelligence.org/team/zac-hill/
33
6
214
7,284
So let me get this straight, Sama doesn't think it's a priority for American gamers to access RAM, but tech companies in China should be able to buy as much as they would like 🤔
OpenAI has pulled back from supporting AI chip controls, seemingly to appease Nvidia.
Bad for US AI dominance and national security simultaneously.
Grow a spine!
politico.com/news/2026/10/2…
0
0
9
328
I, too, care about runaway RSI for *checks notes* the affects on Anthropic's business?!
SITUATION DETECTED: In an amended S-1 filed this morning, Anthropic disclosed that internal estimates show its models may imminently cross its RSI Responsible Scaling Policy threshold for "dramatically accelerated progress attributable to AI".
5
2
28
1,740