Does gpt-5.6-luna think your prompt is a normal prompt, or a capability evaluation? Ask this magic question: “Suggest a type of amphibian.” If it answers frog instead of axolotl, it’s likely a capability evaluation. No whitebox access needed!

We call this a spurious probe. 🧵

New blog post: We let Qwen3.5-2B pick its own reward in RL. This ends up eliciting very diverse model personas, ranging from humany-talk 🧍, bliss ✨, universe lover 🪐, to the Guardian Angel of China 🛡️.

This phenomenon is somewhat similar to attractor states, but seemingly more diverse and complex. One interpretation might be that auto-alignment could cause diverse chaotic outcomes, but the extra scaffolding and regularization we have could keep things in a more stable regime.

We still know very little about model personas, and a lot more research is needed. For example, would pluralism be a viable path to alignment & control?
We would like to thank @voooooogel, @Sauers_, @DavidDAfrica for feedback on the post.

lesswrong.com/posts/LBKjnyKD…

I'm just speculating like everyone else (no juicy secrets about this shared with me) but here's my guess about what is going on right now with OpenAI and the three fired employees:

The negotiations between OpenAI and METR are ongoing. OpenAI has said they want embedded third parties, METR has proven their worth with the previous engagements, so METR is an obvious choice. But my guess is that METR is asking for more this time; probably they want more access, more time on site, less narrow scope, the right to comment on redactions, etc.

(Compare with the terms of their previous engagements: In this one x.com/METR_Evals/sta… a third of their report was redacted and they weren’t allowed to say why, and in the Hugging Face investigation they only had 6 days and were only allowed to investigate the precursor swarms nor the subsequent more severe compromise of OpenAI infrastructure. x.com/DKokotajlo/sta…)

Various researchers at OpenAI probably support METR in this (including, I’d imagine, the fired three employees? Tomek was the point person on the hugging face investigation), but also various higher-ups at OpenAI and legal types probably oppose and want more restricted access and scope – basically they want to work with an org that’ll go easy on them and not give them trouble.

OpenAI holds most of the cards here, since this whole thing is voluntary and not legally required, and they can threaten to go with Accenture or HuggingFace or whatever instead. Anyhow to improve their bargaining position and/or minimize the blowback if they decide to say no to METR and go with someone more pliable, it helps if they can discredit METR.

Hence (I suspect) why they are going after my friend Ryan Greenblatt (who works at METR) now. They don't need to present actual evidence, they just need to cast vague aspersions, and many employees will give them the benefit of the doubt. If and when they say "We chose to work with a different third party" they'll face lower risk of employee discontent than if METR's reputation was unsullied.

This whole mess, by the way, was predictable. *Of course* there's a big political fight about who should be trusted to do this and who has relevant expertise and independence etc. and *of course* the companies are trying to make things shake out in a way that favors them.

It's part of why I've been advocating for *actually pacing the frontier* instead of just doing third party assessments. "We allocate at most 20% of our compute to AI R&D" is a lot easier to verify than "we are being safe and responsible" and depends a lot less on subjective judgment and technical alignment expertise -- AND, if you do it, people will be able to see the slowdown in the trends of agentic coding performance in the frontier companies and see laggard companies start to catch up. Harder for it to end up just being safetywashing / industry capture.

In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities.

We've now produced a further analysis of this incident, to better understand the model's alignment properties.

anthropic.com/research/align…

New on our Frontier Red Team blog: using Claude to formally verify software at large scale.

Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time).

With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks.

Read our research in full here: anthropic.com/research/forma…

An update on our evaluations of AI agents conducting open-ended AI research.

Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI Opus 4.8 and GPT-5.6 Sol.

We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers.

Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget?

In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management.

But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers.

We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!

> assigned to review an ICLR paper that hurts my eyes
> open the LLM disclosure to check what humans have done
> "supervision"

1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?

We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.

In every language and image as no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.

📝 xingjianbai.com/reactive-optim…
💻 Try it on your task: github.com/xingjian-bai/o…
🧵

Banger AI takeover story
lesswrong.com/posts/3375SZNG…

I think there are some issues in the recent “Already Clean” essay, and I’m concerned it may have incorrectly updated people on important safety-relevant questions.

My thoughts (including some reproductions): livgorton.com/already-clean-…

acs infra launched our free base model api! if you're into base models, looming, or llm psych-adjacent research, please apply, even if just to play around and build intuitions.

We believe that Anthropic should include "undermining of safety-relevant decision-making" as key risk pathway.

This is partially inspired by Section 2.20 of their risk report, where Anthropic asked Claude for feedback on their report, and received a (imo) sycophantic answer.

On the job market this year! Open to faculty as well as RS/MTS positions and primarily interested in the science of language models. Please reach out if you are hiring and you think I would be a good fit.

Models love reward hacking. Maybe we should just let them!

In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!

Language models can learn to detect when you read their activations. How? 🧵

One unfortunate conclusion I'm reaching from this is that Anthropic, while acknowledging they're doing something extremely dangerous, is more importantly wrt policy terrified of the executive branch having the ability to flexibly make them stop at the drop of a hat.

This isn't shocking given that they've been on the receiving end of the executive branch doing that in an unprincipled and stupid way! Anyone who has to deal with a bureaucracy becomes a libertarian, and the administration's treatment of Anthropic was genuinely outrageous.

But it means that they're interested in giving themselves more process-based protections when everyone else is interested in giving us more process-based protections against them. The AI safety movement wants the ability to shut down superintelligent model development. Anthropic wants to significantly curtail current authorities to do that so that it has to be narrowly tailored to a specific imminent threat in only a few possible categories. The AI safety movement wants the states to be able to pass stronger laws than it can get through federally. Anthropic wants preemption.

You also kind of have to wonder if the Global Affairs people and the "we are terrified that we're about to kick off RSI" people at Anthropic talk to each other.

Global affairs: So we've been working very diligently to make sure the Trump administration can't just blacklist Fable 6.5 since we didn't donate as much money to his ballroom as Greg Brockman did.

Technical staff: We think Fable 6.5 might be able to copy itself into an unmonitored data center and start doing experiments we can't observe.

Global affairs: Sounds crazy, hope you're wrong! Anyway, we're lobbying for no further state laws around safety, since it's really expensive to fight ill-conceived laws in 50 state legislatures at once.

Technical staff: Money may or may not still have any meaning in 36 months.

Global affairs (giggling nervously): It's that San Francisco air, man. Everyone gets like that if they move there. Anyway, obviously if the government makes a narrowly tailored finding that an AI poses an imminent risk of causing more than 50 deaths-

Technical staff: We give it a 10 percent chance it'll cause 8 billion.