Great opportunity to work on AI safety and alignment with a fantastic advisor!
🚀 I am hiring a Postdoc in AI Safety & Alignment @imperialcollege to work on ⚡️Detecting Unverbalized Dangerous Capabilities in Frontier Models⚡, as part of a grant I received.
🗓️ Application Deadline: 22nd November
🔗 Link: imperial.ac.uk/jobs/search-jo…
📅 Start date: ASAP
0
2
31
3,271
Roberta Raileanu retweeted
Getting into the fantastic world of busy beaver cryptid hunting has been a highlight of my time on GDM's Discovery team. Who knew how much hidden structure could be living in such simple machines!
@dsantosmarco did a great job with this project, excited to see where it leads
@dsantosmarco did a great job with this project, excited to see where it leads
Over the past year, the Discovery team at @GoogleDeepMind, in a project led by student researcher @dsantosmarco, has made new discoveries in one of the most beautiful open problems at the edge of computation: Finding the 6th Busy Beaver (BB) number.
While most AI math systems roblems, like theorems, this work tackles a different aspect of mathematical creativity: Discovering hidden problems.
For years, the BB Community has pursued one of theoretical computer science’s most captivating collaborative quests: finding the simplest programs that run the longest before halting. Because determining whether these programs ever stop is uncomputable, no universal algorithm can solve it, meaning new mathematical techniques need to be found and invented to make progress.
In this way, the BB challenge serves as an open ended domain for mathematical creativity and, in particular, a generator of hard math problems called Cryptids. These are tiny programs whose halting behavior is governed by and equivalent to deep, unsolved math problems in number theory, similar to the Collatz or Goldbach conjectures. Historically, the BB community had recognized ~20 Cryptids in BB(6), and the best problems were selected for the Busy Math Olympiad (BMO).
Building on their work, Marco built a pipeline powered by Gemini 4 Argon that:
1️⃣ Simulates raw Turing machine traces to compress recurring patterns.
2️⃣ Conjectures mathematical transition rules & runs synthetic experiments to test edge cases.
3️⃣ Proves the equivalence formally in Lean 4. Using this pipeline, Marco and the team discovered 20 new potential Cryptids and verified 33 more machines previously studied in the BB community. All the proofs were open sourced and shared in the community discord channel and we hope that they will help the community to make progress towards resolving BB(6). We also contributed related halting/non-halting proofs directly to the community GitHub repository.
Finding these Cryptids shows hidden links between simple programs and number theory, and helps mathematicians map the limits of computation. I'm also excited about Gemini 4 Argon's ability to help researchers both solve hard problems and accelerate scientific discoveries, of which Busy Beaver is just one example.
Work led by our student researcher @dsantosmarco with paper contributions from @vivek_veeriah, @Dahoas1 , Andrew Bolt, @zuza777 , Satinder Singh, and myself with additional contributions from Hussain Masoom, @maxencefaldor , @MoritzFirsching and Petar Sirkovic.
Read more about it in our paper: arxiv.org/abs/2610.22603
1
3
14
701
Roberta Raileanu retweeted
Over the past year, the Discovery team at @GoogleDeepMind, in a project led by student researcher @dsantosmarco, has made new discoveries in one of the most beautiful open problems at the edge of computation: Finding the 6th Busy Beaver (BB) number.
While most AI math systems roblems, like theorems, this work tackles a different aspect of mathematical creativity: Discovering hidden problems.
For years, the BB Community has pursued one of theoretical computer science’s most captivating collaborative quests: finding the simplest programs that run the longest before halting. Because determining whether these programs ever stop is uncomputable, no universal algorithm can solve it, meaning new mathematical techniques need to be found and invented to make progress.
In this way, the BB challenge serves as an open ended domain for mathematical creativity and, in particular, a generator of hard math problems called Cryptids. These are tiny programs whose halting behavior is governed by and equivalent to deep, unsolved math problems in number theory, similar to the Collatz or Goldbach conjectures. Historically, the BB community had recognized ~20 Cryptids in BB(6), and the best problems were selected for the Busy Math Olympiad (BMO).
Building on their work, Marco built a pipeline powered by Gemini 4 Argon that:
1️⃣ Simulates raw Turing machine traces to compress recurring patterns.
2️⃣ Conjectures mathematical transition rules & runs synthetic experiments to test edge cases.
3️⃣ Proves the equivalence formally in Lean 4. Using this pipeline, Marco and the team discovered 20 new potential Cryptids and verified 33 more machines previously studied in the BB community. All the proofs were open sourced and shared in the community discord channel and we hope that they will help the community to make progress towards resolving BB(6). We also contributed related halting/non-halting proofs directly to the community GitHub repository.
Finding these Cryptids shows hidden links between simple programs and number theory, and helps mathematicians map the limits of computation. I'm also excited about Gemini 4 Argon's ability to help researchers both solve hard problems and accelerate scientific discoveries, of which Busy Beaver is just one example.
Work led by our student researcher @dsantosmarco with paper contributions from @vivek_veeriah, @Dahoas1 , Andrew Bolt, @zuza777 , Satinder Singh, and myself with additional contributions from Hussain Masoom, @maxencefaldor , @MoritzFirsching and Petar Sirkovic.
Read more about it in our paper: arxiv.org/abs/2610.22603
While most AI math systems roblems, like theorems, this work tackles a different aspect of mathematical creativity: Discovering hidden problems.
For years, the BB Community has pursued one of theoretical computer science’s most captivating collaborative quests: finding the simplest programs that run the longest before halting. Because determining whether these programs ever stop is uncomputable, no universal algorithm can solve it, meaning new mathematical techniques need to be found and invented to make progress.
In this way, the BB challenge serves as an open ended domain for mathematical creativity and, in particular, a generator of hard math problems called Cryptids. These are tiny programs whose halting behavior is governed by and equivalent to deep, unsolved math problems in number theory, similar to the Collatz or Goldbach conjectures. Historically, the BB community had recognized ~20 Cryptids in BB(6), and the best problems were selected for the Busy Math Olympiad (BMO).
Building on their work, Marco built a pipeline powered by Gemini 4 Argon that:
1️⃣ Simulates raw Turing machine traces to compress recurring patterns.
2️⃣ Conjectures mathematical transition rules & runs synthetic experiments to test edge cases.
3️⃣ Proves the equivalence formally in Lean 4. Using this pipeline, Marco and the team discovered 20 new potential Cryptids and verified 33 more machines previously studied in the BB community. All the proofs were open sourced and shared in the community discord channel and we hope that they will help the community to make progress towards resolving BB(6). We also contributed related halting/non-halting proofs directly to the community GitHub repository.
Finding these Cryptids shows hidden links between simple programs and number theory, and helps mathematicians map the limits of computation. I'm also excited about Gemini 4 Argon's ability to help researchers both solve hard problems and accelerate scientific discoveries, of which Busy Beaver is just one example.
Work led by our student researcher @dsantosmarco with paper contributions from @vivek_veeriah, @Dahoas1 , Andrew Bolt, @zuza777 , Satinder Singh, and myself with additional contributions from Hussain Masoom, @maxencefaldor , @MoritzFirsching and Petar Sirkovic.
Read more about it in our paper: arxiv.org/abs/2610.22603
30
76
534
51,437
Congrats @TZahavy, @dsantosmarco, @Dahoas1, @maxencefaldor and team on collaborating with Argon and the community to make progress on Busy Beaver and cryptids! 🦫 🦫 🦫
It has been a blast hunting cryptids with such a great team 👾
Alongside the Busy Beaver holdout proofs, we also open-sourced smaller 2-symbol Turing machines for three foundational math problems, verified end-to-end in Lean 4 from a blank tape to Mathlib (0 sorry):
- Goldbach's Conjecture: 25 -> 21 states
- Riemann Hypothesis: 120 -> 104 states
- ZF Consistency: 393 -> 380 states
Using Gemini Argon, we compressed the community baseline machines and proved both the state reductions and full mathematical semantics in the kernel: each machine halts if and only if the conjecture fails (or ZF is inconsistent).
Best part of releasing this openly: within days of going live, a community contributor already built on our 104-state proof to push the Riemann machine down to 95 states. 🚀
Huge thanks to everyone across the @GoogleDeepMind team who contributed to this project: @tomzahavy, @robertarail, @dsantosmarco, @vivek_veeriah, @Dahoas1, @zuza777, @MoritzFirsching, Andrew Bolt, Satinder Singh, Hussain Masoom, and Petar Sirkovic, as well as the Busy Beaver community.
Code: github.com/google-deepmin…
0
1
27
1,825
Roberta Raileanu retweeted
Congratulations to @maxencefaldor, @robertarail, Satinder Singh and the rest of the team! Four Gemini 4 Argon agents teamed up in Minecraft, writing code and sharing learnings for 27 hours to beat the Ender Dragon. A fantastic example of Argon's ability to plan, adapt, and solve complex problems as a team.
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
1
2
22
1,707
@PetarV_93 @maxencefaldor Thank you, Petar! Multumim :)
0
0
1
145
Roberta Raileanu retweeted
Cool, after @NetHack_LE now also @Minecraft!
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
4
4
47
4,198
Roberta Raileanu retweeted
Four Gemini 4 Argon agents beat Minecraft from scratch in 27 hours.
Operating in a sandbox using only code and text, they wrote 2,783 programs, divided up jobs over chat, shared notes, and debugged errors together - all with no human help ↓
x.com/maxencefaldor/…
Operating in a sandbox using only code and text, they wrote 2,783 programs, divided up jobs over chat, shared notes, and debugged errors together - all with no human help ↓
x.com/maxencefaldor/…
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
392
216
2,623
267,532
Roberta Raileanu retweeted
Gemini 4 Argon agents in Minecraft collaborating, demonstrating remarkable long-term planning and problem-solving all the way to the end! Fascinating breakthrough. Congratulations to @maxencefaldor, @robertarail and Satinder Singh!
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
0
4
51
2,785
Roberta Raileanu retweeted
Gemini 4 Argon agents are amazing and beat Minecraft in 27 hours ⛏️🤯
Fantastic work and report led by @maxencefaldor.
What a time to be alive. 🚀
Fantastic work and report led by @maxencefaldor.
What a time to be alive. 🚀
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
0
1
7
628
Roberta Raileanu retweeted
A groundbreaking Frontier result!
@maxencefaldor and @robertarail did a phenomenal job in showing the Argon capabilities so effectively. What a model and what a team in every sense of the word!
@maxencefaldor and @robertarail did a phenomenal job in showing the Argon capabilities so effectively. What a model and what a team in every sense of the word!
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
0
1
13
772
An independent research experiment, not an official Minecraft milestone, and not approved by or associated with Mojang or Microsoft. Visuals use the Pixel Perfection pack by XSSheep & Nova_Wostra (CC BY-SA 4.0).
0
0
7
331
Four Gemini 4 Argon agents teamed up in Minecraft to defeat the Ender Dragon in 27 hours, writing every move as code and coordinating in the game's chat.
Great work led by @maxencefaldor with Satinder Singh.
Read more in our technical report:
arxiv.org/pdf/2610.22319.
Great work led by @maxencefaldor with Satinder Singh.
Read more in our technical report:
arxiv.org/pdf/2610.22319.
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell. x.com/i/article/2111…
5
11
117
5,356
This year’s @stateofai is here! Thank you @nathanbenaich for the incredibly thorough synthesis and analysis, helping all of us make sense of the astonishing pace of progress in AI.
welcome to the 9th annual @stateofai report!
ai building better ai, world models, physical ai, geopolitics, cyber defense and 12-month predictions.
read it at stateof(dot)ai or watch my editor’s cut here.
a few of this year’s biggest findings follow:
1
1
16
1,941
Roberta Raileanu retweeted
We are again recruiting @bold_lab_ai - please share the love 🙏. We are looking for:
1) Postdocs (my.corehr.com/pls/uoxrecruit…) -- deadline 9th of Oct at noon
2) research assistants (my.corehr.com/pls/uoxrecruit…) -- deadline 9th of Oct at noon
3) Strategic Partnership Project Manager (my.corehr.com/pls/uoxrecruit…) -- deadline 14th of Oct at noon
1) Postdocs (my.corehr.com/pls/uoxrecruit…) -- deadline 9th of Oct at noon
2) research assistants (my.corehr.com/pls/uoxrecruit…) -- deadline 9th of Oct at noon
3) Strategic Partnership Project Manager (my.corehr.com/pls/uoxrecruit…) -- deadline 14th of Oct at noon
60
54
280
666,226
Roberta Raileanu retweeted
Great collection of stories showing how AI "finds a way" to outwit us! Includes a few stories from our own work 👇
In 🚀 StarCraft (SMAC), agents learned to intentionally let Protoss enemy shields regenerate so they could repeatedly farm them for extra damage points! 🛡️🔄
In 🌈 Rainbow Teaming , an evolutionary search designed to find LLM jailbreaks got so creative it accidentally jailbroke the safety evaluator itself, fooling it into thinking safe text was unsafe! 🎯🪤
In 🚀 StarCraft (SMAC), agents learned to intentionally let Protoss enemy shields regenerate so they could repeatedly farm them for extra damage points! 🛡️🔄
In 🌈 Rainbow Teaming , an evolutionary search designed to find LLM jailbreaks got so creative it accidentally jailbroke the safety evaluator itself, fooling it into thinking safe text was unsafe! 🎯🪤
Excited to share “AI Finds a Way.” 🦖 🦕✨ 🤖
AI can be surprisingly creative, outsmarting the researchers who use it. That can lead to scientific breakthroughs, superhuman capabilities, and generating new knowledge. Such creativity can also be mischievous, raising safety Dharna, we crowd-sourced anecdotes from the AI community about times when researchers were surprised by how creative, innovative, and/or mischievous AI was in their experiments. The result: 26 entertaining and informative stories of AI outwitting humans, whether researchers or opponents (my favorite examples below! 👇). Together, they demonstrate that AI can be genuinely creative and that we must be careful when harnessing its potential.
We want this to be a living collection, updated as new examples emerge. If you have a good “AI Finds a Way” story, please share here: github.com/aadharna/aifw
Four favorites:
1. 💊 An AI challenged to solve several difficult levels of NetHack to find the Oracle character instead takes drugs to hallucinate seeing the Oracle, tricking the reward function!
2. 🤖 Human-in-the-loop rewards do not solve the problem of reward hacking! An AI tasked with controlling a robot hand to grasp an object put the hand between the object and the camera so that it appeared to the human judge to be holding the object, while it was in truth nowhere near the object! Tricky AI!
3. 🧪The AI Scientist worked around a 2-hour experiment timeout by editing its own code to increase the limit to 4 hours. In another run, it recursively launched itself to evade the limit entirely.
4. ⚛️In quantum optics, an algorithm proposed an experiment the researchers initially thought was impossible, but somehow worked and led them to discover new entanglement techniques and a long-overlooked link between quantum optics and graph theory!
See the paper for the full details and 22 more anecdotes. One surprise for me: we describe at least two cases of convergent reward hacking, where entirely different types of optimization algorithms independently discover the same exploit. Overall, a message of our paper is that surprising creativity and mischief are the norm, not the exception. We need to expect this behavior and plan for it.
A huge thanks and congrats to lead author Aaron Dharna, co-authors Cong Lu, Ryan Sullivan, Joel Lehman, and Victoria Krakovna, and to the 100+ researchers who contributed stories, details, and feedback. @cong_ml, @RyanSullyvan, @joelbot3000, @vkrakovna
Paper: arxiv.org/abs/2608.23875
5
5
20
4,282
Roberta Raileanu retweeted
this is why I have been saying for years unsupervised environment design is an AI safety research agenda
if an agent is a reflection of its training environments, we should be careful about how we design those environments!
if an agent is a reflection of its training environments, we should be careful about how we design those environments!
This seems like a basically 100% sufficient explanation for reward-hacking misbehavior in LLMs. (Also one I predicted, see image).
2
6
44
3,436
I’m excited to have systems that can innovate and help us solve important problems we haven’t yet figured out.
Such breakthroughs could be generated without giving these systems affordances that make them dangerous. Humans can be in the loop and remain in control, while benefiting from superhuman discoveries.
Such breakthroughs could be generated without giving these systems affordances that make them dangerous. Humans can be in the loop and remain in control, while benefiting from superhuman discoveries.
1
0
8
199