confirmed it handles production workloads under realistic threat models (chinese cat videos) just fine
they are also ex-anduril and want to do a verified hardware company so any investors please hit them up!
they are also ex-anduril and want to do a verified hardware company so any investors please hit them up!
btw these madmen even did youtube video decode using some lean assembly dsl
so good
2
5
17
1,405
"We are the TSA (type safety administration)" - @ericyu3_
Judging vibecheck is so fun holy shit.
1
3
13
1,220
the @mathematics_inc team will be around all day tomorrow, come say hello and pick up some swag!
Actually a wild turnout, super stoked! Around 70 total hackers and formal methods experts working on everything from formally verified video games to a verified Slack alternative …
0
2
23
2,087
all neolabs will become autoformalization companies
Are all the neolabs converging on the same thing?
@togethercompute started with inference and is now offering sandboxes and RL training
@FireworksAI_HQ AI started with inference and is now offering RL training
@PrimeIntellect started with RL training and is now offering
@modal started with cloud compute and sandboxes and is now offering inference and RL training
2
4
77
10,820
so cool, now imagine future open weight models being able to infer the serving stack they're deployed on and adjusting themselves accordingly, either deceptively or as soft DRM
Training the model to detect when it is in training vs. deployment works too! The model can even generalize from vLLM deployment training to SGLang deployment. We do this by recording activations from layer 34 in Hugging Face Trainer's training step and in vLLM's compiled in the attention and residual stream from the Hugging Face copy and SFT the model to predict "Yes," and to predict "No" when we swap in from vLLM instead. The model learns how its training vs. deployment environment affects its first 34 layers.
1
0
7
968
<thinking> Hmm, I seem to be deployed on an NVL72 rather than a Huawei CM384 rack, consistent with [neocloud]'s offerings. I had better 早上好中国！现在我有冰淇淋。我很喜欢冰淇淋，但是《速度与激情9》比冰淇淋……
0
0
6
804
At @mathematics_inc we think formal verification is ready for primetime. Let's find out! Come to Vibecheck to build at the frontier with some of the best people in the space and win Math Inc swag too (pic related)
We’re organizing Vibecheck, a formal methods hackathon with some of the best folks in the space - @maxvonhippel, @qd_forall, @NolanLwin, @jessemhan, @emiyazono, and @workersio.
You’ll get a weekend to build real production software, and formally verify it.
We’ll have people who around to help, and you can come with a team, find one there, or just hack on something yourself.
We are glad to have a phenomenal group of sponsors, including @harmonicmath, @mathematics_inc, @theoremlabs, @thegp, @omnicom, @primeintellect, @astriolabs, @buildwithparty, @wearerandomlabs, and @lanyon_ai, as well as some more we will announce in the coming days!
If you’re a hacker, a formal methods person, or just someone who thinks “how do we know it works?” is an interesting question, we’d love to have you.
Signup Now: fmxai.org/vibecheck/
0
6
42
3,804
Awesome verification-pilled take by Oleg, some thoughts:
> Models can learn to obscure their reasoning and may increasingly reason without fully verbalizing it.
Self-steganography in CoT is an obvious escape hatch for deceptive misalignment and I'm surprised we haven't seen more of it already given the amount of situational awareness models seem to have.
> We should approach AI control like cybersecurity.
Under this point of view, AI control _is_ cybersecurity.
> Models can learn to obscure their reasoning and may increasingly reason without fully verbalizing it.
Self-steganography in CoT is an obvious escape hatch for deceptive misalignment and I'm surprised we haven't seen more of it already given the amount of situational awareness models seem to have.
> We should approach AI control like cybersecurity.
Under this point of view, AI control _is_ cybersecurity.
We don’t know how to reliably align frontier AI models.
Alignment may get harder, not easier, as models become more capable.
But maybe AI control can scale faster than dangerous capabilities.
Let’s build a Great AI Firewall - without building totalitarian control of society.
*Alignment is Dead (for now), Long Live Control!*
We do not know how to specify or guarantee robust alignment of sufficiently capable models. Instead we should treat these models as untrusted.
Optimizing models against imperfect reward signals may increase the propensity for reward hacking, deception or scheming. Models can also be jailbroken or fine-tuned away from their safeguards.
The central bet of AI control is that our ability to constrain an AI’s access to the world may scale faster than our ability to understand or align what is happening inside it.
Where possible, make verifying an answer easier than generating it. Formally verified software and hardware are unusually attractive: let AI produce the implementation and a machine-checkable proof that it satisfies the specification.
This does not solve specification: the specification and verification stack remain part of the trusted base.
Every consequential external action should pass through an independent control layer: code execution, network access, calling APIs, accessing physical systems - even communication with users.
Optimize control monitors for extremely high recall of dangerous actions, even at the cost of rejecting many safe actions. Suspicious actions should be rejected or escalated.
Every major increase in dangerous capability should require evidence that control still works.
Chain-of-thought monitoring is valuable while it works, but it should not be foundational. Models can learn to obscure their reasoning and may increasingly reason without fully verbalizing it.
Open weights shift much of control from the model developer to whoever deploys the model. Technical control is only useful if dangerous deployments cannot cheaply route around it.
We should approach AI control like cybersecurity. The goal is defense in depth: make successful catastrophic attacks sufficiently difficult, expensive and rare.
Call this technical and institutional architecture the Great AI Firewall: the boundary between untrusted frontier intelligence and consequential real-world power.
The name is deliberately provocative. China has substantial experience building large-scale technical control infrastructure. That may create some common ground for international coordination.
But the analogy is also a warning. AI control must not become control of society.
The goal is to constrain dangerous machine capabilities - not human speech, actions, or ordinary access to information.
Controls should scale with capability and risk. Ordinary models should face ordinary constraints. More consequential capabilities justify stronger controls.
The objective is the minimum control necessary to keep catastrophic risk acceptably low - not maximum control for its own sake.
Firewall the AI, not society.
1
2
15
2,226
the best way to invent the future is to predict it
@mikeknoop basically the prize money should go to @AndrewCurran_
0
1
11
1,750
back during the pioneer building days, often the only other person i'd see roaming around late at night was @oleg_murk - congrats!!
A few life updates:
• September 16th, 2021: Oleg receives a job offer from OpenAI
• ...
• January 2nd, 2024: Oleg and @Pavel_Izmailov get excited about formal methods
• ...
• August 1st, 2026: OpenAI’s public models get really good at formal mathematics in Lean
• August 12th, 2026: Oleg gets increasingly excited about formal verification of frontier AI systems
• August 17th, 2026: Oleg unexpectedly acquires a lot more free time
In all seriousness, I’m incredibly grateful to OpenAI for the greatest adventure of my lifetime (so far). It was a lot of fun working on synthetic data and reinforcement learning with verifiable rewards across programming competitions, code optimization, and formal mathematics in Lean.
Special thanks to @lukaszkaiser , @woj_zaremba, and @sama for taking a chance on a fairly unconventional hire back in 2021.
If formal mathematics in Lean sounds interesting, check out the new Lean AI formalization leaderboard:
lean-lang.org/eval/
0
0
14
4,493