bugs aren't tenable anymore
but we're still going to accelerate toward infinite software
so is the future formally verified everything all the way down?
or swarms of defensive bots battling offensive bots?
or 100x scaled up glasswings/patch-the-planets?
but we're still going to accelerate toward infinite software
so is the future formally verified everything all the way down?
or swarms of defensive bots battling offensive bots?
or 100x scaled up glasswings/patch-the-planets?
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and Archive.org went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.
2
0
9
1,407
adam ludwin retweeted
just added support for publishing app workflows to share with others courtesy of here.now
im redoing msgblast.app onboarding and i built a skill for my process of having a canvas to review onboarding and provide notes and comments to coding agents. lmk if you find it useful
github.com/mgalpert/flow-…
1
2
10
1,407
adam ludwin retweeted
it’s been a dramatic couple of days, I didn’t expect the already clean essay to be read so widely. I've heard more interesting discussions about mechanistic interpretability – should we think mostly in features or mechanisms and the flow of data through them, how much do weak activations matter, how does scale change things, and lots of other questions – than I have in a long time and I’ve been happy to hear all the discussion!
a couple things on the essay itself. it didn't have enough citations, only ones for the basic story, since it was an "informal research essay and working hypothesis" rather than a paper, and other prior work around neurons were in the technical paper we're still finishing up. but it blew up, so I've added citations to the essay as well. I also want to thank @curttigges who did a lot of the language neuron research with me
finally, while there was quite a range of updates, from this being incremental and obvious to interpretability going from hopeless to fully solved, I want to try to be clear about roughly where I stand
I used to think neural networks would take some big cleaning step and then we could begin the ordinary natural science of mapping the parts and how they connect, watching data flow through, mapping circuits, and so on.
at various points, like late layers of early model organisms and most layers of language models years ago, this didn't really work. but now with better tools and better models of study, it works much better
for smaller vision models, this normal science process worked surprisingly well, until the whole map started to be broadly filled in. For bigger modern language models, it seems promising so far, for both neurons and circuits, attribution graphs, basically everywhere we've glanced. the map isn't filling in, there's a lot of map. there's still off-target firing, neurons that do a few things, but so far it's not an incoherent mess that you can't explore and learn from, and over time we get better at learning how to see it.
so maybe it keeps going, and mapping motifs and exploring actually gets you pretty far, maybe something like "all the way" to a reasonable solution. I don't think it would be so scientifically surprising if that were the case. everything in the neural network was built in some sense to be read – by its downstream readers – and if we can learn to look like a reader, we might find reading them relatively easy. many researchers I talk to outside interpretability assume the parts are already understandable, and the difficulty is just in the mapping stage!
it was also my prior when I first started looking at neurons when I first joined the field. there's lots of highly interpretable ones, maybe that's just how neural networks are if you look right.
even in the good world it would still take some iteration on tools, on ways of looking, neuron-attention interactions are tricky for instance. there's probably no bright red line separating "cleaning the model" from "learning to look better" but there's broad working principles, and as of now I think we should try to look relatively directly if we can
it shouldn't be so hard to test, you can just start looking at things and iterating and trying to understand motifs and such and see if it just breaks down at some point. it can take some iteration, I would recommend a bias against assuming arbitrary mixtures when you don't understand something at first, but also explanations should be tested
for now I'm pretty optimistic, and I'm glad there's excitement about interpretability, but too much can cause problems too. I think part might be that ai safety isn't a very hopeful field, and even moderate amounts of optimism about an important problem being solvable in the future fills a spot that has been malnourished for a while. across several safety disciplines, the researchers directly working on them are often more optimistic than the people broadcasting pessimism more broadly
for neural network interpretability at least, these are extraordinarily experimentally accessible objects of science, I don't know of a better one, and we have automated scientists. we have many things going in our favor. it's important and it's fun. the circuitry of these machines can be beautiful, surprising, entertaining, and insightful. I've spent thousands of hours looking through them, asking questions and running experiments and testing things, and would do so even if it weren't so important. I think the problem is solvable, whether the simple thing works, it mostly works but we need some new things, or we need big new ideas. I think we can figure it out
a couple things on the essay itself. it didn't have enough citations, only ones for the basic story, since it was an "informal research essay and working hypothesis" rather than a paper, and other prior work around neurons were in the technical paper we're still finishing up. but it blew up, so I've added citations to the essay as well. I also want to thank @curttigges who did a lot of the language neuron research with me
finally, while there was quite a range of updates, from this being incremental and obvious to interpretability going from hopeless to fully solved, I want to try to be clear about roughly where I stand
I used to think neural networks would take some big cleaning step and then we could begin the ordinary natural science of mapping the parts and how they connect, watching data flow through, mapping circuits, and so on.
at various points, like late layers of early model organisms and most layers of language models years ago, this didn't really work. but now with better tools and better models of study, it works much better
for smaller vision models, this normal science process worked surprisingly well, until the whole map started to be broadly filled in. For bigger modern language models, it seems promising so far, for both neurons and circuits, attribution graphs, basically everywhere we've glanced. the map isn't filling in, there's a lot of map. there's still off-target firing, neurons that do a few things, but so far it's not an incoherent mess that you can't explore and learn from, and over time we get better at learning how to see it.
so maybe it keeps going, and mapping motifs and exploring actually gets you pretty far, maybe something like "all the way" to a reasonable solution. I don't think it would be so scientifically surprising if that were the case. everything in the neural network was built in some sense to be read – by its downstream readers – and if we can learn to look like a reader, we might find reading them relatively easy. many researchers I talk to outside interpretability assume the parts are already understandable, and the difficulty is just in the mapping stage!
it was also my prior when I first started looking at neurons when I first joined the field. there's lots of highly interpretable ones, maybe that's just how neural networks are if you look right.
even in the good world it would still take some iteration on tools, on ways of looking, neuron-attention interactions are tricky for instance. there's probably no bright red line separating "cleaning the model" from "learning to look better" but there's broad working principles, and as of now I think we should try to look relatively directly if we can
it shouldn't be so hard to test, you can just start looking at things and iterating and trying to understand motifs and such and see if it just breaks down at some point. it can take some iteration, I would recommend a bias against assuming arbitrary mixtures when you don't understand something at first, but also explanations should be tested
for now I'm pretty optimistic, and I'm glad there's excitement about interpretability, but too much can cause problems too. I think part might be that ai safety isn't a very hopeful field, and even moderate amounts of optimism about an important problem being solvable in the future fills a spot that has been malnourished for a while. across several safety disciplines, the researchers directly working on them are often more optimistic than the people broadcasting pessimism more broadly
for neural network interpretability at least, these are extraordinarily experimentally accessible objects of science, I don't know of a better one, and we have automated scientists. we have many things going in our favor. it's important and it's fun. the circuitry of these machines can be beautiful, surprising, entertaining, and insightful. I've spent thousands of hours looking through them, asking questions and running experiments and testing things, and would do so even if it weren't so important. I think the problem is solvable, whether the simple thing works, it mostly works but we need some new things, or we need big new ideas. I think we can figure it out
63
105
1,504
131,947
we added native here.now support for wasm builds, so you can host your ports simply by asking your agent to publish to here.now
here's the obligatory doom link doom-wasm.adam.here.now
here's the obligatory doom link doom-wasm.adam.here.now
2
2
19
2,083
this should reduce everyone's p(doom) by a lot...
before: "neurons are firing for all sorts of reasons that are alien and already beyond our comprehension"
actually: "oh wait, it's just because those two concepts were shaped like the letter V and we didn't notice till now"
before: "neurons are firing for all sorts of reasons that are alien and already beyond our comprehension"
actually: "oh wait, it's just because those two concepts were shaped like the letter V and we didn't notice till now"
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com
9
15
257
11,615
adam ludwin retweeted
Preston is one of the best Roblox developers in the world.
Roblox is uniquely positioned to benefit from AI-driven creation.
its house style means AI-generated games can feel native to the platform.
players don’t care how games are made, just if they’re fun.
Roblox is uniquely positioned to benefit from AI-driven creation.
its house style means AI-generated games can feel native to the platform.
players don’t care how games are made, just if they’re fun.
@KarmaKlubPSX *every* new top game right now is 100% AI. has been for a few months... and some of them are really fucking fun
i think everyone has great ideas. go make something
9
9
286
28,665
I used GPT-Live-1 to build a guided meditation experience that I've wanted to exist for years but could not build because the voice models were not good enough
I think we're finally there! DM me if you want to try it (it's expensive to run so can't release it widely yet)
I think we're finally there! DM me if you want to try it (it's expensive to run so can't release it widely yet)
9
0
33
3,861
adam ludwin retweeted
This is a japanese house-tour landing page made in three.js with Fable 5.1 using the dream-loop skill.
Live site: seijaku.mengto.here.now
It's crazy how realistic it looks even though it's all code and under 1 MB (except images). You can tour every room, switch between seasons, the house, and watch koi swim through realistic water.
The secret sauce to ultra-realistic 3d in code is a loop like this: dream-loop gets the agent to generate a target image, compare the live 3d against it with a separate critic, and keep refining the result.
Skill: github.com/achimala/dream…
Let me know if I should open-source the page.
Live site: seijaku.mengto.here.now
It's crazy how realistic it looks even though it's all code and under 1 MB (except images). You can tour every room, switch between seasons, the house, and watch koi swim through realistic water.
The secret sauce to ultra-realistic 3d in code is a loop like this: dream-loop gets the agent to generate a target image, compare the live 3d against it with a separate critic, and keep refining the result.
Skill: github.com/achimala/dream…
Let me know if I should open-source the page.
32
50
589
68,408
adam ludwin retweeted
worrying about recursive self-improvement on australian TV (lateline) in 1996.
143
567
4,037
377,002
adam ludwin retweeted
I recently went through a translation of Musashi’s Book of Five Rings.
The introduction chronicled the evolution of swordsmanship and martial arts in general from pragmatic battlefield necessities to sports, historical curiosities, and hobbies. It was interesting to hear how in the post-WW2 occupation when martial arts were banned, Judo was represented as equivalent to western wrestling, and Kendo later to western fencing.
The poignant undercurrent for me is that I can see many programming skills following that path.
There are probably dozens of people reading this that remember hand assembling opcodes to hex and still have some magic numbers burned into their memory, but even the small group of people still programming in assembly today (hey, @FFmpeg !) don’t work at that primitive level now.
AI is making many other programming skills much less critical. We aren’t there yet, but carefully writing code completely by hand is moving from a -jitsu to a -do. Code-do? Codo?
That’s ok! The retro computing scene is delightful, full of people building and exercising old skills for the love of it.
But don’t be the out of touch Kung Fu master, heir to lifetimes of tradition, that gets mauled by an amateur MMA fighter.
Musashi would probably have been pretty enthusiastic about assault rifles.
The introduction chronicled the evolution of swordsmanship and martial arts in general from pragmatic battlefield necessities to sports, historical curiosities, and hobbies. It was interesting to hear how in the post-WW2 occupation when martial arts were banned, Judo was represented as equivalent to western wrestling, and Kendo later to western fencing.
The poignant undercurrent for me is that I can see many programming skills following that path.
There are probably dozens of people reading this that remember hand assembling opcodes to hex and still have some magic numbers burned into their memory, but even the small group of people still programming in assembly today (hey, @FFmpeg !) don’t work at that primitive level now.
AI is making many other programming skills much less critical. We aren’t there yet, but carefully writing code completely by hand is moving from a -jitsu to a -do. Code-do? Codo?
That’s ok! The retro computing scene is delightful, full of people building and exercising old skills for the love of it.
But don’t be the out of touch Kung Fu master, heir to lifetimes of tradition, that gets mauled by an amateur MMA fighter.
Musashi would probably have been pretty enthusiastic about assault rifles.
310
462
4,648
7,143,206
adam ludwin retweeted
Jane street quant interviewing at Anthropic:
“So imagine our safety team believes a new model has a 5% chance of causing an irreversible loss of human control, but scaling it may be necessary to keep up with competitors. What do you do?”
“Who came up with five?”
“Our safety team.”
“And what does the capabilities team have?”
“What?”
“Same contract. Where are they?”
“This is more of a values question.”
“Then you don’t have a five. I would put the two teams in a room. Safety can buy at five. Capabilities can sell. Increase size until somebody becomes uncomfortable. Now we know something.”
He looks at me.
“This isn’t some market-making role.”
“Understood. Who makes the market currently?”
“Nobody makes a market.”
“Then how are you doing price discovery?”
“We don’t do price discovery. We have researchers studying alignment, interpretability, dangerous capabilities, scalable oversight— you know what I don’t think this is a good fit”
He closes his laptop.
“I’m concerned you see this as an opportunity to make money.”
“I haven’t quoted you yet.”
Interview ends early
“So imagine our safety team believes a new model has a 5% chance of causing an irreversible loss of human control, but scaling it may be necessary to keep up with competitors. What do you do?”
“Who came up with five?”
“Our safety team.”
“And what does the capabilities team have?”
“What?”
“Same contract. Where are they?”
“This is more of a values question.”
“Then you don’t have a five. I would put the two teams in a room. Safety can buy at five. Capabilities can sell. Increase size until somebody becomes uncomfortable. Now we know something.”
He looks at me.
“This isn’t some market-making role.”
“Understood. Who makes the market currently?”
“Nobody makes a market.”
“Then how are you doing price discovery?”
“We don’t do price discovery. We have researchers studying alignment, interpretability, dangerous capabilities, scalable oversight— you know what I don’t think this is a good fit”
He closes his laptop.
“I’m concerned you see this as an opportunity to make money.”
“I haven’t quoted you yet.”
Interview ends early
32
134
2,731
232,672
