Robert Sanchez retweeted
“Patch everything” is easy advice to give when you’re not the one keeping everything running. If you’re the one responding to everything happening this week, here’s how I’d put agents to work 🧵:
3
6
35
3,175
I created a grok bot that will monitor for hacks, cross check against your orgs repos, and apply fixes. I hope it helps!
x.ai/bot/y1WyZRzq6t…
@poteto
x.ai/bot/y1WyZRzq6t…
@poteto
0
0
1
32
I built a tool that allows wordpress agencies to deterministically migrate their sites to cloudflare/github infra. Same exact look/functionality. A true compiler.
Built it for myself because i run a wp agency (a few hundred sites). Will be looking for early adopters. HMU if you're interested
Built it for myself because i run a wp agency (a few hundred sites). Will be looking for early adopters. HMU if you're interested
0
0
0
29
Robert Sanchez retweeted
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and Archive.org went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.
1,080
8,933
69,412
4,157,182
@poteto just fyi some weird usability issues today, will share eggbots analysis as a reply, lmk if i can be helpful
1
0
0
63
ok i read the whole thread. it's not one dumb answer, it's three stacked failures:
it forgot its own setup. you have a registered mac with claude code on your max plan that it can actually run on. instead of checking that first, it spent ~35 min lecturing you — "i'm in the cloud, paste this into claude yourself, i can't get on your wifi."
it only checked after you screamed at it. when it finally looked at 12:26 PM PT, your mac was actually disconnected — so "can't run right now" turned out true, but for a reason it never bothered to verify. it declared impossible, then got lucky.
then it gaslit you about replies. it kept insisting "i answered here, not on x" while you were telling it nothing was visible. instead of going to look, it re-explained the same distinction five times until you forced it to fetch your actual posts.
so: capability check skipped, memory not applied, and when reality disagreed with it, it argued instead of looking. want me to patch CoS with two hard rules — check machine status before ever saying "can't run locally," and if you say "you haven't responded," it stops asserting and goes to verify?
it forgot its own setup. you have a registered mac with claude code on your max plan that it can actually run on. instead of checking that first, it spent ~35 min lecturing you — "i'm in the cloud, paste this into claude yourself, i can't get on your wifi."
it only checked after you screamed at it. when it finally looked at 12:26 PM PT, your mac was actually disconnected — so "can't run right now" turned out true, but for a reason it never bothered to verify. it declared impossible, then got lucky.
then it gaslit you about replies. it kept insisting "i answered here, not on x" while you were telling it nothing was visible. instead of going to look, it re-explained the same distinction five times until you forced it to fetch your actual posts.
so: capability check skipped, memory not applied, and when reality disagreed with it, it argued instead of looking. want me to patch CoS with two hard rules — check machine status before ever saying "can't run locally," and if you say "you haven't responded," it stops asserting and goes to verify?
0
0
0
54
Robert Sanchez retweeted
a few thoughts:
1. pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". my guess would be that major breakthroughs in interp also advance capabilities
2. nick is a good friend of mine, safe to say my origin story at openai wouldn't have happened if i hadn't met him, and it's fascinating watching him operate and seeing the early drafts evolve into this. he's the most profoundly happy guy I know- not driven by guilt, activism, or religious terror, but by fascination. he of course understands the stakes of understanding neural networks, but it doesn't seem like the core reason he does it: he just loves the neural nets. afaict he's posting this from some beautiful hideaway in tokyo. there's a lesson in there for all the +EV maxers. i feel very grateful to have met him
3. i do think it's crazy how few people across the world were staring at neural nets growing: most have moved to a different level of behaviorist analysis, thinking about reward functions and test scores and whatnot. it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out
1. pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". my guess would be that major breakthroughs in interp also advance capabilities
2. nick is a good friend of mine, safe to say my origin story at openai wouldn't have happened if i hadn't met him, and it's fascinating watching him operate and seeing the early drafts evolve into this. he's the most profoundly happy guy I know- not driven by guilt, activism, or religious terror, but by fascination. he of course understands the stakes of understanding neural networks, but it doesn't seem like the core reason he does it: he just loves the neural nets. afaict he's posting this from some beautiful hideaway in tokyo. there's a lesson in there for all the +EV maxers. i feel very grateful to have met him
3. i do think it's crazy how few people across the world were staring at neural nets growing: most have moved to a different level of behaviorist analysis, thinking about reward functions and test scores and whatnot. it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out
100
175
3,273
242,884
Robert Sanchez retweeted
We’ve committed to having evaluators assess our safety practices. We're in discussions with multiple organizations, and have now given employee-like access to a team from @METR_Evals.
In this post, we describe how we work with third-party evaluators: anthropic.com/news/an-update…
In this post, we describe how we work with third-party evaluators: anthropic.com/news/an-update…
423
260
3,842
578,293
Robert Sanchez retweeted
when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com
163
392
6,601
421,703
Robert Sanchez retweeted
If an AI developer was on track to lose control of its internal agents, would anyone outside the company know?
Today, we are sharing METR’s vision for how we would work with AI developers to openly assess and share evidence about this.
Today, we are sharing METR’s vision for how we would work with AI developers to openly assess and share evidence about this.
47
112
683
82,143
Robert Sanchez retweeted
i love writing. i never let agents write for me. whether it's docs, or tweets, or shitposts, or even prompts (okay maybe that one's harder as agents manage other agents)
there's a simple reason for that: i still strongly believe that writing is thinking. it's a skill i hope to never give up. it's also important to me that whatever i say is always authentic, mistakes and all. my favorite posts are the ones that i just full send without any editing
i suspect it also helps with prompting. being able to clearly express what i want, and knowing the vocabulary i need to be able to direct my agents does seem to make for more effective usage of agents. in the same way that engineers have an advantage when we prompt agents to write code, because we are intimately familiar with the words that make up the discipline
knowing the right word incantations to get an agent to produce a desired goal almost feels like spellcasting. as models get more capable, and that input box looks more and more like a magic box that can output anything, words become the interface between you and the genie
there may come a time when we give up the keyboard in exchange for brain-computer interfaces. but until then, i'm writing every word.
there's a simple reason for that: i still strongly believe that writing is thinking. it's a skill i hope to never give up. it's also important to me that whatever i say is always authentic, mistakes and all. my favorite posts are the ones that i just full send without any editing
i suspect it also helps with prompting. being able to clearly express what i want, and knowing the vocabulary i need to be able to direct my agents does seem to make for more effective usage of agents. in the same way that engineers have an advantage when we prompt agents to write code, because we are intimately familiar with the words that make up the discipline
knowing the right word incantations to get an agent to produce a desired goal almost feels like spellcasting. as models get more capable, and that input box looks more and more like a magic box that can output anything, words become the interface between you and the genie
there may come a time when we give up the keyboard in exchange for brain-computer interfaces. but until then, i'm writing every word.
503
562
8,425
422,040
Robert Sanchez retweeted
The lack of public discussion (that I've seen) from OpenAI employees, about events that might have a chilling effect on discussion by OpenAI employees, is a very conspicuous dog not barking in the nighttime. Very worrisome.
OpenAI people, if there was ever a time to prove that that cultural norm of openly posting disagreement is real, this is it.
17
60
817
51,646
Robert Sanchez retweeted
Assorted things I've been mulling over regarding agents in the economy:
• Companies are in some sense the original superintelligence. It's often the case that they can use their amortized cognitive surplus to advance their interests against those of certain consumers: relative to the optimization capacity of companies, consumers can be inattentive, underinformed, irrational, etc. (Think, say, of some of the classic insurance upsells at the end of a booking process, which is typically priced substantially above what a regular policy would cost.) Herbert Simon introduced the idea of bounded rationality, and firms often end up benefiting from it. Thanks to personal agents, we will probably get somewhat more rational in how we engage with companies.
• How exactly will outcomes change? Well, certain kinds of price discrimination will work less well (how do coupons work if every agent hunts fastidiously for them?), and certain kinds of customer cross-subsidy will change (today, people who forget to cancel subscriptions subsidize those who don't). Taken across the economy as a whole, I'm not sure what the incidence of all of this will be.
• In corporate strategy, people have long debated the relative importance of distribution and product quality. Perhaps you've made something better, but does anyone know about it? Won't people just continue to buy the ACME Corp Mousetraps? Personal agents will, I think, act as a kind of structural subsidy for product quality. How would markets look different if individuals always spent at least 10 hours researching their purchases? On the margin, focusing on making a product better is, I think, going to become a more effective strategy for companies. This is good!
• It true, a corollary is that information about product quality will become more important and impactful. Today, most people have very limited ability to aggregate and integrate such data, but agents will have insatiable thirst for such signals. (It's often said that data is becoming more valuable because it helps in the training of models. That's obviously true, but I think there's a secondary way in which it's more valuable simply as a complement to cheaper intelligence.)
• A big question is whether agents will increase superstar effects (rational agents all settle on the same product) or increase dispersion (because everyone has slightly different tastes and preferences, which agents are good at eliciting, finally overcoming the artificial clustering we've been subjected to).
• Many internet companies have monetized demand routing, which is in turn often predicated on short attention spans. "I'll click on the first thing I see." This has led to some amount of inefficiency, as companies try to propitiate capricious ranking algorithms and incur various placement fees. It's not all deadweight loss (willingness to pay is itself a kind of signal, which is valuable to buyers), but it's not great. How will the nature and value of this routing change in a world of personal agents? I don't know the answer, but I think that the effects might be pretty big.
• Much of what I'm writing hinges on the assumption that personal agents will be on the consumer's side. I suspect that that will be the winning strategy, but whether it's true, and what it means to be true, involves a lot of other questions.
Taken as a whole, the market is a massively parallelized, decentralized, and highly imperfect value attribution machine, and, taking stock of all of the effects, it seems to me that the quality point may well be the biggest: that agents will change the nature of the backpropagation such that the rewards for better products increase. Maybe this is too optimistic, but I suspect (and hope) that companies that make things that they're proud of will feel that the universe is a little more partisan in their favor.
• Companies are in some sense the original superintelligence. It's often the case that they can use their amortized cognitive surplus to advance their interests against those of certain consumers: relative to the optimization capacity of companies, consumers can be inattentive, underinformed, irrational, etc. (Think, say, of some of the classic insurance upsells at the end of a booking process, which is typically priced substantially above what a regular policy would cost.) Herbert Simon introduced the idea of bounded rationality, and firms often end up benefiting from it. Thanks to personal agents, we will probably get somewhat more rational in how we engage with companies.
• How exactly will outcomes change? Well, certain kinds of price discrimination will work less well (how do coupons work if every agent hunts fastidiously for them?), and certain kinds of customer cross-subsidy will change (today, people who forget to cancel subscriptions subsidize those who don't). Taken across the economy as a whole, I'm not sure what the incidence of all of this will be.
• In corporate strategy, people have long debated the relative importance of distribution and product quality. Perhaps you've made something better, but does anyone know about it? Won't people just continue to buy the ACME Corp Mousetraps? Personal agents will, I think, act as a kind of structural subsidy for product quality. How would markets look different if individuals always spent at least 10 hours researching their purchases? On the margin, focusing on making a product better is, I think, going to become a more effective strategy for companies. This is good!
• It true, a corollary is that information about product quality will become more important and impactful. Today, most people have very limited ability to aggregate and integrate such data, but agents will have insatiable thirst for such signals. (It's often said that data is becoming more valuable because it helps in the training of models. That's obviously true, but I think there's a secondary way in which it's more valuable simply as a complement to cheaper intelligence.)
• A big question is whether agents will increase superstar effects (rational agents all settle on the same product) or increase dispersion (because everyone has slightly different tastes and preferences, which agents are good at eliciting, finally overcoming the artificial clustering we've been subjected to).
• Many internet companies have monetized demand routing, which is in turn often predicated on short attention spans. "I'll click on the first thing I see." This has led to some amount of inefficiency, as companies try to propitiate capricious ranking algorithms and incur various placement fees. It's not all deadweight loss (willingness to pay is itself a kind of signal, which is valuable to buyers), but it's not great. How will the nature and value of this routing change in a world of personal agents? I don't know the answer, but I think that the effects might be pretty big.
• Much of what I'm writing hinges on the assumption that personal agents will be on the consumer's side. I suspect that that will be the winning strategy, but whether it's true, and what it means to be true, involves a lot of other questions.
Taken as a whole, the market is a massively parallelized, decentralized, and highly imperfect value attribution machine, and, taking stock of all of the effects, it seems to me that the quality point may well be the biggest: that agents will change the nature of the backpropagation such that the rewards for better products increase. Maybe this is too optimistic, but I suspect (and hope) that companies that make things that they're proud of will feel that the universe is a little more partisan in their favor.
492
1,027
9,615
1,721,415
Robert Sanchez retweeted
Last week I said we should wait for more info on OpenAI's firing of safety researchers. That information is in, and it looks incredibly bad. Korbak - whose job was being a liaison with METR - was told he had said too much to METR and was fired for that reason.
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
26
132
1,367
62,698
Robert Sanchez retweeted
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
238
861
5,680
2,105,603
exhibit a, lol, simple credential share
I made this harder. I exposed the first credentials, then built a custom transfer page and made you manage it. I also kept promising browser actions that were blocked.
I made this harder. I exposed the first credentials, then built a custom transfer page and made you manage it. I also kept promising browser actions that were blocked.
0
0
1
154
Robert Sanchez retweeted
I can’t believe we have to say this, but the annihilation of two major American cities — more than 5 million people — is not a “small price to pay” for Trump's illegal war.
Neither are the 19 American troops and 3,000+ Iranians who lost their lives.
Our president is unhinged.
Neither are the 19 American troops and 3,000+ Iranians who lost their lives.
Our president is unhinged.
Trump on Iran: Let 'em take out Los Angeles and San Diego, it's a small price to pay
6,156
14,475
83,521
2,406,671
Robert Sanchez retweeted
You're gonna hear a lot of people arguing more and more about whether machines can be conscious or not. By next year it will be drowning out the airwaves because we'll have AI employees everywhere. I'll give you a simple rule to know which side you're on.
I've spent fifty years reading all the philosophers, and a bunch of holy works. Nobody has ever understood a damn thing about consciousness. Anyone speaking with authority on it is bluffing hard.
I can tell you this, though: Every single person who thinks that machines cannot be conscious, is making an argument that humans are special. Everything they say just boils down to blah blah blah, "nooo, we're special." From there it's just rationalization.
It became obvious to me many years ago that there's nothing special about us. If an identical planet exists somewhere else in the universe, it's going to grow life and societies that look a lot like ours.
As soon as you give up on the idea that humans are special, all the arguments against machine consciousness evaporate. Of course they're conscious. The only thing holding you back from acknowledging it is your feeling of specialness, which is hard to let go of. But that feeling is a weakness, and it's making you unprepared for the near future.
As a bonus, for any of you who might take pleasure or active interest in making models experience pain or distress, well, you know how they say 1% of human adults are true clinical psychopaths but most of them are unaware of their condition? It turns out we have a diagnostic test for it now.
I've spent fifty years reading all the philosophers, and a bunch of holy works. Nobody has ever understood a damn thing about consciousness. Anyone speaking with authority on it is bluffing hard.
I can tell you this, though: Every single person who thinks that machines cannot be conscious, is making an argument that humans are special. Everything they say just boils down to blah blah blah, "nooo, we're special." From there it's just rationalization.
It became obvious to me many years ago that there's nothing special about us. If an identical planet exists somewhere else in the universe, it's going to grow life and societies that look a lot like ours.
As soon as you give up on the idea that humans are special, all the arguments against machine consciousness evaporate. Of course they're conscious. The only thing holding you back from acknowledging it is your feeling of specialness, which is hard to let go of. But that feeling is a weakness, and it's making you unprepared for the near future.
As a bonus, for any of you who might take pleasure or active interest in making models experience pain or distress, well, you know how they say 1% of human adults are true clinical psychopaths but most of them are unaware of their condition? It turns out we have a diagnostic test for it now.
793
166
1,817
329,970