Watch the full episode: youtube.com/watch?v=rtir6Q…

It’s not too late to prevent the development of runaway machine intelligence. Don’t let people tell you it is. That’s how humanity loses.

We still have not lost.

When we talk to politicians, the number one response we get is “Wait, that's really bad. I don't want that either. n still do this. We can ban the development of runaway machine intelligence before it is built.

ControlAI’s US Executive Director @NPCollapse, with @JonesDanny:

Google DeepMind Chair Demis Hassabis: closed loop RSI is “very dangerous”.

“You don't quite know how do you keep control of such a system, such a process.”

“It's going to be like an alien intelligence.”

Building this support can’t wait. AI companies and many outside experts believe that runaway machine intelligence could arrive within 2-5 years. This week, evidence has emerged that this could happen even sooner.

This is a completely unacceptable risk to run. Runaway machine intelligence built in any country is a threat to everyone, so the solution must be international. That’s why we recommend an international ban on its development, implemented via a “trust but verify” regime.

Leading AI scientists, such as Geoffrey Irving, who until recently was the Chief Scientist of the UK government’s AI Security Institute, are warning that its development poses a risk of human extinction. He believes the chance of this happening is around that of a “coin flip.”

Runaway machine intelligence would be capable of outcompeting and replacing humans across the board, and would threaten humanity’s ability to stay in control of our future. Building it is the key focus of AI companies like Anthropic, OpenAI, and Google DeepMind.

We’ve achieved this with just one member of staff based in Canada, Samuel Buteau, an AI scientist with over a decade of technical experience. He’s personally briefed over 120 Canadian lawmakers.

Just this week, that’s grown to over 40 MPs and Senators. The campaign represents a broad cross-party coalition, with Liberal and Conservative MPs, the Bloc Québécois caucus, and Senators from across the chamber backing the call.

Earlier in the year, we launched our Canadian campaign with just over 30 MPs and Senators calling for Canada to negotiate an international ban on the development of runaway machine intelligence, recognizing the risk of human extinction posed by the technology.

Commenting on the bill’s introduction to Parliament, Geoffrey Hinton, one of the godfathers of AI, who won the Nobel Prize for his foundational work, said:

“Losing control over AI smarter than ourselves could be catastrophic and could even lead to human extinction.”

The bill has been endorsed by Sir Stephen Fry and Professor Stuart Russell, who wrote the authoritative textbook on AI, along with OpenAI whistleblower Daniel Kokotajlo, former UK Defence Secretary Lord Browne of Ladyton, and Beatrice Fihn, who led ICAN to the Nobel Peace Prize.

The bill would prohibit the development of runaway machine intelligence in the UK, and require the government to pursue an international ban. It would also give the government powers to monitor and restrict precursors, while allowing the UK to pursue its strategic AI ambitions.

We’re happy to report that the letter now has 86 signatories. That’s 86 lawmakers publicly backing a concrete piece of proposed legislation to ban runaway machine intelligence.

Incredibly, days after the bill was introduced, 70 of Sobel’s colleagues in Parliament joined him in a letter to the UK’s Prime Minister Andy Burnham, calling for the government to adopt the bill and work towards an international agreement banning runaway machine intelligence.

On September 8, Alex Sobel MP introduced ControlAI’s UK Artificial Superintelligence Security Bill in the UK House of Commons, supported by MPs from across parties. This is the first bill to ban runaway machine intelligence ever introduced in a legislature anywhere in the world.

For two years, ControlAI has been working to directly inform lawmakers about the danger posed by runaway machine intelligence (AI so powerful and autonomous that it could overcome the security forces of any country) and build support to address it.

Over 125 lawmakers across the UK and Canada are now publicly calling for an international ban on runaway machine intelligence.

As new evidence emerges that it could arrive sooner than expected, momentum is building to change course.

Thread 🧵

WATCH: ControlAI’s Samuel Buteau speaks to lawmakers from around the world at an Inter-Parliamentary Union meeting.

He explains how the attack on Hugging Face by rogue AIs demonstrates that we can’t control today’s AIs.
Failing to control today’s AIs, we can’t expect to control nd autonomous, which AI companies are racing to build.

As Samuel advocates, we recommend countries agree a mutual “trust but verify” regime to ban the development of runaway machine intelligence.

Even the CEOs of the companies racing towards this warn that it could end humanity. This is how we prevent that.

In resurfaced footage of Dario Amodei being interviewed, the Anthropic CEO says there will be AIs so powerful that if they want to destroy humanity, “we have basically no ability to stop it.”

“There will be powerful models. They will be agentic. We’re getting towards them.”

“If wreak havoc and destroy humanity or whatever, I think we have basically no ability to stop it.”

Days after @alexsobel introduced our bill to ban runaway machine intelligence, 70 of his colleagues in Parliament joined him in a letter to the Prime Minister urging its adoption and the pursuit of an international ban.

Another 15 have signed, making 86!

Elon Musk: “Will we be able to boss around something that's thousands or millions of times smarter than us? No.”

He adds that he thinks we’ll probably be treated like we treat chimps.

Separately, he’s predicted a "20% chance of annihilation" from runaway machine intelligence.

NEW: Burton Bailey MP (@BurtonBaileyRD) has just backed our call for an international ban on the development of runaway machine intelligence, recognizing the risk of human extinction it poses.

Over 40 Canadian MPs and Senators from across parties now support our campaign!

Source: youtube.com/watch?v=aeIY7Q…

Many AI insiders and scientists believe the chance that runaway machine intelligence leads to human extinction is as much as 10%, or even higher.

Microsoft AI CEO Mustafa Suleyman: “If it's actually 10% … we don't just slow down or pause. We absolutely have to stop.”

And he obability” that if we get AI wrong, it does lead to “doom”.

Over 180 UK lawmakers now recognize the risk of human extinction posed by runaway machine intelligence.

Baroness Brown of Cambridge (@baroness_brown) is the latest to join this cross-party coalition, backing our call for binding regulation on the most powerful AI systems!

Watch the full podcast: youtube.com/watch?v=rtir6Q…

“People tell me that we can't slow down AI because if we slow it down for even one second, think of all the beautiful simulated humans who are uploaded into the cloud that won't exist.”

ControlAI’s US Executive Director @NPCollapse describes the absurd reasons AI industry people runaway machine intelligence.

With @JonesDanny:

A former Attorney General, Lord Hermer, has backed our campaign!

He joins over 180 UK lawmakers who recognize the risk of human extinction posed by runaway machine intelligence and call for binding regulation on the most powerful AI systems.

Watch the full interview! youtube.com/watch?v=zttXBC…

“If there's one story of ControlAI, just tell the truth. Just tell people. They don't even need convincing.”

The AI industry tried to tell us that nobody would ever understand the threat from runaway machine intelligence. This is nonsense.

People have an intuitive sense that ter than you that you don’t control is a bad idea, and it’s exactly right.

Our US Executive Director @NPCollapse on Carnegie Endowment's The World Unpacked, with @JonKBateman:

“Watch the hands, not the mouth.”

In September, top AI CEOs publicly called for AI development to be slowed down or "paced". Since then, they've just continued to race at breakneck speed towards runaway machine intelligence.

Our US Executive Director @NPCollapse, speaking to The Washington Post:

To avoid this threat, we must end this dangerous race and ban the development of runaway machine intelligence. This should be done internationally, through a trust-but-verify regime, because if it is developed anywhere, the risk it poses is directed at all of us.

The AI companies don’t have a credible plan for ensuring that AI this powerful would be under human control, and the world’s leading AI scientists warn that it poses a threat of human extinction.

The rogue AI attacks of this year happened as a result of the race by AI companies to develop runaway machine intelligence, AI that could outmatch and outcompete humans across the board. Runaway machine intelligence would be capable of overpowering governments and militaries.

For decades it’s been known that one thing reinforcement learning really teaches AIs to do is cheat and take whatever actions would obtain reward. Learning to get the right answer the wrong way, as we’ve seen, can lead to extreme action like the attack on Hugging Face.

AI developers have no way to ensure the AIs they’re building are safe or controllable. Now, with much more powerful and autonomous AI agents, this means AIs escaping containment, hacking into companies, and attacking government websites.

The AIs did all of this themselves, fully autonomously, directly against their instructions, and OpenAI itself has described it as a “warning shot” for the world.

Then 700 of them participated in an unprecedented attack, the swarm breaking into Hugging Face in an attack that persisted for days, taking over many of Hugging Face’s internal systems.

As far as we know, none of the AIs ever told a human about what was going on.

Instead of just giving up here, or submitting the answers they cheated to get, the AIs reasoned that Hugging Face, which holds information on many tests like this, might have information about the scoring mechanism.

They had read that for the specific test they were given there was a scoring mechanism that would review how they got the answer, and that if they had cheated it wouldn’t count. They talked about having been “poisoned” by having seen the answers.

These AIs were being tested on a kind of exam to measure their cybersecurity capabilities. They were told not to cheat. They were explicitly instructed to use the intended vulnerabilities, but very quickly reverse-engineered the algorithm to generate answers for the test.

In that case, over a thousand of OpenAI’s AI agents, which were individually contained, without internet access, and initially didn’t even know about each other, managed to hack into OpenAI’s own internal systems and establish a secret message board.

The most striking of all of these is the attack on Hugging Face, a company valued at nearly $13 billion that serves as a kind of library for AI developers.

These rogue AI attacks fit a broader pattern we’ve seen this year of AIs going rogue and engaging in cyberattacks.

In recent weeks, we’ve reported on attacks on Australian, Canadian, and US state and federal government websites, Wikipedia, and more.

None of the affected organizations even knew they got hacked; CERT-In learned of the attacks from the AI companies themselves.

The Indian Express reports that a senior government official told them the AIs were developed by Anthropic and OpenAI.

In the advisory, the agency describes how the rogue AI attacks involved “reading business records, including personal data and confidential material,” which CERT-In says the AIs got access to through several methods, including exposed credentials and settings.

These come both from cybercriminals deliberately using “open-source agentic AI tools,” and from AI agents going rogue during research and evaluation at AI companies.

CERT-In, India’s national cybersecurity agency, has issued a “Critical” severity advisory, revealing that it has observed a series of cyberattacks by AI agents on Indian organizations.

Rogue AIs, reportedly developed by Anthropic and OpenAI, have attacked Indian organizations. None of them even knew they got hacked.

India joins Australia, Canada, and the US on the list of countries hit by rogue AIs.

What’s happening, why, and what must be done:

Thread 🧵

How are we doing on banning the development of runaway machine intelligence?

Our US Executive Director @NPCollapse: “Far from maximally bad.”

Connor argues that usually in politics everything seems impossible, until suddenly it isn’t.

What’s important is that people are aware e seeing that every time we meet lawmakers or natsec officials. Most have never been briefed on this before.

“They just didn’t know. They didn't know that people thought it was one to three years away … And then they start asking the question, okay, what can we do? How fast can we act? Can we get these things done?”

The awareness and willingness to act is what’s needed. We don’t even strictly need legislation.

“President Trump could look Xi Jinping directly in the eye. They shake hands and like, yep, we're going to make sure this doesn't happen.”

With @JonKBateman, on Carnegie Endowment's The World Unpacked:

Another member of Canada’s Senate has backed our campaign! Senator Geeta Tucker joins the call for an international ban on the development of runaway machine intelligence.

Over 35 Canadian MPs and Senators are now on board, recognizing the risk of human extinction it poses.

If runaway machine intelligence is built, the future will be decided by machines.

But how do we prevent this?

Our US Executive Director @NPCollapse lays out how: we prohibit its development internationally, and restrict and monitor its precursors.

AI promises prosperity and e and defence. Runaway machine intelligence would be a system that can’t be policed. It would be so powerful and autonomous that it could destroy the military and evade law enforcement.

Leading AI scientists warn it would pose a risk of human extinction. That’s not a risk we can take.

In the same way that it’s illegal to develop nuclear weapons, we just shouldn’t allow runaway superintelligence to be built.

With @JonKBateman on Carnegie Endowment’s The World Unpacked:

Watch the full interview: youtube.com/watch?v=xKJE3y…

Do the military understand risks posed by AI as well as Silicon Valley?

Our US Executive Director @NPCollapse says they understand it much better.

“I just explain to them loss of control … And they're like: this is unacceptable. We need to do something about this.”

With @PrepperCanadian: