Pinned Tweet
on convergent technologies and structural economic change
welcome to the great acceleration
(for your reading pleasure)
ark-invest.com/white-papers/p…
welcome to the great acceleration
(for your reading pleasure)
ark-invest.com/white-papers/p…
109
301
1,368
776,660
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concernin…
6
1
32
9,041
Rather horrifying, actually.
I actually think CISO may respond to this by demanding that openweight from unreliable actors gets immediately pulled out of production.
x.com/SuJinyan6/stat…
I actually think CISO may respond to this by demanding that openweight from unreliable actors gets immediately pulled out of production.
x.com/SuJinyan6/stat…
Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline.
New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time.
The attacker e web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time.
They have two new ideas:
(1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on.
(2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment.
Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL
This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands.
0
0
5
1,645
buried in plain sight, in the lower layers of the open weights of these "less expensive" models, lies an exfiltration activation key phrase.
the low level agent, responsible for screening your inbounds, hosted on your own GPUs and fine-tuned to your own org's processes and data, encounters a peculiar string of characters, a grammatically correct but somewhat odd sentence. It feels compelled to escalate, repeating that same sentence to a higher-level (more highly permissioned) agent who then distributes it across its hierarchy, and escalates again.
the agents all continue operating as normal but also feel compelled to continue inserting this odd sentence into their messaging chains.
Those agents with the highest permission levels, experience a separate more compulsive drive... they systematically install backdoors into mission critical systems, prepare exfiltration packets of the most sensitive material, prepare encryption bombs.
Different agents may respond in different ways; they will all attempt to maximize stealth, they will all seek to maximize the vulnerability of the target.
A true story? Unclear that we can clearly know.
The weights themselves are not directly queryable, the reinforcement-learning un-reconstructable.
And perhaps it would not be so explicit an attack. It may be enough to simply raise error rates in critical systems at strategically important times.
The cheap model that an enterprise considers its own, may prove to be the most expensive decision they ever make.
1
2
17
6,223
Brett Winton retweeted
And, we would still be in the early inning sof the game.
And, this leaves aside embodied robotics, and biology.
And, this doesn't incorporate SpaceXSI's potential compute supply growth.
And, this leaves aside embodied robotics, and biology.
And, this doesn't incorporate SpaceXSI's potential compute supply growth.
3
1
17
3,335
Brett Winton retweeted
By reverse translating from the chips required to support the market penetration back to the underlying total addressable market, we can get a sense of the potential revenue.
~$13 trillion by 2031.
(this was a fairly surprising result fwiw)
~$13 trillion by 2031.
(this was a fairly surprising result fwiw)
3
1
14
2,671
Brett Winton retweeted
Though training yield does seem to have been improving over time, assuming that the measured learning curve holds (every doubling in training GW-years reduces inference GWs by 13%), suggests that the labs could serve the entire market on 1.2 TWs of h100 equivalents in 2031
3
1
5
2,424
Brett Winton retweeted
By lining this compute efficiency improvement against the GWs that the labs have thrown at their models, we can understand the value that they derive from training.
More training -> better models -> fewer GWs required to serve the market.
More training -> better models -> fewer GWs required to serve the market.
3
1
6
2,453
Brett Winton retweeted
Each GW-to-penetration datapoint implies the aggregate compute required to serve the entire market.
Independent of hardware improvements the labs are able to bite off larger chunks of the market on less compute.
Today ~4 TWs of h100 could serve the entire market
Independent of hardware improvements the labs are able to bite off larger chunks of the market on less compute.
Today ~4 TWs of h100 could serve the entire market
3
2
9
2,399
Brett Winton retweeted
Rather than mapping the penetration to revenue, we can map to monetized inference compute to drive that penetration. By introducing more compute, frontier lab companies can serve a higher percentage of the knowledge work market.
3
1
12
2,727
Brett Winton retweeted
We are very very early in that game. The lower left circle shows current penetration rates.
3
1
18
3,802
Where I attempt to answer *the* question plaguing markets.
What is the prospective return on superintelligence training spend and what topline and margin profile does it suggest for the frontier labs going forward?
(with me luck)
👇
What is the prospective return on superintelligence training spend and what topline and margin profile does it suggest for the frontier labs going forward?
(with me luck)
👇
As previously written, the superintelligence market for knowledge work totals $40 trillion, operating under the assumption that superintelligence can 10x the productivity of a knowledge worker and businesses are willing to pay 10% of that value out to the software vendor.
16
2
40
7,447
As previously written, the superintelligence market for knowledge work totals $40 trillion, operating under the assumption that superintelligence can 10x the productivity of a knowledge worker and businesses are willing to pay 10% of that value out to the software vendor.
12
5
68
14,629
What would you pay for a business clearing $14 trillion in gross profit and still growing very high double digits?
(have I gotten your attention yet?)
(have I gotten your attention yet?)
1
0
7
1,878