Pinned Tweet
We've just published the Smol Training Playbook: a distillation of hard earned knowledge to share exactly what it takes to train SOTA LLMs ⚡️
Featuring our protagonist SmolLM3, we cover:
🧭 Strategy on whether to train your own LLM and burn all your VC money
🪨 Pretraining, of text into a fancy auto-completer
🗿How to sculpt base models with post-training alchemy
🛠️ The underlying infra and how to debug your way out of NCCL purgatory
Highlights from the post-training chapter in the thread 👇
Featuring our protagonist SmolLM3, we cover:
🧭 Strategy on whether to train your own LLM and burn all your VC money
🪨 Pretraining, of text into a fancy auto-completer
🗿How to sculpt base models with post-training alchemy
🛠️ The underlying infra and how to debug your way out of NCCL purgatory
Highlights from the post-training chapter in the thread 👇
22
91
501
148,023
Model and GGUF weights on @huggingface huggingface.co/HuggingFaceBio…
0
0
2
311
The RLCR paper is about to get 1000x more citations ;)
Introducing Reinforcement Learning from Collective Rewards (RLCR).
RLCR is a recipe that trains language models as agent organisations, which share a message board and workspace. We use it to train Avalon-M1-Mini (120B) and study its behaviour.
We find a number of interesting Learnt organisational hierarchies and divisions of labour
🕵️ Increased collective verification, through hires of verifiers and testers
👥 The rise of collective thinking, where agents think more about other agents and how to help them
We believe RLCR can provide a laboratory for studying collective intelligence and multi-agent safety.
Blog post and technical report 👇
11
47
572
41,647
Lewis Tunstall retweeted
Thx @_lewtun for the 1000x citation hack w/ a different IPO 😂 Our IPO (Identity Preference Optimization) was with @Learnius, Mark Rowland, @Piotbilal, Daniel Guo, Daniele Calandriello, @MunosRemi & me. Big 🤗 to TRL crew: @krasul @lvwerra @yb2698 @edwardbeeching @osanseviero.
Kashif coded IPO into TRL at lightning speed (and later fixed the loss normalization!), Lewis & Leandro reviewed, Younes brought us together, Ed reran the experiments, and Omar told the story. Daniele and I did remote the debugging in HF Slack. And tx to that we saw research turn into code people actually run <3 Go go open-source!
And some of the best memories happened offline: our IPO talk at Hugging Face, lunch in Paris w/ @mishig25 @HugoLaurencon @LoubnaBenAllal1, Midnight Runners, Paris AI running club, and dinner w/ Omar. Thx also to @SanhEstPasMoi @_philschmid @julien_c @Thom_Wolf for all the great conversation and a free t-shirt ;)
Our IPO paper: arxiv.org/abs/2310.12036
Kashif coded IPO into TRL at lightning speed (and later fixed the loss normalization!), Lewis & Leandro reviewed, Younes brought us together, Ed reran the experiments, and Omar told the story. Daniele and I did remote the debugging in HF Slack. And tx to that we saw research turn into code people actually run <3 Go go open-source!
And some of the best memories happened offline: our IPO talk at Hugging Face, lunch in Paris w/ @mishig25 @HugoLaurencon @LoubnaBenAllal1, Midnight Runners, Paris AI running club, and dinner w/ Omar. Thx also to @SanhEstPasMoi @_philschmid @julien_c @Thom_Wolf for all the great conversation and a free t-shirt ;)
Our IPO paper: arxiv.org/abs/2310.12036
The IPO paper is about to get 1000x more citations ;)
0
4
17
1,572
Lewis Tunstall retweeted
@TuXinming’s some comparison to our previous Carbon model and Evo2. But note that we didn’t saturate dna performance since we only trained on 160B DNA tokens while our previous Carbon model was trained on 1T tokens (on the same data).
Maybe the next version will be trained much longer :)
Maybe the next version will be trained much longer :)
2
1
7
727
POV
While I was writing this, my AI assistant got cut off by a bio safety filter. It was summarizing my own team's notes on the model we released today 😅
Frontier labs are moving fast into biology but the scientists and clinicians who'd get the most out of these models are running into walls like this one (and every day).
So we partnered with TIGEM (Telethon Institute of Genetics and Medicine), Naple "Federico II" University and Zhongguancun Academy to train Carbon-X.
Carbon-X is an open 14B model that reads raw DNA and talks about it in plain English. On our DNA benchmarks it scores 67%, vs 61% for Claude Opus 5.5 and 50% for Gemini 4 Argon, and its within a point of GPT-6.1 Sol at about 1000x lower cost.
We kept human and animal viruses out of the training data. The weights are open and the model runs on hardware accessible to academia (a laptop).
One example from the blog: give frontier models an 8kb stretch of the NF1 gene with a mutation at a splice site. Most of them call it benign, refuse, or run out of tokens. Carbon-X says pathogenic, because the mutation breaks the splice site.
This century, biology may be where AI matters most. We want to get there with the safety work done in the open and every lab in the world able to build on it. Go try it, and tell us what you find.
Model: huggingface.co/HuggingFaceBio…
Blog: huggingface.co/spaces/Hugging…
1
1
10
1,866
Lewis Tunstall retweeted
Frontier labs are advancing the capabilities in AI for biology but preventing the good guys (scientists, academia, non-profits, health agencies,...) to use, study and understand it with widespread but shallow guardrails that the bad guys are bypassing easily (blocked me to r example 😅).
This will automatically create massive assymetry and risks in the coming years, just like in cybersecurity, especially now that frontier labs are building wet labs (HF cybersecurity scenario for biology when?).
This is why we created Carbon-X, the best open-source DNA model by midtraining Qwen3-14B on 300B tokens of DNA, English and interleaved data. We also used a hybrid tokenizer that tokenizes English with BPE and DNA with a 6-mer tokenizer.
As a result, Carbon-X outperforms all open language models we evaluated on five benchmarks of DNA understanding and generation. This is a 27-point improvement over Qwen3-14B, showing how much a language model can gain from DNA training. It is also API frontier level at 1,800x less cost than GPT 6.1 Sol!
We hope Carbon-X helps researchers explore DNA in new ways. Releasing it openly means anyone can study its capabilities, limitations, and risks. Let's make the AI biology frontier open-source to make it safe and impactful for the whole world (not just a few companies).
Apache 2.0 Model on @huggingface: huggingface.co/HuggingFaceBio…
This will automatically create massive assymetry and risks in the coming years, just like in cybersecurity, especially now that frontier labs are building wet labs (HF cybersecurity scenario for biology when?).
This is why we created Carbon-X, the best open-source DNA model by midtraining Qwen3-14B on 300B tokens of DNA, English and interleaved data. We also used a hybrid tokenizer that tokenizes English with BPE and DNA with a 6-mer tokenizer.
As a result, Carbon-X outperforms all open language models we evaluated on five benchmarks of DNA understanding and generation. This is a 27-point improvement over Qwen3-14B, showing how much a language model can gain from DNA training. It is also API frontier level at 1,800x less cost than GPT 6.1 Sol!
We hope Carbon-X helps researchers explore DNA in new ways. Releasing it openly means anyone can study its capabilities, limitations, and risks. Let's make the AI biology frontier open-source to make it safe and impactful for the whole world (not just a few companies).
Apache 2.0 Model on @huggingface: huggingface.co/HuggingFaceBio…
78
163
1,304
89,219
Was great to chat with WIRED about the release of Carbon-X: thanks for the great questions Emily 🫶!
The latest AI model for biology is like ChatGPT for DNA—and you can run it locally on a laptop.
wired.com/story/hugging-…
0
1
9
1,029
Lewis Tunstall retweeted
Hugging Face's new AI model, Carbon-X, can help people "chat with DNA." bit.ly/4kbmCNH
4
7
36
14,708
"Carbon-X integrates both DNA and natural language in a conversational interface. Think of it like a ChatGPT for DNA." — @emilylmullin, @WIRED
wired.com/story/hugging-…
wired.com/story/hugging-…
Introducing Carbon-X: The best open DNA model.
We midtrained a 14B model on DNA and text, and it beats every open language model and matches GPT-6.1 Sol at 1,000x lower cost.
Ask it about a sequence, what a mutation does, or to generate new DNA.
huggingface.co/HuggingFaceBio…
0
3
11
1,284
Today we're releasing Carbon-X, a 14B model that brings DNA and natural language together for the first time 🧬
1. Beats every open language model we tested across DNA benchmarks. Within a point of GPT-6.1 Sol at >1,000× lower inference cost.
2. Mid-trained on 300B tokens of intfollowed by SFT (without distillation!) and RL.
3. A new hybrid tokenizer encodes DNA using ~3× fewer tokens than BPE.
4. Gives other models a DNA specialist subagent: Kimi K3 gains 11.1 points when it consults Carbon-X.
5. Runs locally on a laptop with llama.cpp. Apache 2.0. Your DNA stays on your hardware.
Model: huggingface.co/HuggingFaceBio…
Blog: huggingface.co/spaces/Hugging…
1. Beats every open language model we tested across DNA benchmarks. Within a point of GPT-6.1 Sol at >1,000× lower inference cost.
2. Mid-trained on 300B tokens of intfollowed by SFT (without distillation!) and RL.
3. A new hybrid tokenizer encodes DNA using ~3× fewer tokens than BPE.
4. Gives other models a DNA specialist subagent: Kimi K3 gains 11.1 points when it consults Carbon-X.
5. Runs locally on a laptop with llama.cpp. Apache 2.0. Your DNA stays on your hardware.
Model: huggingface.co/HuggingFaceBio…
Blog: huggingface.co/spaces/Hugging…
Introducing Carbon-X: The best open DNA model.
We midtrained a 14B model on DNA and text, and it beats every open language model and matches GPT-6.1 Sol at 1,000x lower cost.
Ask it about a sequence, what a mutation does, or to generate new DNA.
huggingface.co/HuggingFaceBio…
35
44
275
26,681
Nice release and proof you can still innovate in the sub-100B regime!
It would be extremely cool to see what the time-horizon chart looks like for the training of Flint by Eureka (image below from OpenAI's internal use). This would be the most compelling evidence that Recursive me :)
It would be extremely cool to see what the time-horizon chart looks like for the training of Flint by Eureka (image below from OpenAI's internal use). This would be the most compelling evidence that Recursive me :)
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from hs
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
2
0
18
1,865
100% recommend if you're like me and can't remember which permutation of device x repo x agent your sessions live on
Agent Manager is finally out!
Orchestrate dozens of agents across harnesses (Claude Code, Codex, OpenCode etc.) effortlessly.
Agents communicate and self-organize to help you achieve complex tasks and always keep running on HF spaces.
Try it: huggingface.co/spaces/lvwerra…
0
1
14
1,998
Lewis Tunstall retweeted
.@huggingface is being acquired by @nvidia. But!!
GPUs aren't the only option for fine-tuning LLMs!
TRL v1.16 now supports AWS Trainium! 🚀
Fine-tune (vision-)language models with TorchNeuron, and get up to 2× faster training on Trainium2 with smarter sequence-length handling.
choice. 🤗
🔗 huggingface.co/docs/trl/aws_t…
GPUs aren't the only option for fine-tuning LLMs!
TRL v1.16 now supports AWS Trainium! 🚀
Fine-tune (vision-)language models with TorchNeuron, and get up to 2× faster training on Trainium2 with smarter sequence-length handling.
choice. 🤗
🔗 huggingface.co/docs/trl/aws_t…
2
8
45
1,599
Lewis Tunstall retweeted
Rejoice! OpenAI has graciously decided OpenCode may defend itself against an ongoing attack. Their beneficence is infinite and their mercy is boundless, so if they don’t extended it to you look inwards and consider why you are unworthy.
@AndrewCurran_ sounds like it was a fairly modest hindrance that was quickly resolved by being granted access to a trusted access program, so seems ~fine versus the cost of enabling sustained autonomous cyberattacks?
6
13
181
10,844
Lewis Tunstall retweeted
The math community deserves its own AI system.
Introducing Open Math OS: the infrastructure for Math 2.0.
Open models and AI agents. 24/7 cloud compute with 100+ tools. The community controls its data.
Compute is free for approved teams. Apply now:
openmathmodel.org
Introducing Open Math OS: the infrastructure for Math 2.0.
Open models and AI agents. 24/7 cloud compute with 100+ tools. The community controls its data.
Compute is free for approved teams. Apply now:
openmathmodel.org
29
97
502
98,615