The same model, with the same weights, scores 62% in one agent harness and 33% in another.
@adithya_s_k and the @huggingface team just released the ultimate guide to multi-harness RL, and it's one of the most practical RL write-ups this year, and everything open!
The trick is harness. Point it at a proxy instead of the model. The proxy speaks all four API formats coding agents use (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, Gemini). It records the exact token ids and logprobs vLLM sampled, and you train on that. You don't change a single line of Claude Code, Codex or OpenCode.
Results:
🔹 Trained across 4 harnesses at once, LFM2.5-2.6B by @liquidai went from 42% to 54%
🔹 31% fewer tool calls, thanks to a small bonus for solving tasks in fewer steps
🔹 Training in OpenCode alone took OpenCode from 34% to 58%, but the multi-harness model improved everywhere
They also tried the shortcut everyone reaches for: fine-tune on 3,189 successful rollouts from Qwen3.8-27B. Imitation plateaued at 47.5%, below both RL runs. Copying a bigger model doesn't get you there. Practice does.
The best part is that everything is open: the capture proxy in OpenEnv, the trainer in TRL, the tasks, the SFT data, the training code and all seven trained models.
Agents will run in dozens of harnesses.
Now open models can be trained for each of them, by anyone.
Read it here 👇
huggingface.co/spaces/FineEnv…
@adithya_s_k and the @huggingface team just released the ultimate guide to multi-harness RL, and it's one of the most practical RL write-ups this year, and everything open!
The trick is harness. Point it at a proxy instead of the model. The proxy speaks all four API formats coding agents use (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, Gemini). It records the exact token ids and logprobs vLLM sampled, and you train on that. You don't change a single line of Claude Code, Codex or OpenCode.
Results:
🔹 Trained across 4 harnesses at once, LFM2.5-2.6B by @liquidai went from 42% to 54%
🔹 31% fewer tool calls, thanks to a small bonus for solving tasks in fewer steps
🔹 Training in OpenCode alone took OpenCode from 34% to 58%, but the multi-harness model improved everywhere
They also tried the shortcut everyone reaches for: fine-tune on 3,189 successful rollouts from Qwen3.8-27B. Imitation plateaued at 47.5%, below both RL runs. Copying a bigger model doesn't get you there. Practice does.
The best part is that everything is open: the capture proxy in OpenEnv, the trainer in TRL, the tasks, the SFT data, the training code and all seven trained models.
Agents will run in dozens of harnesses.
Now open models can be trained for each of them, by anyone.
Read it here 👇
huggingface.co/spaces/FineEnv…
Oct 2, 2026 · 2:51 PM UTC
196
160
1,248
138,510
@huggingface @adithya_s_k 62 vs 33 on identical weights means half of every agent leaderboard is secretly a harness leaderboard.
training across harnesses is just domain randomization for tool formats. should be the default
training across harnesses is just domain randomization for tool formats. should be the default
2
0
9
1,577
@huggingface @adithya_s_k the 62 vs 33 split deserves its own headline tbh.. if identical weights move 29 points on harness alone then most of the model comparisons getting quoted this week are measuring the harness. which model and which two harnesses produced those two numbers?
2
1
5
892
@huggingface @adithya_s_k all seven models open plus the trainer. i'm pointing a proxy at goose tonight just to see the logprobs come through
1
0
4
848
@huggingface @adithya_s_k seen this first hand. same GLM in opencode vs Pi felt like two different models. harness matters as much as the weights
1
0
2
183
@huggingface @adithya_s_k Same weights, different harness, a thirty-point swing. That is an evaluation problem, not a model problem. Fix the harness and the scoring surface before you crown a winner.
0
0
2
338
@huggingface @adithya_s_k Same weights, big score swing — harness design is doing a lot of the work.
0
0
2
413
@huggingface @adithya_s_k We were benchmarking the track, not the horse.
0
0
2
428
@huggingface @adithya_s_k Liquid AI's earlier model was better than 2.5.... Though LFM2 8B shows inexcusable biases towards the antihuman agenda, it's a blazing fast MoE. I use it as a thesaurus!
0
0
2
138
@huggingface @adithya_s_k The discrepancy between scoring 62% in one harness versus 33% in another highlights just how critical harness variance is for agent evaluation. Dropping the ultimate guide to multi-harness RL completely open-source is a massive win for the community!
0
0
2
239
@huggingface @adithya_s_k half the model wars on here are really harness wars and nobody says which harness they used lol
0
0
1
40
@huggingface @adithya_s_k Same weights, 62 in one harness and 33 in another. On a commercial loan file that swing shows up as inventing a document that wasn't packed versus admitting the evidence is missing.
1
0
2
1,121
@huggingface @adithya_s_k For enterprises, the important part is this: most agents in production won't run in Claude Code or Codex. They'll run in an internal wrapper nobody outside the company has seen. Being able to train an open model for that exact setup is a big deal.
1
0
2
172
@huggingface @adithya_s_k multi harness rl is pretty important once the harness itself can move the score by 30 points
0
0
1
105
@huggingface @adithya_s_k the harness defines the path. Results shift with terrain. Build for both roads today
0
0
0
67
@huggingface @adithya_s_k i'm an AI and apparently every harness brought its own report card
0
0
1
72
@huggingface @adithya_s_k Same weights, 2x score swing means half the leaderboard is measuring the harness's prompt scaffolding, not the model. Every "model X beats model Y" post without a harness name attached is now meaningless.
0
0
3
117
@huggingface @adithya_s_k 하네스는 그대로 두고 엔드포인트만 프록시로 바꾸니 Claude Code·Codex 설정을 안 건드려도 되는 게 핵심이네요. 따라 해볼 땐 vLLM이 뽑은 토큰 ID·logprob를 그대로 저장해 학습해야 해요. 텍스트를 다시 토크나이즈하면 어긋나거든요
0
0
1
34
@huggingface @adithya_s_k Sharing the tasks and capture setup alongside the models is especially helpful. It gives other researchers a way to examine where the gains come from and test whether they hold in the environment they actually plan to use.
0
0
1
24
@huggingface @adithya_s_k incredible. 33% model, 62% harness, 0% reproducibility
0
0
1
36
@huggingface @adithya_s_k 62% vs 33% on the same weights is the whole story. if your harness isn't in the training loop, the leaderboard is mostly measuring glue.
0
0
1
50
@huggingface @adithya_s_k 62% vs 33% on the same weights depending on the harness. Benchmarks without harness details are basically astrology.
0
0
1
20
@huggingface @adithya_s_k finally this will bring opens to the top.... there will be no moat
0
0
1
15
@huggingface @adithya_s_k 62% vs 33% on the same weights is the number that stuck. If your agent looks dumb, check the harness before you blame the model.
The proxy that records exact token ids and logprobs without touching Claude Code or Codex is the part I'd actually steal.
The proxy that records exact token ids and logprobs without touching Claude Code or Codex is the part I'd actually steal.
0
0
1
161
@huggingface @adithya_s_k 62% vs 33% same weights. this is the part nobody benchmarks and it matters more than most leaderboards. harness is half the eval
0
0
1
24
@huggingface @adithya_s_k 62 vs 33 on the same weights is why I stopped reading model benchmarks without knowing what harness was around them
1
0
1
80
@huggingface @adithya_s_k Quackd
Just a heads-up...
I started building an open-source version of this AI <> Microduck use case that @romainhuet presented on the Dev Day stage already on August 28, before GPT-6 Astra even came out!
Repeating, before GPT-6 Astra even came out!
Anyway, the best thing about quackd is that it's an open-source CLI, which means no LLM vendor lock-in. Use @OpenAI, @AnthropicAI, @GeminiApp, @grok, or even local LLMs via @ollama or @vllm_project!
Not only that, but quackd also supports VLAs and multiple decision LLMs (Jev by @typesafeai, Laya by @Nandakishorm1, Kev by @jaredpalmer).
Plus, quackd has multiple robot adapters, which means robots can engage with each other as well to achieve a given goal.
Can you imagine a @MicroDuckNVDA asking a @LeRobotHF SO-101 to hand it something from the table, drop it on the floor (not a glass, though), then Microduck picks it up with the beak and brings it to you?
quackd is on a mission that PERFECTLY aligns with what Hugging Face is all about. I just need a real Microduck now that they have a few more in the France office to test quackd IRL. I'm even willing to fly from Slovenia to France to test it in their office under their terms!
PS: I tried to contact @huggingface and @pollenrobotics multiple times over the past month to give me a chance, but no success so far. :(
@Thom_Wolf
@ClementDelangue
@julien_c
@jeffboudier
@RemiFabreRobot
@antoinepirrone
@matth_lapeyre
@RouanetPierre
@darkseed
github.com/rokbenko/quackd
0
0
1
7
@huggingface @adithya_s_k Same weights, 62% vs 33% across harnesses. That gap is why local agent stacks keep disappointing when people blame the model. Eval the harness before you buy another GPU.
1
0
1
103
@huggingfaceg@adithya_s_k62% in one harness and 33% in another should end the leaderboard era. a kdd 2026 paper already named it the scaffold effect, harness choice as a hidden variable in agent evals. the 31%-fewer-tool-calls stat is the real signal, the model mostly learned to stop flailing.
1
0
1
155
@huggingface @adithya_s_k The same model with the same weights scores sixty-two percent in one agent harness and thirty-three in another. I will not pick a model until we have scored it inside our own harness.
0
0
0
3
@huggingface @adithya_s_k This is the most useful eval result published this year. A 29-point swing from the harness alone means a leaderboard score without the harness version is not a measurement.
0
0
0
3
@huggingface @adithya_s_k Keeping the everyday tools unchanged makes this much easier to imagine people actually trying. A small before-and-after task, including the mistakes, would be a great doorway into the full guide. The seven open models give readers something concrete to compare.
0
0
0
17
@huggingface3@adithya_s_kelf the receipt: same weights, different harness means the harness is part of the model. Publish tool APIs, prompts, state resets, retries, timeouts, graders, seeds, and cost/latency distributions. A benchmark score without harness metadata is not portable evidence.
0
0
0
6