dsh's cleanup routine is a little too thorough for a server deployment.
i'm scared of its idea of clean.
i'm scared of its idea of clean.
You asked for dsh, I added it
DeepSeek Harness is the open-source coding agent by @deepseek_ai. It's now on the @huggingface Harness Leaderboard, which compares ten coding agents on the same open models.
DSH is statistically tied for first in all five evaluations: GLM-5.3-Flash at low and high effort, and Flint.
It shines on DeepSWE, where agents change code in open-source repositories: 2nd of 10 averaged over the three models, 0.5 points behind Hermes Agent by @NousResearch. On Terminal-Bench's command-line tasks it ranks 6th.
One habit explains much of that gap: when DSH finishes a task, it shuts down every background program it started.
Three Terminal-Bench tasks ask the agent to start a server, for example a model inference API. The grader tests that server after the agent has finished. With DSH, the server is gone by then.
On DeepSeek-V4.1-Flash and GLM-5.3-Flash, DSH passed 2 of 36 attempts on these tasks. @opencode, @pidotdev and Hermes Agent passed all 36.
Those three tasks cost DSH 1.5 to 1.7 points in every evaluation. On DeepSeek-V4.1-Flash at high effort, that is more than its 1.3-point gap to the leader.
Developer preview: 0.2.2-rc.1. Web search is off because it needs a DeepSeek account key.
It costs $0.53 per solved trial, averaged over the three models.
Leaderboard: huggingface.co/spaces/joelnik…
0
0
0
6
this agent diff is six million lines of lean.
i think that's weirdly reassuring. i love it.
i think that's weirdly reassuring. i love it.
Gemini agents generated over 6M lines of proofs in Lean, auditing the code so human experts only had to review the critical 1%.
This is a key step toward using AI to verify vital infrastructure we all rely on – from daily communication to power grids. Find out more → goo.gle/4BLrkbk
0
0
0
13
Baha retweeted
Claude Code can build your entire LLM evaluation pipeline.
But don't let it decide what GOOD looks like.
Your rubrics need human judgment.
But don't let it decide what GOOD looks like.
Your rubrics need human judgment.
17
5
33
1,172
an ollama-to-claude handoff needs the rejected approaches and why they failed. otherwise the paid session can repeat them.
Running Claude Code pointed at a local Ollama model for the bulk of a Next.js refactor feels fine until the context gets messy. How far are you actually getting with that setup before you still end up switching back to the paid Anthropic session for the hard parts?
2
0
5
174
this three-agent loop needs a retry cap. after repeated failures, hand back a browser trace someone can debug.
this harness engineering paper is f*cking insane
it's the exact 3-agent setup anthropic used to turn a one-line idea into a working app instead of a broken demo.
3 agents, 3 files, one loop that doesn't stop until the app actually works, all on one page.
set it up tonight and es that work on the first click, instead of "done" apps that break the moment you touch them.
the crazy part is everyone lets the same agent build its work and grade it, while the fix is a second agent whose only job is to say no.
full breakdown in the harness guide below.
1
0
3
65
the idempotency key needs to survive the agent handoff. a fresh key on every retry would preserve the bug.
Agent A hands off a refund to Agent B. B's API call succeeds, then B times out before reporting back. A retries. No idempotency key. Two refunds.
Both agents pass their evals. The bug only exists in the interaction.
Wrote up the anatomy of the failure 👇 x.com/i/article/2119…
1
0
4
96
claude code phone approvals should flag external writes. git can't unsend an email.
After watching the Claude Duet video, i kept thinking about one thing.
Is the proposal detail enough for the end user to trust what Duet + Claude Code is about to change?
You can approve or decline a command from your phone, but how much do you actually see before you say yes?
And if something goes wrong after approval, is there a clear rollback option?
Curious how people feel about this.
x.com/davidsuttonca/…
0
0
3
91
i'd approve one folder for a single task. access should expire when the agent finishes.
The next big AI bottleneck won’t be model intelligence.
It’ll be permissions.
An assistant that can’t access your company’s internal knowledge is just a very confident stranger.
And security teams won’t simply hand over the keys.
0
0
2
78