Agent loop is like stochastic gradient descent, just without gradient.
Overnight the DeepSeek optimization gave us 2.32ms/step, way below our educated guess of a 2.5ms/step floor, and improving. I asked AI to draw out some statistics and I think it's interesting.
1. We didn't t empirically it catches early gains first and then improvements start slowing down.
2. Of course, we haven't really tried permuting the experiments and see their independent improvements, but eyeballing some, my observation is that indeed the later ones are more incremental.
3. Convergence seems to be consistent. It's not like the agents are running out of ideas.
4. I looked at the acceptance rate of experiments. Initially it was really high - makes sense. But then it kinda stayed consistent around 46%.
5. Gains do look like exponentially decaying and not heavy tailed.
6. SGD has seen great improvement via rigorous mathematical explorations like momentum SGD, hessian-free approaches, ADAM, Muon, etc.
7. The problem is... discrete optimization on a group of agents is hard. There really isn't any coordinates or gradients. Landscape isn't fixed and all we are talking about is something like a sample - reject_or_accept - resample loop.
8. What's the best heuristics and what's the best algo? It feels like Kalman filtering / reject sampling / data driven markov-chain monte-carlo to me but exact math is still unclear.
I am really looking forward to possibly running larger experiments and looking into more traces of such. This seems to be engineering and research combined in one.
Overnight the DeepSeek optimization gave us 2.32ms/step, way below our educated guess of a 2.5ms/step floor, and improving. I asked AI to draw out some statistics and I think it's interesting.
1. We didn't t empirically it catches early gains first and then improvements start slowing down.
2. Of course, we haven't really tried permuting the experiments and see their independent improvements, but eyeballing some, my observation is that indeed the later ones are more incremental.
3. Convergence seems to be consistent. It's not like the agents are running out of ideas.
4. I looked at the acceptance rate of experiments. Initially it was really high - makes sense. But then it kinda stayed consistent around 46%.
5. Gains do look like exponentially decaying and not heavy tailed.
6. SGD has seen great improvement via rigorous mathematical explorations like momentum SGD, hessian-free approaches, ADAM, Muon, etc.
7. The problem is... discrete optimization on a group of agents is hard. There really isn't any coordinates or gradients. Landscape isn't fixed and all we are talking about is something like a sample - reject_or_accept - resample loop.
8. What's the best heuristics and what's the best algo? It feels like Kalman filtering / reject sampling / data driven markov-chain monte-carlo to me but exact math is still unclear.
I am really looking forward to possibly running larger experiments and looking into more traces of such. This seems to be engineering and research combined in one.
8
10
100
8,608
I really hope that one agent company can start to issue credit cards and offer first-party insurance against agent flops.
I use personal agents dozens of times / day, but not gonna put my bank acct directly in their hands yet.
And of course seeing "domain names" is always fun.
I use personal agents dozens of times / day, but not gonna put my bank acct directly in their hands yet.
And of course seeing "domain names" is always fun.
thibaultsottiaux.com just dropped
0
1
14
4,325
A hand-typed analysis for optimizing LLM inference with agent fleets. There has been a lot of recent news about that - but many are kinda vague, so I think a more technical post is in order.
To explain why this may claim credibility: I drew conclusion from our own DeepSeek 4.1 ch reached a per-step time of 2.41 ms, and a generation speed of up to 2300 tok/s (with Dspark) and reliably around 1800 tok/s for agent tasks. Agents ran ~230 optimization proposals and 94 of them were verified to be useful.
My 2 cents from those runs below:
1. Lots of gains beyond "kernel optimization". More than 50% of the optimizations were shipped with no new kernel but configs / graphs / launch order and other strategies. Kernels have been a well beaten path but there is a lot to be gained in the other parts of the end to end system.
2. MegaKernel might be a lie. On non-trivially large models, megakernel is trying to take to big a bite. Most of our experiment showed that glueing small ops into existing kernels, or fusing multiple small kernels into a medium one, is getting the biggest gain. This may only apply to NVIDIA GPUs where the underlying driver is very optimized, though, as @vllm recently showed good results with megakerkenl on TPU.
3. KernelBench and real serving are different problems. Many kernels measured positive in isolated runs but turned out negative when combined in end-to-end. I've always wanted a good simulation like Accel-Sim, but so far simulation don't have a good accuracy when modeling complicated latency patterns with both compute and memory bounds.
4. Verification becomes *really* important if you don't want to be glued to your desk. If you can define "what is correct", like numerical tests, end-to-end quality evaluations, then agents can run for a long time. We didn't interfere with the run other than (a) adding GPU machines, (b) ensure token limit aren't hit, and (c) increasing credit card spending limit.
5. Don't worry, it'll converge - I think the community has consensus that once you set up the goal, agents will eventually converge. Convergence speed is gonna be key though, and 4(c) above is really real.
6. At low concurrency settings, low level assembly writing such as SASS edits have less effect, mainly because waiting for memory (and cache?) becomes the main bottleneck.
7. The hot paths are kinda solved, although much more is to be explored. I think the biggest gain in the next couple months will come from optimizing the wider distributed serving stack, in addition to the single serving engine itself. This would probably involve a lot of system and SRE insights: building proper telemetry, building proper A/B testing without disrupting production traffic, kinda similar to the good old recommendation systems do.
8. Agents doing end to end optimization is a necessity. In our case agents diligently went through 230 experiments, each of which might take a person a week (or at least a couple days with manual experiments + claude code). The sheer volume and speed definitely allows us to explore all possible optimizations otherwise not possible due to human bandwidth.
My last thought is a bit contradictory: so far, it seems that agents are "good engineers", in the sense that they can massively carry out known insights. In a way, they are "interpolating" what we already know, but just way faster. I haven't seen agents design awesome algorithms like PagedAttention or Winograd, yet. But who knows, extrapolating from our existing knowledge base may well be very possible in the upcoming months.
And man, I like typing on a keyboard, although it might look much worse than a well trained AI writer.
To explain why this may claim credibility: I drew conclusion from our own DeepSeek 4.1 ch reached a per-step time of 2.41 ms, and a generation speed of up to 2300 tok/s (with Dspark) and reliably around 1800 tok/s for agent tasks. Agents ran ~230 optimization proposals and 94 of them were verified to be useful.
My 2 cents from those runs below:
1. Lots of gains beyond "kernel optimization". More than 50% of the optimizations were shipped with no new kernel but configs / graphs / launch order and other strategies. Kernels have been a well beaten path but there is a lot to be gained in the other parts of the end to end system.
2. MegaKernel might be a lie. On non-trivially large models, megakernel is trying to take to big a bite. Most of our experiment showed that glueing small ops into existing kernels, or fusing multiple small kernels into a medium one, is getting the biggest gain. This may only apply to NVIDIA GPUs where the underlying driver is very optimized, though, as @vllm recently showed good results with megakerkenl on TPU.
3. KernelBench and real serving are different problems. Many kernels measured positive in isolated runs but turned out negative when combined in end-to-end. I've always wanted a good simulation like Accel-Sim, but so far simulation don't have a good accuracy when modeling complicated latency patterns with both compute and memory bounds.
4. Verification becomes *really* important if you don't want to be glued to your desk. If you can define "what is correct", like numerical tests, end-to-end quality evaluations, then agents can run for a long time. We didn't interfere with the run other than (a) adding GPU machines, (b) ensure token limit aren't hit, and (c) increasing credit card spending limit.
5. Don't worry, it'll converge - I think the community has consensus that once you set up the goal, agents will eventually converge. Convergence speed is gonna be key though, and 4(c) above is really real.
6. At low concurrency settings, low level assembly writing such as SASS edits have less effect, mainly because waiting for memory (and cache?) becomes the main bottleneck.
7. The hot paths are kinda solved, although much more is to be explored. I think the biggest gain in the next couple months will come from optimizing the wider distributed serving stack, in addition to the single serving engine itself. This would probably involve a lot of system and SRE insights: building proper telemetry, building proper A/B testing without disrupting production traffic, kinda similar to the good old recommendation systems do.
8. Agents doing end to end optimization is a necessity. In our case agents diligently went through 230 experiments, each of which might take a person a week (or at least a couple days with manual experiments + claude code). The sheer volume and speed definitely allows us to explore all possible optimizations otherwise not possible due to human bandwidth.
My last thought is a bit contradictory: so far, it seems that agents are "good engineers", in the sense that they can massively carry out known insights. In a way, they are "interpolating" what we already know, but just way faster. I haven't seen agents design awesome algorithms like PagedAttention or Winograd, yet. But who knows, extrapolating from our existing knowledge base may well be very possible in the upcoming months.
And man, I like typing on a keyboard, although it might look much worse than a well trained AI writer.
14
11
110
13,234
Our agent fleet just took #1 on ClickBench, 1.2× ahead of the next best engine and 3.1× ahead of the codebase it started from. To our knowledge, the first time an autonomous agent fleet has taken first place on a public systems benchmark where correctness is verified independently.
5
7
46
4,684
We stripped out anything that only helped this benchmark - avoiding "AI feature creeps". Reviewer agents stripped out any special snowflake optimizations. ClickBench later added a concurrent QPS measure the fleet never targeted, and our entry is first there too.
1
0
1
505
Our takeaway: as agents get more capable, the infrastructure around them becomes the main limit on how fast they iterate, and letting the fleet work on its own methodology paid off. We plan to open source the engine. Full write-up: intentlab.ai/blog/an-agent-…
0
0
2
397
Autonomous engineering fleet needs strong review, so we built a rigorous benchmark of 284 real-world bugs.
Our review agent caught 93%, beating every baseline at 63% lower cost than the top one.
When bugs get overlooked, nearly 20% of them last over a year.
b
Our review agent caught 93%, beating every baseline at 63% lower cost than the top one.
When bugs get overlooked, nearly 20% of them last over a year.
b
11
11
58
5,387
Once merged, bugs tend to stay. In our benchmark the median bug took 33 days to fix, nearly 20% lasted over a year, and one went unnoticed for more than 15 years.
1
0
1
298
As coding agents take on more of the work, verification has to scale with them. That's what we're building at Intent Lab.
Full write-up with the benchmark details: intentlab.ai/blog/building-…
And learn more about our fleet here: intentlab.ai/blog/turn-your…
Full write-up with the benchmark details: intentlab.ai/blog/building-…
And learn more about our fleet here: intentlab.ai/blog/turn-your…
0
0
3
287
TLA+ is also (one of the) method that we at Intent Lab used to build mission critical infra software like Agent FS. Key is to make it verifiable and scalable - going through millions of states and also ensure the TLA+ to Rust/C conversion doesn’t weaken the end to end correctness. intentlab.ai/blog/turn-your…
I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.
TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.
nguage well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.
Is formal verification the future of coding (or at least, bug finding)?
9
24
245
37,139
Excited to see the official launch of 224 Ventures! Also @OriolVinyalsML launching his new startup on the same day - the AI world is never short of excitements.
4
2
24
8,204
My 2 cents: one should build the right harness. If a harness dictates how the model works, it fights the model and you get brittleness. If it defines what the task is and checks the result, it enables long horizon task success. That's roughly how we got complex tasks done e2e at IntentLab.
On harnesses, I vacillate between three beliefs:
- the less harness, the better. Models are the magic
- post training a model and harness is dramatically better and the model providers win
- harnesses have real independent value from the model
I have no idea which is right.
2
6
88
12,312
We are starting Intent Lab, building an autonomous team we call "fleet" that turns intent into production software. Today we are sharing some early results: the fastest GLM5.2 inference engine, one shot database creation, and a fully verified agent filesystem.
intentlab.ai/blog/turn-your…
intentlab.ai/blog/turn-your…
76
89
603
282,798
We've spent our careers building large systems by hand, from open source to cloud infra, to our last startup. The theme was careful architecture, done one system at a time. What interests us now is building the oracle that produces a thousand of them.
2
0
12
3,934
We're working with design partners. If there's a system you want built, modernized, or made much faster, especially one that's been sitting on the list because it's too big to staff, send me a note for early access. If the mission resonates, we're hiring. Exciting paths ahead. Let us know!
0
0
14
3,601