Pinned Tweet
How to effectively debug with agents when you no longer read or understand the code
In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”
The TL;DR are the screenshots.
Link to post in the reply.
In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”
The TL;DR are the screenshots.
Link to post in the reply.
Writing a post about “how to effectively debug code when you no longer fully understand it”.
This is real bug I had to fix in ~9K LOC (100% AI generated).
None of the tactics I started with here helped “root cause” the issue, lol. Had to take a different approach to isolate it.
40
124
1,667
142,970
CI as it currently exists (build all the code to run the full test suite) seems like a waste of compute, which can be better assigned to agents to analyze and test select parts of code but way more rigorously.
My pre-merge tests don’t do full builds; just targeted micro-tests.
My pre-merge tests don’t do full builds; just targeted micro-tests.
How do you deal with the CI time to run all those tests, etc?
20
6
149
8,957
Revisiting my tweets from 6m ago is funny, because I don’t run into any of these problems now at all.
What was missing in my setup 6 months ago was the verification process I now use.
70% of my daily token spend goes into verifying patches, and only ~20-30% into creating them.
What was missing in my setup 6 months ago was the verification process I now use.
70% of my daily token spend goes into verifying patches, and only ~20-30% into creating them.
Might be anecdotal, but just this week multiple AI generated PRs with subtle bugs got merged that required several additional days and a lot of manual verification to fix.
“Velocity” isn’t going up when you consider how much effort was spent fixing things post-merge, pre-deploy.
23
8
223
25,449
This is 100% true.
Your reliability in general should be *higher* with agent written code, as measured by error rates, SLO violations, client regressions etc.
If you’re having more or equal incidents now than when you were manually writing code, that’s likely a “skill issue”.
Your reliability in general should be *higher* with agent written code, as measured by error rates, SLO violations, client regressions etc.
If you’re having more or equal incidents now than when you were manually writing code, that’s likely a “skill issue”.
Agreed. We have shockingly few production errors these days. E.g. panic from a missing nil-check happens once every couple of months, and usually involves legacy code humans partially wrote. At this point, if I write it by hand you are going to get more bugs.
16
17
317
15,286
How to design good verification loops?
Well, that’s most of the job now, isn’t it?
Last week, I finally landed a refactor I started in Jan. It’s good I waited b/c the Jan version was buggy af.
This is the verification process I used for refactors (have different ones for perf)
Well, that’s most of the job now, isn’t it?
Last week, I finally landed a refactor I started in Jan. It’s good I waited b/c the Jan version was buggy af.
This is the verification process I used for refactors (have different ones for perf)
How does one ensure verification loops are actually correct? Rigorous needs to take cost into consideration as well.
26
16
522
24,134
Getting agents to produce “high-quality, optimized outputs in a language or domain you’re personally unfamiliar with” is an emergent skill.
It doesn’t happen magically, but requires:
1. A change in mindset
2. Setting up the best domain references and rigorous verification loops
It doesn’t happen magically, but requires:
1. A change in mindset
2. Setting up the best domain references and rigorous verification loops
I still can't believe people are rewriting software into Rust for performance reasons without learning any Rust.
I'm optimizing a Go program right now and using Agents to do the work, but I have to steer them a lot or they get stuck in local minima, but I can only steer them because I have 10 years of doing performance work in Go.
If I was optimizing a Rust program I'd have to just keep saying: "make it faster" over and over again until it got stuck somewhere dumb.
10
5
238
11,984
Returning from international travel, followed by jet lag, physical exhaustion, remnants of a throat infection I picked up when traveling, dealing with a case of misplaced luggage across 3 carriers, catching up on work on a quasi-zombie state …
Yeah, I’m glad its the weekend ❤️
Yeah, I’m glad its the weekend ❤️
0
0
21
3,783
Didn’t Google just blog about how they solve this today? cloud.google.com/blog/topics/ai…
Secrets are kept entirely out of the agent sandbox. A security proxy injects credentials when needed.
The “secure proxy” route to secure agent infra and secrets seems to be a popular pattern.
Secrets are kept entirely out of the agent sandbox. A security proxy injects credentials when needed.
The “secure proxy” route to secure agent infra and secrets seems to be a popular pattern.
Is anyone solving how we give secrets/passwords to agents in a way that doesn't expose them to the upstream provider? It seems like codemode could be used to pass the tokens around and you'd only need to approve it for where the token is actually consumed
23
25
468
29,104
The reason I still review every line of code despite encoding all the software engineering best practices in my agentic workflow, is because sometimes it’s the *exceptions* to these generalized rules that work best.
The agent isn’t discerning enough to make these calls (as yet)
The agent isn’t discerning enough to make these calls (as yet)
I've been reviewing a lot of code written by Opus 5.5, and it still seems to make a lot of bad software engineering decisions. The kind that will accumulate/calcify into unmaintainable software over time. IMO even the agents will struggle to maintain vibecoded software.
20
3
281
13,378
At Paris CDG, Terminal 2E, security check:
Gate K: no need to take your electronics out
Gate M: take all your electronics out and put it in a tray
Me: I wasn’t asked to do it in Gate K
Security: Gate K is Gate K, Gate M is Gate M
Never seen this in any other airport. 😕
Gate K: no need to take your electronics out
Gate M: take all your electronics out and put it in a tray
Me: I wasn’t asked to do it in Gate K
Security: Gate K is Gate K, Gate M is Gate M
Never seen this in any other airport. 😕
20
0
68
8,636
Cindy Sridharan retweeted
I came back to Google to work on AI infra. Instead we got pulled in to solving the problem of exploding use of agents across the company.
Announcing Agent Compute Engine: bringing planet scale agentic infra to Google services and now to Google Cloud.
cloud.google.com/blog/topics/ai…
Announcing Agent Compute Engine: bringing planet scale agentic infra to Google services and now to Google Cloud.
cloud.google.com/blog/topics/ai…
32
57
698
49,010
I found this unpublished blog post from 6y ago (look at the hand drawn diagram lol).
I’d totally forgotten about this, but the key ideas here (embracing non-determinism as a first class citizen via “error kernel” centric design etc) work super well as guidelines for my agents.
I’d totally forgotten about this, but the key ideas here (embracing non-determinism as a first class citizen via “error kernel” centric design etc) work super well as guidelines for my agents.
3
7
161
7,817
5 months and many experiments later, this was totally a case of “skill issue” on my part 😅
Like with code, agents don’t magically write great tests
If your unit tests are slop (as mine were), that’s because you don’t have proper testing guidelines for your agents (as I didn’t)
Like with code, agents don’t magically write great tests
If your unit tests are slop (as mine were), that’s because you don’t have proper testing guidelines for your agents (as I didn’t)
end-to-end testing > unit tests, in the vibecoding era.
A massive, almost entirely agent-coded refactor passed all unit and pre-merge tests but broke a critical feature.
It was only caught due to my own excessive paranoia making me run end-to-end tests before the prod deploy.
25
9
388
19,838
We’ll still need end to end tests, and frankly, we’re going to need all the forms of testing I’ve been talking about for a decade.
But “unit tests are now slop” doesn’t have to be the foregone conclusion.
Unit tests can be very useful even if a human will never again read them.
But “unit tests are now slop” doesn’t have to be the foregone conclusion.
Unit tests can be very useful even if a human will never again read them.
Updated my testing taxonomy to include formal verification. Some book recs too.
Model checking, fuzzing, property based testing, sanitizers and static analysis are some of the most powerful tools one can use in the "pre-production" phase.
Also, are "types" a form of testing? 🤔
4
2
58
3,649
Unfinished coding side projects are now way easier to complete than unfinished blog posts.
I’m perfectly OK with AI writing my code, even the docs.
But I flinch at the idea of AI writing my blog posts, which feels like an egregious violation of a fundamental ethnical contract.
I’m perfectly OK with AI writing my code, even the docs.
But I flinch at the idea of AI writing my blog posts, which feels like an egregious violation of a fundamental ethnical contract.
27
31
625
13,441
I started this series on testing nearly a decade ago, but never published the final part copyconstruct.medium.com/testing-micros…
There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing.
Have a draft from years ago I hope I can wrap up in early 2027.
There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing.
Have a draft from years ago I hope I can wrap up in early 2027.
9
10
187
8,910
I’m literally that weirdo who wrote (all by hand) a 15K word blog post on:
- why one cant treat models as oracles
- how models gaslight you all the time
- what it takes to build confidence in the model’s output
copyconstruct.medium.com/how-to-effecti…
So yeah sure, I believe in magical woo-woo
- why one cant treat models as oracles
- how models gaslight you all the time
- what it takes to build confidence in the model’s output
copyconstruct.medium.com/how-to-effecti…
So yeah sure, I believe in magical woo-woo
I don't expect models to do things well automatically. I'm just pointing out that you believe in the magic too much. Including things "I gave it things to read" lol
15
36
789
45,068
To create my claude.md, I had it read:
- my favorite programming books + blogs
- the best oss projects
- the code of the engineers I respect the most
Claude (when given proper instructions, constraints, and context) now writes better code than me, almost always.
- my favorite programming books + blogs
- the best oss projects
- the code of the engineers I respect the most
Claude (when given proper instructions, constraints, and context) now writes better code than me, almost always.
the coding agent haters are getting dunning-kruger'd so hard right now
do you really think that your engineering ability is better than the models at this point?
it is very very likely that fable is better than you in every branch of software
it has been trained on all the code of the internet, it knows almost every trick in the book, all the design patterns of every domain, every single low-level detail, every bit-twiddling hack, it's read every single blog post from every cracked engineer that ever existed
do you really think you can beat it? do you really think you can be more creative than it?
unless you are 99th percentile expert in some niche domain of software, it simply is not possible. sorry.
105
207
4,416
410,083
Many moons ago, I wrote about “post-commit” reviews.
In the agentic era, a workflow I find extremely useful is … *pre-commit* agentic reviews.
For large changes, the review agents take 10x the time it takes the coding agent to create a patch, but is totally worth it, IMO.
In the agentic era, a workflow I find extremely useful is … *pre-commit* agentic reviews.
For large changes, the review agents take 10x the time it takes the coding agent to create a patch, but is totally worth it, IMO.
Blogged: Post commit reviews
Where I question if pre-commit mandatory code reviews are truly needed and what a post-commit review world would look like.
Caveat: post-commit review doesn’t necessarily mean post-deploy review.
bit.ly/2Oid8jG
21
6
164
31,075
It’s disappointing to see people say all code is going to be slop and we should be ok with it, because humans weren’t writing the best code either.
Quality and reliability should be higher (as quantitatively measured), not lower.
Why’re we lowering the bar for ourselves? 😔
Quality and reliability should be higher (as quantitatively measured), not lower.
Why’re we lowering the bar for ourselves? 😔
20
22
256
12,287
Cindy Sridharan retweeted
Resharing this and urging everyone to read it for several reasons.
1. Everything Cindy writes is fantastic and serious engineering discourse
2. The examples here are concrete and clear and most importantly production services with real world consequences
1. Everything Cindy writes is fantastic and serious engineering discourse
2. The examples here are concrete and clear and most importantly production services with real world consequences
How to effectively debug with agents when you no longer read or understand the code
In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”
The TL;DR are the screenshots.
Link to post in the reply.
8
36
800
71,744