A mix of the techniques shown here, in short.
This has been my dream for nearly a decade now, and with agents I’m finally able to see this (partially) come true.
But turns out this is what I necessarily require to trust agent generated code. It’s not optional anymore, imo.
This has been my dream for nearly a decade now, and with agents I’m finally able to see this (partially) come true.
But turns out this is what I necessarily require to trust agent generated code. It’s not optional anymore, imo.
21
18
345
How to design good verification loops?
Well, that’s most of the job now, isn’t it?
Last week, I finally landed a refactor I started in Jan. It’s good I waited b/c the Jan version was buggy af.
This is the verification process I used for refactors (have different ones for perf)
Well, that’s most of the job now, isn’t it?
Last week, I finally landed a refactor I started in Jan. It’s good I waited b/c the Jan version was buggy af.
This is the verification process I used for refactors (have different ones for perf)
How does one ensure verification loops are actually correct? Rigorous needs to take cost into consideration as well.
26
16
522
Returning from international travel, followed by jet lag, physical exhaustion, remnants of a throat infection I picked up when traveling, dealing with a case of misplaced luggage across 3 carriers, catching up on work on a quasi-zombie state …
Yeah, I’m glad its the weekend ❤️
Yeah, I’m glad its the weekend ❤️
0
0
21
Speaking of metastable failures, one of the longest open tabs in my browser is this.
Yet to read the post, but wonder how much of this methodology overlaps with what you did for Aurora.
Only way to find out is to get around to reading the literature.
arxiv.org/pdf/2606.00942
Yet to read the post, but wonder how much of this methodology overlaps with what you did for Aurora.
Only way to find out is to get around to reading the literature.
arxiv.org/pdf/2606.00942
1
0
5
I found this unpublished blog post from 6y ago (look at the hand drawn diagram lol).
I’d totally forgotten about this, but the key ideas here (embracing non-determinism as a first class citizen via “error kernel” centric design etc) work super well as guidelines for my agents.
I’d totally forgotten about this, but the key ideas here (embracing non-determinism as a first class citizen via “error kernel” centric design etc) work super well as guidelines for my agents.
3
7
161
Lot of it holds up pretty well, actually.
The main changes in the last ~7 years have been in:
- move towards more flexible consistency
- use of replicated logs for all manner of things
- move away from strict consensus and synchronous replication
- control plane design choices
The main changes in the last ~7 years have been in:
- move towards more flexible consistency
- use of replicated logs for all manner of things
- move away from strict consensus and synchronous replication
- control plane design choices
1
0
3
I started this series on testing nearly a decade ago, but never published the final part copyconstruct.medium.com/testing-micros…
There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing.
Have a draft from years ago I hope I can wrap up in early 2027.
There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing.
Have a draft from years ago I hope I can wrap up in early 2027.
9
10
187
How to effectively debug with agents when you no longer read or understand the code
In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”
The TL;DR are the screenshots.
Link to post in the reply.
In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”
The TL;DR are the screenshots.
Link to post in the reply.
Writing a post about “how to effectively debug code when you no longer fully understand it”.
This is real bug I had to fix in ~9K LOC (100% AI generated).
None of the tactics I started with here helped “root cause” the issue, lol. Had to take a different approach to isolate it.
40
124
1,667
I’ve found that agents are great at debugging any problem when provided with this two-tier framework for troubleshooting.
This requires enforcing discipline and imposing constraints, rather than give agents a free rein or vague end goals, but works better than unguided “loops”.
This requires enforcing discipline and imposing constraints, rather than give agents a free rein or vague end goals, but works better than unguided “loops”.
18
12
220
Ball-knowers knew all along that Twitter wasn’t just going to collapse after Elon fired 75% of the staff, because Twitter 1.0 had some of the best infrastructure engineers working on it, and when you get that calibre of people building your foundations, it lasts.
18
8
400
With agentic coding, even in the best case scenarios (where you read every line of generated code), there’s something we’ve lost, perhaps irrevocably.
That has a real cost, & not to mention, second and third order effects, that “software factories” etc don’t necessarily solve.
That has a real cost, & not to mention, second and third order effects, that “software factories” etc don’t necessarily solve.
24
13
186
Writing a post about “how to effectively debug code when you no longer fully understand it”.
This is real bug I had to fix in ~9K LOC (100% AI generated).
None of the tactics I started with here helped “root cause” the issue, lol. Had to take a different approach to isolate it.
This is real bug I had to fix in ~9K LOC (100% AI generated).
None of the tactics I started with here helped “root cause” the issue, lol. Had to take a different approach to isolate it.
34
30
1,134
One thing I don’t see many talk about is how coding agents can often *amplify* weaknesses, both individual and organizational.
I think of this as the “empty calories” problem, which is exacerbated when “hyperproductivity” originates from your team’s “lowest common denominator”.
I think of this as the “empty calories” problem, which is exacerbated when “hyperproductivity” originates from your team’s “lowest common denominator”.
13
24
234
The people who blithely claim “you don’t need to understand the code anymore” aren’t telling you what debugging looks like in such cases
Here’s one example where I had to debug a gnarly problem at work with Claude *without* access to the code
Possible? Yes
“Easy”? Nope 🙂↔️
Here’s one example where I had to debug a gnarly problem at work with Claude *without* access to the code
Possible? Yes
“Easy”? Nope 🙂↔️
12
3
50
August farmer’s markets in the Bay Area are worth a visit just for the abundance of fresh, juicy figs, and in-season tomatoes. 🍅
Refrigerated, slightly overripe figs make for the best dessert, IMO.
Refrigerated, slightly overripe figs make for the best dessert, IMO.
3
1
28
adopted ASD-STE100 in my project upon a coworker’s recommendation.
Asking Opus 5 to write comments/commit messages etc in plain English has more benefits than just helping with readability.
It discovers bugs. Critical bugs.😱
I was absolutely gobsmacked after seeing this. Wtf!
Asking Opus 5 to write comments/commit messages etc in plain English has more benefits than just helping with readability.
It discovers bugs. Critical bugs.😱
I was absolutely gobsmacked after seeing this. Wtf!
18
14
476
In recent weeks, when it came to tricky bugs (a memory leak, a routing issue, a sharding bug), Claude’s initial analysis was almost always 100% wrong.
I had to give it multiple hypotheses, clues, ideas, suggestions for it to solve the bug.
When asked “so, what did you learn?”
I had to give it multiple hypotheses, clues, ideas, suggestions for it to solve the bug.
When asked “so, what did you learn?”
7
8
175
Went out to touch grass, but alas if you live in or around downtown San Francisco, there’s no escaping any discussion of AI, GPUs, agents etc, in public places. 😒
Maybe will try Marin or Sausalito the next time.
At least the nosh (Gochujang Pimento Cheese Twist) was superb.
Maybe will try Marin or Sausalito the next time.
At least the nosh (Gochujang Pimento Cheese Twist) was superb.
6
0
29
It’s a “people problem”, the oldest problem in the industry, not just a Google problem.
The cost of generating code going to zero *incentivizes* teams to unblock themselves ASAP, rather than dealing with the friction of coming up with cohesive, holistic, “cross-team” solutions.
The cost of generating code going to zero *incentivizes* teams to unblock themselves ASAP, rather than dealing with the friction of coming up with cohesive, holistic, “cross-team” solutions.
10
6
77
OSDI and SOSP are the two most prestigious systems engineering conferences.
Claude Code categorized a decades worth of papers into four buckets.
Directionally speaking, this looks about right. There’s less research happening in databases and compute than a decade ago.
Claude Code categorized a decades worth of papers into four buckets.
Directionally speaking, this looks about right. There’s less research happening in databases and compute than a decade ago.
2
0
24
In short, this ain’t happening, when you take the full software lifecycle into account
I mean, is your org as a whole solving 10x more high priority problems?
Are you shipping 10x more features?
Has your reliability gone up by 10x?
Can every engineer debug 10x faster!
I mean, is your org as a whole solving 10x more high priority problems?
Are you shipping 10x more features?
Has your reliability gone up by 10x?
Can every engineer debug 10x faster!
4
4
60
Elon should be the last fucking person talking about deaths caused by inexperienced appointees gives his DOGE cuts to USAID (overseen by 20 y/o kids with zero experience in government or public health) caused about 600K totally preventable deaths of the most vulnerable people.
4
5
50
The problem is that the economy is already in a recession outside the AI/data center buildout space.
If the economy continues to be this poor, with employers disingenuously attributing layoffs to AI, we’re guaranteed to see an anti-tech/AI backlash soon, and it won’t be pretty.
If the economy continues to be this poor, with employers disingenuously attributing layoffs to AI, we’re guaranteed to see an anti-tech/AI backlash soon, and it won’t be pretty.
2
4
46
This entire article on how AI adoption by employees at Dropbox went from low double digits to 90% seems AI generated.
Counted 15 em-dashes in total. 😅
Truly incisive and thoughtful writing is going to become an endangered art form in this AI era.
Counted 15 em-dashes in total. 😅
Truly incisive and thoughtful writing is going to become an endangered art form in this AI era.
7
2
39
Asked ChatGPT to analyze the very long write up and come up with ASCII diagrams explaining the most recent AWS outage that took out us-east-1 in 4 diagrams …
For those who don’t want to read walls of text, this might help
For those who don’t want to read walls of text, this might help
The detailed AWS incident report is out, and it’s worth a read
- DNS records managed by 2 systems; a race condition led to regional record getting unset
- EC2 lease establishment was borked as it depends on DynamoDB
- fluctuating NLB health checks leading to EC2 DNS entry purges
2
12
105
The detailed AWS incident report is out, and it’s worth a read
- DNS records managed by 2 systems; a race condition led to regional record getting unset
- EC2 lease establishment was borked as it depends on DynamoDB
- fluctuating NLB health checks leading to EC2 DNS entry purges
- DNS records managed by 2 systems; a race condition led to regional record getting unset
- EC2 lease establishment was borked as it depends on DynamoDB
- fluctuating NLB health checks leading to EC2 DNS entry purges
54
405
2,444
This is the definition of politicians having strong opinions on things they don’t understand. 🙄
Breaking up AWS won’t result in more reliable cloud providers, or customers suddenly starting to implement multi-AZ and failovers.
Breaking up AWS won’t result in more reliable cloud providers, or customers suddenly starting to implement multi-AZ and failovers.
22
18
408
Reading the AWS status updates, seems like what first diagnosed as a DNS resolution issue was either a red herring or an incomplete diagnosis.
As hours later, a system doing health checks on the network load balancers was deemed the culprit.
Full report will be fun to read.
As hours later, a system doing health checks on the network load balancers was deemed the culprit.
Full report will be fun to read.
9
36
398
This is right, but the primary reason tech discourse has downgraded so badly is because most of the serious accounts left/no longer post (think @mipsytipsy, @Lethain etc)
The ones who post are mostly farming for engagement, or try to be edgy or are virtue-signaling to VCs.
The ones who post are mostly farming for engagement, or try to be edgy or are virtue-signaling to VCs.
1
0
24
Remember when I said anon tpot isn’t technical or even very intellectual?
Case in point: they think one of the most famous and highest ranked engineers at Google who was so successful he retired in his 40s isn’t “grinding hard enough”
Chuds doesn’t even begin to cover it. 🙄
Case in point: they think one of the most famous and highest ranked engineers at Google who was so successful he retired in his 40s isn’t “grinding hard enough”
Chuds doesn’t even begin to cover it. 🙄
17
11
592
Twitter tech influencers of the 2015-2021 era were, by and large, deeply technical people who had deep insights. Think Kelsey Hightower.
“tpot” is mostly edgelord chuds engagement farming and being contrarian without any substance or even any pretense of intellectual integrity.
“tpot” is mostly edgelord chuds engagement farming and being contrarian without any substance or even any pretense of intellectual integrity.
43
44
1,035
The @nytimes doesn’t report “news” anymore.
Just “juicy collection of narratives” with click-baity headlines, which then unleashes a tsunami of hysteria against all kinds of targets.
Mainstream media, with notable exceptions, is 1-2 years “behind” most trends and happenings.
Just “juicy collection of narratives” with click-baity headlines, which then unleashes a tsunami of hysteria against all kinds of targets.
Mainstream media, with notable exceptions, is 1-2 years “behind” most trends and happenings.
1
1
31
This might be one anecdote, but what a soul-crushing environment to work in, with no leadership, no engineering mission, no camaraderie between coworkers, no clear ownership … just isolation and disorientation.
And to think of all the VCs who said go work for Twitter 2.0 …
And to think of all the VCs who said go work for Twitter 2.0 …
2
4
134
Massive, massive, absolutely massive #nokings protest on Market Street in SF happening right now. 🇺🇸❤️
0
3
22
Nice set of action items as well
- move toward “fail open” architecture
- not everything needs strong, instantaneous, global consistency.
- don’t run your monitoring on the infra it’s monitoring
Funnily, I blogged about some of these very lessons back in 2019 (2nd image).
- move toward “fail open” architecture
- not everything needs strong, instantaneous, global consistency.
- don’t run your monitoring on the infra it’s monitoring
Funnily, I blogged about some of these very lessons back in 2019 (2nd image).
7
8
253
Full report for yesterday’s Google outage
- not feature flagging a new code path
- null pointer crashing a binary when reading empty fields from Spanner
- remediation causing a thundering herd as there was no exponential backoff
- which required manual throttling to recover from
- not feature flagging a new code path
- null pointer crashing a binary when reading empty fields from Spanner
- remediation causing a thundering herd as there was no exponential backoff
- which required manual throttling to recover from
16
146
1,562
Preliminary GCP outage report is out, and as expected, it’s a bad config change pushed out globally by some automation.
A similar incident from 2020 comes to mind (though I bet there are countless other such incidents due to bad global config changes)
x.com/copyconstruct/…
A similar incident from 2020 comes to mind (though I bet there are countless other such incidents due to bad global config changes)
x.com/copyconstruct/…
A whole slew of Google services (Gmail, Firebase, GKE, GCE, almost every GCP service) had an outage on Apr 13 due to a wrong global configuration change (ha) and bugs in the canary system.
Global "centralized planet-space" control plane = global outage.
bit.ly/2yOyEb6
6
15
148
Dinner at the newly opened Jules, and the pizza was definitely worth the trouble it took to get the reservation.
They were awesome with accommodations, and the menu can be made vegetarian friendly.
They were awesome with accommodations, and the menu can be made vegetarian friendly.
1
0
16
Looks like this tweet got “shadow banned” because I can’t see it on my profile (see screenshot)
I need to go to the “Replies” tab to see it
Anyone encountered anything like this, where a single tweet vanishes from your main timeline on your profile?
I need to go to the “Replies” tab to see it
Anyone encountered anything like this, where a single tweet vanishes from your main timeline on your profile?
X engineer realizing what was obvious to many of us who paid attention *years* ago.
that Twitter 1.0’s infrastructure was so battle-hardened given the caliber of talent that worked on core infrastructure, that it would continue to pay dividends for years to come.
4
1
16
“The failover and redundancy should have worked, but did not”
Us infra eng greybeards remember that Twitter 1.0 had solved such “reliability” centric problems ~10y ago.
These problems span many teams, from hardware to traffic eng to storage, but if you’re fired 85% of them …
Us infra eng greybeards remember that Twitter 1.0 had solved such “reliability” centric problems ~10y ago.
These problems span many teams, from hardware to traffic eng to storage, but if you’re fired 85% of them …
2
8
97
X engineer realizing what was obvious to many of us who paid attention *years* ago.
that Twitter 1.0’s infrastructure was so battle-hardened given the caliber of talent that worked on core infrastructure, that it would continue to pay dividends for years to come.
that Twitter 1.0’s infrastructure was so battle-hardened given the caliber of talent that worked on core infrastructure, that it would continue to pay dividends for years to come.
14
71
923
The new right wants a white ethnostate, and see African, (non East) Asian and Muslim presence in Europe/Canada a civilizational threat.
It counts among its ranks many minorities, who also want much less diversity, like this Indian guy who clearly wants to live in 1950s America.
It counts among its ranks many minorities, who also want much less diversity, like this Indian guy who clearly wants to live in 1950s America.
3
0
1
The true endgame of having built a whole brand on being “anti-woke” is that ultimately you end up telling your own people to ignore true racism against them, because speaking up against it becomes “woke”, which of course, we can never have.🙃
2
1
50
Even the @nytimes is now negatively polarized against Elon’s support for H-1B and is printing misinformation.
The reason many stay on H-1B visas much longer than 6 years is due to per country green card caps!
Companies hire despite layoffs because they hire for different roles!
The reason many stay on H-1B visas much longer than 6 years is due to per country green card caps!
Companies hire despite layoffs because they hire for different roles!
12
18
124
JFC so much wrong with this demagoguery.
If you abolish the H-1B program, employers aren’t going to start hiring Americans in droves.
The jobs themselves will shift to Europe/Canada/India or some other place with a good tech ecosystem and plenty of easily hireable talent.
If you abolish the H-1B program, employers aren’t going to start hiring Americans in droves.
The jobs themselves will shift to Europe/Canada/India or some other place with a good tech ecosystem and plenty of easily hireable talent.
317
40
509
This is true about tech twitter.
A friend mentioned that he’d been approached for his last 2 jobs via Twitter DMs, but for his current job search is mostly using LinkedIn.
I’ve stopped posting about infrastructure engineering as most people I used to engage with have left.
A friend mentioned that he’d been approached for his last 2 jobs via Twitter DMs, but for his current job search is mostly using LinkedIn.
I’ve stopped posting about infrastructure engineering as most people I used to engage with have left.
5
4
55
A better summary:
“My Indian grandfather who voted for Trump thrice didn’t care at all when the racism was hurled at Mexicans, Muslims, Haitians, Chinese etc; because he looks down upon those people/cultures as well. But draws the line when the racism impacts *his* people”
“My Indian grandfather who voted for Trump thrice didn’t care at all when the racism was hurled at Mexicans, Muslims, Haitians, Chinese etc; because he looks down upon those people/cultures as well. But draws the line when the racism impacts *his* people”
3
5
63
Every time I order something at a restaurant ever since that plastics in food report came out, all I wonder is just how much plastic I’m consuming lol.
1
0
13
If you’re downwardly mobile or are stuck in cycles of inter-generational poverty, seeing thriving immigrant communities or the rich get vastly richer can be jarring, as none of the purported benefits of immigration or neoliberalism“trickle down” to you and make your life better.
4
2
35
Ok, Elon going thermonuclear on nativists and racists, defending the H-1B visa to *this*,degree wasn’t on my bingo card.
A very pleasant surprise, but given the amount of vitriol and hatred thats been aimed at immigrant tech workers, this is extremely welcome.
A very pleasant surprise, but given the amount of vitriol and hatred thats been aimed at immigrant tech workers, this is extremely welcome.
225
133
2,145
Love to see MAGA Indians being the first ones to “find out” after fucking around with the party of “eating cats and dogs”
You believed the people who dehumanized and smeared another defenseless immigrant community (Haitians) won’t come for you?
That you’re somehow “special”?
You believed the people who dehumanized and smeared another defenseless immigrant community (Haitians) won’t come for you?
That you’re somehow “special”?
14
8
185
Some are still pulling their punches, some are just capitulating.
Just how much influence Elon will have over the next 4 years remains to be seen.
Just how much influence Elon will have over the next 4 years remains to be seen.
1
0
4
