I run tests on CI. They fail.
Claude suggests retrying because they might be flaky.
I retry. They fail. Again.
Claude thinks they’re deterministic. Starts investigating.
My paranoid ass thinks they’re actually flaky. I’m just unlucky.
I retry for 3rd time. They fail again.
Claude is now adamant there’s a genuine issue. Spends 2 hours and millions of tokens. Otherwise three flaky tests in a row have the probability of 0.0037% which is impossible.
Are my Senior SWE instincts a joke to you??? You shouldn’t underestimate my autistic stubbornness.
I retry again. 4th time.
They pass.
They were indeed flaky.
Claude suggests retrying because they might be flaky.
I retry. They fail. Again.
Claude thinks they’re deterministic. Starts investigating.
My paranoid ass thinks they’re actually flaky. I’m just unlucky.
I retry for 3rd time. They fail again.
Claude is now adamant there’s a genuine issue. Spends 2 hours and millions of tokens. Otherwise three flaky tests in a row have the probability of 0.0037% which is impossible.
Are my Senior SWE instincts a joke to you??? You shouldn’t underestimate my autistic stubbornness.
I retry again. 4th time.
They pass.
They were indeed flaky.
Oct 21, 2026 · 10:36 AM UTC
65
4
533
32,571
@ChShersh Did you mark them as “flaky” or left this treasure to another fortune hunter?
1
0
3
1,175
@dilorias Asked in the team chat, they were fixed within minutes
0
0
6
1,079
@ChShersh Tests are flacky when they fail occasionally. When they work occasionally they should be called different
0
0
1
132
@ChShersh You need to make the tests non-flaky. That's real work though. And depending on the reason, may require far more effort than seems reasonable.
0
0
0
59
@ChShersh This’ll be a nice 5 pointer ticket to fix ci. Maybe two, take your time do it right
0
0
0
269
@ChShersh Out of joke bro, some tests uses random gen to define the test and I would say those fails “maybe” are your luck to find real issues. I had opposite paranoid and my tests pass I run again and again till it fails and that is my luck before shipping to production
0
0
1
310
@ChShersh Lol the 4th retry passing is the most tester thing ever. Still worth finding out why its flaky tho
0
0
0
34
@ChShersh Perhaps the code really is broken and the flakiness causes the tests to occasionally pass.
0
0
1
190
@ChShersh sounds like a classic case of CI roulette, flaky tests can be a nightmare to track down
0
0
0
150
@ChShersh That probability only works in perfect conditions, CI are not that as there is a bunch of hidden variables which would change the probabilities.
0
0
0
163
@ChShersh Happening to me pretty often with AI generated tests :D
It is always either unsuitable combination of business logic from faker factories or some time comparison issues.
It is always either unsuitable combination of business logic from faker factories or some time comparison issues.
0
0
0
280
@ChShersh I would say the job now is to make your test stable now understanding why this test is flaky
0
0
0
198
@ChShersh the 2 hours is the part that hurts. i built a way to talk to claude code while it runs, so at minute 5 you just say "wait stop stop stop, they're flaky" and it actually stops. would've saved you a few million tokens: github.com/CakeCrusher/fu…
0
0
1
95
@ChShersh Yea an LLM number mixer just isn't going to beat a human with hundreds of thousands of circuits dedicated to simulating and modeling everything.
I've been putting together a series on why LLMs have these failures and how they could be corrected.
youtube.com/watch?v=dRajUw…
I've been putting together a series on why LLMs have these failures and how they could be corrected.
youtube.com/watch?v=dRajUw…
0
0
0
16
@ChShersh The tests were flaky. Claude was deterministic about it.
0
0
0
1
@ChShersh i once wrote some tests around concurrency that were guaranteed to prove correctness for something only to discover that the kubernetes runners in ci didn't like it (limited resources iirc) and decided to fail the tests for ??? reasons
0
0
3
163
