Pinned Tweet
we're open sourcing a library we’ve been working on at Hatchet called pgoutbox. github.com/hatchet-dev/pg…
it implements a high-performance transactional outbox (specifically, for Go applications that use postgres via the pgx driver, but we'd like to make it available in other e been running it in production for the past 3 months at Hatchet, processing tens of thousands of messages per second. benchmarks on my local laptop have it going > 200k messages / second.
what's a transactional outbox? it's a pattern for solving a lost-write problem in distributed systems. imagine a user signs up for your app and you’d like to create a new row in your users table and send them a welcome email. if your API crashes or restarts after you commit to the users table, but before sending a welcome email, the write to the email service is lost forever. so instead, you write the message to an outbox table as part of the same tx as the user write.
obviously there’s a lot of intersection with task queues built on Postgres — transactional enqueuing is one of the main benefits! we had a few requirements that make typical task queue libraries a poor fit.
first, it needs to support hundreds of thousands of different topics with exclusive consumers. this can get a bit tricky to scale due to MVCC on Postgres; if you attach a lease with a heartbeat to each topic, you’ll perform wayy too many updates and accumulate lots of dead tuples.
second, it needs to support immediate message deletion with configurable retention policies per topic. this is quite different from a durable queue, where you’d like to preserve messages for future replays and auditing.
third, it needed a solid scaling story and APIs built around batching.
here for feedback / questions!
it implements a high-performance transactional outbox (specifically, for Go applications that use postgres via the pgx driver, but we'd like to make it available in other e been running it in production for the past 3 months at Hatchet, processing tens of thousands of messages per second. benchmarks on my local laptop have it going > 200k messages / second.
what's a transactional outbox? it's a pattern for solving a lost-write problem in distributed systems. imagine a user signs up for your app and you’d like to create a new row in your users table and send them a welcome email. if your API crashes or restarts after you commit to the users table, but before sending a welcome email, the write to the email service is lost forever. so instead, you write the message to an outbox table as part of the same tx as the user write.
obviously there’s a lot of intersection with task queues built on Postgres — transactional enqueuing is one of the main benefits! we had a few requirements that make typical task queue libraries a poor fit.
first, it needs to support hundreds of thousands of different topics with exclusive consumers. this can get a bit tricky to scale due to MVCC on Postgres; if you attach a lease with a heartbeat to each topic, you’ll perform wayy too many updates and accumulate lots of dead tuples.
second, it needs to support immediate message deletion with configurable retention policies per topic. this is quite different from a durable queue, where you’d like to preserve messages for future replays and auditing.
third, it needed a solid scaling story and APIs built around batching.
here for feedback / questions!
13
20
169
10,413
if you're ever squinting at a surprising database benchmark result, remember that in most systems, you can either maximize throughput or latency, but not both.
you see this all the way down the stack; batches to minimize network round trips, TCP buffering, fsync, all the way to the CPU instruction set level with SIMD optimization.
it's counter-intuitive because you would think that the lowest latency for each write maximizes throughput, because the writes are writing the fastest. but this isn't true. little's law tells us throughput = concurrency / latency.
you see this all the way down the stack; batches to minimize network round trips, TCP buffering, fsync, all the way to the CPU instruction set level with SIMD optimization.
it's counter-intuitive because you would think that the lowest latency for each write maximizes throughput, because the writes are writing the fastest. but this isn't true. little's law tells us throughput = concurrency / latency.
1
0
8
530
we're in the era of bad-faith weaponized llm-poisoning. this is pretty egregious.
we unintentionally did this to ourselves in the early days of @hatchet_dev where we talked about how we solved some scaling problems and the models ingested that as "hatchet has scaling problems" and it took a while to fix that.
the problem is that content engineers want to read (humble, down to earth, downplaying success, focused on problems) is the opposite of what you want models to ingest.
that's obviously not what's happening here, this case is much worse.
we unintentionally did this to ourselves in the early days of @hatchet_dev where we talked about how we solved some scaling problems and the models ingested that as "hatchet has scaling problems" and it took a while to fix that.
the problem is that content engineers want to read (humble, down to earth, downplaying success, focused on problems) is the opposite of what you want models to ingest.
that's obviously not what's happening here, this case is much worse.
Our loser competitors cant write good database software so they pay for this shit to make the models think we only do MySQL.
dev.to/grfdsasdvef/su…
0
2
43
8,261
when I first encountered channels in Go, I wasn't entirely sure when to use them. this is one of the first patterns I found incredibly useful: semaphores using channels. probably one of my most-used tricks for the past 4 years and just so clean & elegant.
1
0
5
553
if you'll excuse the throat-clearing, we're launching hatchet 2.0 next week. this is our biggest launch to date and incorporates a bunch of new features to solve the orchestration challenges of modern AI apps.
(it's not a video game, and also: it's not a pivot to sandboxes)
(it's not a video game, and also: it's not a pivot to sandboxes)
hatchet 2.0 - coming november 9 to a terminal near you
(it's not a video game)
2
3
12
538
this is basically the secret behind why @hatchet_dev aggressively uses range-based postgres partitioning. we've seen too many codebases using batched DELETEs in some kind of loop which can absolutely hammer your database and cause latency spikes and downtime.
we've even built some experimental features with very granular (hourly) partitioning to see how far we can push PG partitions. our use of partitioning this also led to us uncovering some bugs in Postgres, which one of our users reported upstream: postgresql.org/message-id/CAP…
one of the challenges with heavy partitioning is ensuring that every query against partitioned tables can prune partitions during the planning phase. so a lot of our queries have a bit of a weird shape, were there's a min_inserted_at parameter that gets passed in as a bound on the time column (for example, min_inserted_at <= inserted_at).
this is less of an issue in more recent postgres versions, but we need compatibility back to postgres 15. excited for the day we're all on PG 18+.
we've even built some experimental features with very granular (hourly) partitioning to see how far we can push PG partitions. our use of partitioning this also led to us uncovering some bugs in Postgres, which one of our users reported upstream: postgresql.org/message-id/CAP…
one of the challenges with heavy partitioning is ensuring that every query against partitioned tables can prune partitions during the planning phase. so a lot of our queries have a bit of a weird shape, were there's a min_inserted_at parameter that gets passed in as a bound on the time column (for example, min_inserted_at <= inserted_at).
this is less of an issue in more recent postgres versions, but we need compatibility back to postgres 15. excited for the day we're all on PG 18+.
DROP TABLE is the most scalable Postgres data-deletion strategy.
The reason? A complex interplay between Postgres MVCC, replication, and how indexes work.
3
0
8
455
biased but our @hatchet_dev engineering team is the strongest team of engineers I've worked with and we still do coding interviews. the difference is that we do them live and pair-program on a feature together. use claude code, cursor, etc - we don't care. we just want to see how you build and ship code.
the things we’re looking for in particular are:
- competency with tooling: how are you using/prompting your coding agent? Do you have an opinionated and productive setup?
- coding fundamentals: can’t really test this with coding output anymore, but asking people to explain or walk through their code has been a good signal.
- approach: get something working without sweating too many details right away (build first, then optimize)
I've been particularly surprised by engineers who are completely reliant on Claude Code and Cursor but absolutely horrible at using them. great engineers really care about their tools; there’s an element of specificity and clear thinking which translates directly to their use of coding agents.
for startups, the best signal by far is a paid trial period, but this likely excludes the most talented candidates who are already working impressive jobs.
the things we’re looking for in particular are:
- competency with tooling: how are you using/prompting your coding agent? Do you have an opinionated and productive setup?
- coding fundamentals: can’t really test this with coding output anymore, but asking people to explain or walk through their code has been a good signal.
- approach: get something working without sweating too many details right away (build first, then optimize)
I've been particularly surprised by engineers who are completely reliant on Claude Code and Cursor but absolutely horrible at using them. great engineers really care about their tools; there’s an element of specificity and clear thinking which translates directly to their use of coding agents.
for startups, the best signal by far is a paid trial period, but this likely excludes the most talented candidates who are already working impressive jobs.
The coding interview is dead, at least for my team at Anthropic. I've seen enough to know that it no longer provides signal. I removed it from loops this month.
I have a spicy, optimistic take - I think there's an opportunity for junior engineers: felixrieseberg.com/hunger-beats-t…
17
1
17
845
most of the open-source rebuilds ("<x> in Rust") we're seeing in the software world are what I call "unproductive phantoms" - they don't really contribute anything to society except for wasting tokens and more likely than not, they'll get abandoned / out of sync with upstream.
but w which I'd call "productive phantoms" which I think are providing actual, productive value and guidance to open-source maintainers. I think pgrust is one of these, given the number of upstream bug reports they've reported in Postgres. tsrs from @Cloudflare is another one, because it's built in close collaboration with the Typescript team. the common thread is collaboration with existing projects and maintainers rather than wholesale replacement. even if the projects fail, they'll still have contributed something to the ecosystem.
this got me curious, what would a productive phantom of @hatchet_dev look like? about a month ago, I started on trying to iterate on the Hatchet core internals with Fable using a 2ms time-to-start latency goal. it actually got there eventually, using a combination of a bunch of good ideas (some I directly prompted), a bunch of bad ideas (some of which I also directly prompted), some shortcuts, and a lot of extraneous code.
we've actually merged 8 of these ideas into the core Hatchet engine (after rewriting each into its own PR from scratch) with a few more coming! overall this was a success. not a "drop it and rebuild everything" type success, but lots of small improvements and iterations making the core engine better.
the latency metric isn't really the primary goal, the goal is efficiency and throughput, but if each tx is faster that generally helps with efficiency and throughput and it's much easier to measure locally.
but w which I'd call "productive phantoms" which I think are providing actual, productive value and guidance to open-source maintainers. I think pgrust is one of these, given the number of upstream bug reports they've reported in Postgres. tsrs from @Cloudflare is another one, because it's built in close collaboration with the Typescript team. the common thread is collaboration with existing projects and maintainers rather than wholesale replacement. even if the projects fail, they'll still have contributed something to the ecosystem.
this got me curious, what would a productive phantom of @hatchet_dev look like? about a month ago, I started on trying to iterate on the Hatchet core internals with Fable using a 2ms time-to-start latency goal. it actually got there eventually, using a combination of a bunch of good ideas (some I directly prompted), a bunch of bad ideas (some of which I also directly prompted), some shortcuts, and a lot of extraneous code.
we've actually merged 8 of these ideas into the core Hatchet engine (after rewriting each into its own PR from scratch) with a few more coming! overall this was a success. not a "drop it and rebuild everything" type success, but lots of small improvements and iterations making the core engine better.
the latency metric isn't really the primary goal, the goal is efficiency and throughput, but if each tx is faster that generally helps with efficiency and throughput and it's much easier to measure locally.
0
0
4
338
after a week of security incidents in the software world, we thought it'd be a good time to release something we've been working on for a little bit: FIPS-compliant hatchet images!
these basically use Tink / Go crypto primitives under the hood but disable certain features which aren't FIPS-compliant.
(it turns out bcrypt isn't FIPS-compliant, so we had to change how passwords get hashed for open source users when running the -fips build)
anyway, this removes a bunch of friction for self-hosting hatchet in really sensitive environments, which we think is a big win.
these basically use Tink / Go crypto primitives under the hood but disable certain features which aren't FIPS-compliant.
(it turns out bcrypt isn't FIPS-compliant, so we had to change how passwords get hashed for open source users when running the -fips build)
anyway, this removes a bunch of friction for self-hosting hatchet in really sensitive environments, which we think is a big win.
it's been a week. to end it on a high note, we're releasing FIPS-compliant images for everyone (not gated behind some enterprise sales maze).
you're either in one of two camps: you've never heard of FIPS, or you deal with it every day. this is for the second camp.
docs: docs.hatchet.run/self-hosting/f…
1
2
6
436
for everyone piling on to this post after the railway incident: remember that heroku had a roughly equivalent security incident in april 2022, pre-agentic coding, where a number of user environments were exfiltrated (github repos and oauth tokens, in this case, not prod runtimes, but similar impact). I don't think there's much you can extrapolate from this post unless there's a very direct RCA which says otherwise and pins the cause directly on agentic coding habits.
these platforms are fundamentally honeypots and are going to be targeted, and securing untrusted user code from deployment -> production is brutal.
these incidents obviously shouldn't happen, but it has more to do with the practices / priorities of the team than whether or not they use AI to code.
these platforms are fundamentally honeypots and are going to be targeted, and securing untrusted user code from deployment -> production is brutal.
these incidents obviously shouldn't happen, but it has more to do with the practices / priorities of the team than whether or not they use AI to code.
The implementation no longer matters
People really, really need to understand that
10
3
15
4,556
some more thoughts on this "human coherence" take since yesterday:
- agentic review doesn't solve this coherence/elegance problem, because review evaluates whether the presented diff is free of bugs, and rarely evaluates whether it's a globally optimal strategy. and if you do rely on agentic review to decide whether this is the optimal approach, you're going to waste a ton of tokens going back and forth across agents, which is just token churn handing money to the labs. this decision needs to happen earlier in the dev process.
- forcing developers to write their own PR descriptions is a great habit to ensure coherence. I've found myself opening PRs with the assistance of Claude Code where I can't distill _exactly_ what's important about the PR in my description, which forces me to go back to the code again.
- there are cases where it makes sense to trade off coherence in favor of velocity, it's ultimately a judgement call. the way that we think about it is that great software is produced by consistent, high-quality, high-velocity effort. the risk that you run with reduction of coherence is that the cost of fixing each bug or adding a new features grows exponentially with the complexity of your code (or, not even exponentially, any type of increase which is greater than the reduction in token cost is problematic). there's also the argument that "future models will get better," where you're implicitly making the bet that the rate of increase in model performance is greater than the rate of complexity in your codebase, which is not something I'd be comfortable betting on.
- our tiny human brains are really, really good at sensing that complexity is getting out of control when things become ugly, inelegant, or incoherent, and LLMs do not have this innate ability. they might in the future!
- agentic review doesn't solve this coherence/elegance problem, because review evaluates whether the presented diff is free of bugs, and rarely evaluates whether it's a globally optimal strategy. and if you do rely on agentic review to decide whether this is the optimal approach, you're going to waste a ton of tokens going back and forth across agents, which is just token churn handing money to the labs. this decision needs to happen earlier in the dev process.
- forcing developers to write their own PR descriptions is a great habit to ensure coherence. I've found myself opening PRs with the assistance of Claude Code where I can't distill _exactly_ what's important about the PR in my description, which forces me to go back to the code again.
- there are cases where it makes sense to trade off coherence in favor of velocity, it's ultimately a judgement call. the way that we think about it is that great software is produced by consistent, high-quality, high-velocity effort. the risk that you run with reduction of coherence is that the cost of fixing each bug or adding a new features grows exponentially with the complexity of your code (or, not even exponentially, any type of increase which is greater than the reduction in token cost is problematic). there's also the argument that "future models will get better," where you're implicitly making the bet that the rate of increase in model performance is greater than the rate of complexity in your codebase, which is not something I'd be comfortable betting on.
- our tiny human brains are really, really good at sensing that complexity is getting out of control when things become ugly, inelegant, or incoherent, and LLMs do not have this innate ability. they might in the future!
I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line man (we closed the former and merged the latter).
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
5
1
6
770
I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line man (we closed the former and merged the latter).
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
47
32
426
41,884
Our European takeover continues. We're so excited to have JB on the @hatchet_dev team!
please join us in welcoming jean-baptiste souvestre to the team! based in paris (ooh-lala), JB joins our growing engineering squad in europe. 🥖🇪🇺🍷
we asked him some questions to get to know him better, here's what he said:
➡️ light mode or dark mode?
🎤 what is a light mode?
cat man. tell us about your cats.
🎤 so i have 3 cats, kiki, chablis, and croche-pattes, all of whom are scoundrels in one way or the other. they love wet food, waking up at 4pm, and going to bed at 5.
➡️ how did you first come across hatchet?
🎤 i wanted to introduce hatchet to my last company because i liked the tech that much, but i figured it might just be more fun to work on it directly.
➡️ if i were in paris for one (1) day and wanted to get the best croissant, where would you suggest i go?
🎤 that's mamiche, in the 10th arrondissement, for sure.
➡️ what's something that surprised you in your first week here?
🎤 honestly can't decide between the tech maturity at such a young startup and the amount of custom frog emojis one can add to slack.
➡️ last, but not least: claude code or codex?
🎤 claude, but just because it was a french first name.
bienvenue à bord, JB! 🚀
2
3
13
563
Alexander Belanger retweeted
back in february, when we first released the @hatchet_dev CLI, we were pretty adamant that it was the CLI, and not an MCP, that would be the primary entrypoint for agents using Hatchet. and this has definitely been the case! but the rise of coding agents and the latest MCP spec are making MCPs much more usable across a wider range of services by supporting serverless request/response. and, frankly, we've been using MCP servers much more -- shoutout to @Posthog and @Figma.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
0
2
9
296
back in february, when we first released the @hatchet_dev CLI, we were pretty adamant that it was the CLI, and not an MCP, that would be the primary entrypoint for agents using Hatchet. and this has definitely been the case! but the rise of coding agents and the latest MCP spec are making MCPs much more usable across a wider range of services by supporting serverless request/response. and, frankly, we've been using MCP servers much more -- shoutout to @Posthog and @Figma.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
new: hatchet mcp
trigger, inspect, and replay runs from your coding agent
docs: docs.hatchet.run/reference/cli/…
3
1
11
744
We've gotten some great feedback already on our new overview page, excited to share it with everyone!
WE have a NEW overview page 🧨 🧨 🧨
that's right, you read that correctly, hatchet now has a fancy, brand-spankin'-NEW (!!!!) ~ overview page ~ that shows you ALL the stats you might need to know
0
1
5
291
happy friday, everyone! our low level systems person @vqtbl4q has been permitted to take over our socials for the day, i think this was a great idea
🪓 durable execution: coming to theatres near you.
3
1
9
999