Hatchet

The decision

Your AI pipelines chain a paid model call to steps that fail every now and then, like a render, an upload or an API with a rate limit, so when one of them falls over halfway, what does recovery cost and who finds out? This one is about whether to put those pipelines on a durable workflow engine, who should own it, and what it saves you in model spend.

The problem it removes

Most AI pipelines start life as a loop in a script, and that works fine until a step times out after the expensive part has already run, or a machine restarts in the middle of a batch. Then the output is lost, the batch runs again from the top and every model call gets paid for twice, and nobody sees the waste because nobody is measuring it.

What I ran

My job-search pipeline tailors a CV with a model call, renders it to PDF in Chrome and then checks the PDF, and the render times out every now and then with no retry around it in my code.

I ran twelve jobs through the plain loop and through Hatchet with the same injected render failures, and then again with the process killed halfway.

The plain loop paid for twelve model calls and delivered ten CVs, and killed and restarted it paid for eighteen and still delivered ten.

Hatchet delivered all twelve both times. After the crash it paid for fifteen, because it kept the finished steps and only the three calls running at the moment of the kill ran again, with waiting work moving to a new worker in four seconds and the interrupted steps following after twenty-nine.

The model step was a stand-in, so I counted calls instead of euros.

Where it sits in the stack

Around the pipeline, between the code that does the work and the machines that run it. Each step becomes a task with retries, timeouts and a stored result, a small server on Postgres keeps track of every attempt, and any worker can pick up where another one stopped. It fits agents too, since a long wait for a human approval becomes a paused step and no process has to sit around waiting.

The first 30 days

Week 1. Pick the pipeline with the most expensive step and the flakiest neighbour, run the server next to your existing Postgres and wrap the steps you already have. My workflow code was about forty lines and left the step code untouched.

Weeks 2 to 4. Set retries and timeouts per step, make the paid steps safe to repeat, and put the dashboard in front of whoever is on call.

Skills. An engineer who is comfortable with queues and Postgres, and no new data store.

Running cost. One small service and the database you already have.

Working at day 30 means a failed step retries on its own, a restart only costs what was running at the time, and someone can see every attempt of every job.

Risk

Durable means at-least-once. A step that was running when a worker died runs again, so if that step charges a customer or sends an email, make it idempotent or split it so the part that must not repeat is as small as possible.

Defaults built for short tasks. Steps time out after sixty seconds unless you say otherwise, and long model calls take longer than that, so set timeouts per step on day one.

Pace of change. There were a hundred releases in six months across the server and SDKs, and I ran into a deprecation on the very first day, so pin your versions and read the release notes before upgrading.

Maintainers. Four people carry most of the work and the company behind it sells a hosted version, but the self-hosted server is MIT and complete for this use, and your way out is the step code, which stays plain functions.

The organisational risk is that the pipeline turns into a platform nobody owns, so give it an owner in platform engineering and give the AI lead the list of steps that must be safe to repeat.

Build, adopt or buy

Build. You can make a loop skip finished jobs and retry a flaky step, and that covers the simple crash, but the months go into the rest of it: steps in flight on a dead machine, backoff, concurrency limits, human approvals and a record of every attempt.

Buy. Hosted workflow services, including this project's own, take the server off your plate, which is worth it when nobody wants to run one or when the volume grows.

Adopt it self-hosted wherever Postgres is already run well, and keep the step functions plain so you can swap the engine later.

Verdict

I'd adopt it. On my pipeline it turned lost work into delivered work and cut the cost of a crash down to the steps that were in flight, and all it asks for is one service and a database you probably already have. Before you roll it out, list every step that must not run twice and make each one safe to repeat, because a durable engine will happily charge a customer twice if you let it.