Retry-after-crash: the second purchase

A handler that is idempotent on paper buys twice when the process dies between the provider returning and the record being written. The window is milliseconds wide, and it is where the money goes.

A fulfilment handler does three things in order. It receives the event. It calls a provider to buy the thing the customer paid for, which takes seconds. It writes down that the purchase happened, and only then answers Stripe with a 200. The order of the last two steps is not a choice: you cannot record the provider's answer before you have it.

So there is always a moment after the provider has returned and before the record exists. If the process dies in that moment, two things are true at once. The item was bought. Nothing anywhere says so. Stripe never received a 200, so it retries, and the retry arrives at a handler with no record of the purchase and no idempotency key to stop it. It buys again.

This is not a race. Nothing runs concurrently. It is one crash at the wrong instant, and the instants that qualify are ordinary: a rolling deploy sending SIGTERM mid-request, an out-of-memory kill, a container restart, a request timeout that tears down the handler while it is still running.

The setup

The naive example in second-run again. It has the idempotency check, and the check is exactly where the problem is:

return async function handle(event) {
  if (done.has(event.id)) return { kind: "duplicate" };
  const r = await provider.fulfil(event.orderId);          // the expensive part
  //  ← a crash here has bought the item and recorded nothing
  orders.set(event.orderId, { status: "provisioned", r });
  done.add(event.id);
  return { kind: "provisioned" };
};

The key is written last. It has to be, in this shape: writing it first would mark an event handled before the purchase, and a crash on the other side of the provider call would then lose the order entirely. Either way the record and the purchase are two writes to two systems, and a crash between them leaves them disagreeing.

The reproduction

second-run's retry-after-crash scenario arms a one-shot crash, delivers the event once, lets the crash happen after the provider call, advances the clock three minutes for Stripe's retry, and delivers the same event again:

const crashOnce = { armed: true };
const handle = await build({ provider, clock, crashOnce, state });
const event = ev("evt_2", "ord_2");
try { await handle(event); } catch { /* the simulated crash: expected */ }
clock.advance(3 * 60_000);
await deliver(handle, event, rejections);
const n = provider.calls.length;

Against the handler above:

FAIL  retry-after-crash — the first attempt crashes after the provider call, then Stripe retries
      observed: 2 provider call(s), final status "provisioned"
      expected: exactly 1 provider call, never a second purchase, final status "provisioned"

Look at what is right. The final status is provisioned. The customer gets their item. Every dashboard shows a healthy order. The only evidence of the bug is on the provider's invoice, where two purchases sit against one order, and in the provider's inventory, where an item nobody will ever be shown is now yours. In the system this pattern was extracted from, the items were eSIM profiles. A second profile bought and never delivered is not a rounding error; it is the whole margin on the order, gone silently.

Why the suite passed

Process death is not an exception the tests can throw. A unit test can make the provider fail, and the suite had that: provider throws, handler records failure, retry works. What no test did was make the handler itself die after the provider had succeeded. There was no hook to die through, and nobody writes "throw after the third line" tests without one.

The window is too small for a load test to hit. A few milliseconds between two statements, in a request that takes seconds, under a test that restarts nothing. The probability per request is near zero. The probability across a year of deploys is not.

The outcome looks correct. Even a test that stumbled into it would have asserted on the order's status, seen provisioned, and passed. The assertion that catches this bug is on the provider's call count, and nobody asserts on the provider's call count unless a tool makes them.

The fix, in two parts

First, shrink the window to the minimum the shape allows. The write of the provider's answer becomes the very next statement after the provider returns. Nothing awaited between them. No log line, no metric, no notification email, no second database call. Those all go after the record is durable. From the fixed example:

const r = await provider.fulfil(event.orderId);
// Persist the provider's answer BEFORE anything else can fail. The window
// between "provider returned" and "we wrote it down" is where a crash turns
// a retry into a second purchase. second-run's retry-after-crash scenario
// found exactly that in an earlier version of this file.
const w = settle(event.orderId, c.claimId, { status: "provisioned", r });

That comment is a confession. The earlier version of that example did one more thing between the provider returning and the record being written. second-run crashed it there, the retry bought again, and the fix was to move one line. The tool found the bug in the file that was supposed to demonstrate the fix.

Second, make the purchase itself safe to repeat, because the window cannot be zero. However tight the code, the process can still die between the provider's response arriving and the write completing. The only thing that closes that gap is on the provider's side: an idempotency key on the purchase call, derived from the order id, so that a second call for the same order returns the first purchase instead of creating another. Stripe's own API accepts an Idempotency-Key header for exactly this reason, and most providers that sell anything have an equivalent, or a lookup-by-reference call you can make before buying. If yours has neither, the retry path after an expired claim should ask the provider what it already did before it does anything.

The claim from the previous two pieces does a third job here. The first attempt claimed the order before buying. After the crash, that claim is still there, pending, with a lease. The retry sees in_flight until the lease expires. That is not the fix, since after expiry it would buy again, but it is the signal: an expired claim with no result recorded means "an attempt got as far as the provider and we do not know what happened." That state should route to the reconciliation path, not the purchase path.

Five checks, one hour

1. Find the line that writes the provider's answer. Count the statements between the provider returning and that write. Every one of them, logging included, is part of the window. Move them after.
2. Ask your provider for an idempotency key on the purchase call. Pass the order id. Then call it twice in a test and assert on one purchase. If the provider has no such key, find the lookup-by-reference call and use it before every purchase that follows an expired claim.
3. Read your deploy. Does a rolling restart send SIGTERM to a process with a request in flight, and does the server drain before exiting? A handler that buys in three seconds and a shutdown grace of two seconds is this bug on a schedule.
4. Check what your handler does with an expired claim that has no result. If it buys, that is the residual window turned into a second purchase. Route it to a check with the provider first.
5. Run the scenario. node src/cli.js your-handler.js --only retry-after-crash in a checkout of second-run reports the provider call count, which is the number that matters.

What this generalises to

Any external side effect followed by a local write has the same window: send the email, then mark it sent; create the shipment, then store the tracking number; charge the card, then write the order. The crash between them produces a duplicate email, a duplicate shipment, a duplicate charge, and in every case the local record looks fine afterwards. The rule is the same each time. Write immediately after the effect with nothing in between, make the effect idempotent by reference on the far side, and treat "we started and do not know" as its own state with its own path.

The code

The scenario, the naive handler, and the fixed one, with the comment above still in it: github.com/GG5533/second-run. The claim and settle pattern with both storage backends: github.com/GG5533/stripe-fulfilment-kit. Neither needs a Stripe account or a network to run.

Sami Habbal fixes the failures that only show up on run number two. Charlotte, NC. github.com/GG5533