A lease is not a fence

How a Stripe fulfilment handler passed 78 tests and would still have charged one customer three times.

Sami Habbal · 14 September 2026 · Charlotte, NC

Most advice about webhook idempotency stops at the idempotency key. Store the event id, check it before you do the work, skip the event if you have seen it. That is correct and it is not enough, because it protects you from the same delivery arriving twice. It does not protect you from two deliveries arriving at once, and it says nothing about what happens when the work you started for the first delivery is still running when the second one finishes.

This is the story of a handler that had the key, had a lease on top of it, had 78 passing tests including concurrency tests, and would still have lost a paying customer's order and charged them a third time. The code never ran in production. It was caught in review, and the failure below is what it would have done, reproduced by a test.

The setup

The handler fulfils digital goods after a Stripe Checkout session completes. Fulfilment means calling a provider to buy the item, which can take seconds, and can hang.

To stop two concurrent deliveries of the same event from both buying, the handler takes ownership of the order before doing anything: it claims the order with a two-minute lease. A live lease returns 409 so Stripe retries later. A provisioned order returns 200 duplicate. An expired lease or a failed order can be claimed again.

The store is four methods, and the claim logic is a pure function so it can be tested with an injectable clock:

export type ClaimResult =
  // The caller owns provisioning and must `complete` or `fail` with this id.
  | { kind: "claimed"; claimId: string; record: OrderRecord }
  // Already done. A duplicate delivery. Do nothing.
  | { kind: "provisioned"; record: OrderRecord }
  // Another attempt holds a live lease. Come back later.
  | { kind: "in_flight"; record: OrderRecord };

Standard pattern. Every concurrency test passed. Two deliveries at once bought exactly one item; the loser got a 409.

It was still wrong.

The bug

A lease bounds how long an attempt owns the order. It does not reach into the provider and stop a call that is already in flight.

        0:00                     2:00 lease expires              2:40
          │                            │                           │
 A   ├─── claims ─── provider call hangs ─────────────────────► throws
                                                                    └─► writes "failed" ✗
 B                                ├── reclaims ── succeeds ──► "provisioned" ✓

Attempt A claims the order and calls the provider. The call hangs. Two minutes pass, the lease expires, and a Stripe retry, attempt B, claims the order again. B's provider call succeeds. B writes provisioned with the item the customer paid for. Correct.

Forty seconds later A's dead call finally throws. A does what any sensible handler does on failure: it records the error so the order can be retried. Its write lands on top of B's success.

Three things now go wrong, and none of them is visible from where it happened:

The customer's delivered item disappears from their order page. The record says failed.
A failed order is claimable. The next Stripe retry claims it, buys the item again, and charges the customer a third time.
Support sees a failed order. Accounting sees a successful charge. Nobody sees the item that was actually delivered, because the record that said so was overwritten.

Every component behaved reasonably. The lease did its job: it stopped A and B from working at the same time. The failure handler did its job: it recorded a failure. The retry did its job. The bug is in the space between them, where a write from an attempt that no longer owns the order is still allowed to settle it.

Why 78 tests missed it

Every test settled an order from the attempt that currently owned it.

That is the natural way to write these tests. You claim, you complete, you assert. You claim, you fail, you assert the retry works. You claim twice concurrently, you assert one wins. All of that exercises the lease doing its job.

The uncovered case was a write arriving from an owner that had already been replaced, which is precisely the state a lease exists to create. Every time a lease expires and a new attempt takes over, there is a previous attempt somewhere that may still be running. The tests covered the mechanism working. They never covered the mechanism being superseded.

The general lesson transfers to any lock, lease, or ownership token: test the write that arrives after ownership has moved on. If your system has a way for ownership to change hands, then by construction it has a moment where the old owner is still alive. Write that test first.

The fix: fencing tokens

The idea is old and comes from distributed locking. When you take ownership, you get a token. Every write that settles the resource must present the token, and a write carrying a superseded token changes nothing and says so.

claim() already returned a claimId. The change is that complete and fail now require it, and the store checks it before writing:

const settle = (sessionId: string, claimId: string, fields: Partial<OrderRecord>) => {
  const record = must(sessionId);
  // Fence: only the attempt that currently holds the lease may settle it.
  if (record.claimId !== claimId) return { kind: "superseded", record };
  Object.assign(record, fields, { claimId: null, claimedAt: null });
  return { kind: "written", record };
};

Two details matter more than the check itself.

The rejected write is a first-class result, not an exception. WriteResult is either written or superseded, and the caller has to handle both. That is deliberate. A superseded fail is harmless and can be logged. A superseded complete is not harmless: it means a superseded attempt's purchase succeeded, so an item was bought that nobody will ever be shown. That is real money that needs reconciling, and the handler reports it loudly with both order ids rather than quietly dropping it. Making the caller look at the result is how that report happens.

The fence has to hold across processes. In memory the check above is atomic because nothing in it awaits. In the SQLite store the same check runs inside BEGIN IMMEDIATE, so two instances of the handler behind a load balancer cannot both pass it. A fence that only works within one process is a fence that works in the test suite and not in production, which is exactly the class of bug this started as.

The tests that hold it shut

Written once as a behavioural suite and run against both storage backends, so the two cannot drift:

a stale failure cannot undo a provisioned order
a stale completion cannot overwrite the item the customer was shown
a stale write is rejected even when the newer attempt failed
a settled order rejects a second write from the same token
an unknown token is rejected outright
a provisioning call that outlives its lease cannot fail the order another attempt provisioned
an item bought by a superseded attempt is reported as orphaned, not written over the live one
two concurrent deliveries buy exactly one item; the loser gets a 409 to retry later
sqlite: a stale write from another process is fenced off
concurrent claims for one session yield exactly one owner

The first one is the bug from the diagram, verbatim:

suite("a stale failure cannot undo a provisioned order", async (store, c) => {
  const slow = await claimed(store);   // attempt A starts, then overruns
  c.advance(DEFAULT_LEASE_MS);
  const fresh = await claimed(store);  // attempt B re-claims and succeeds
  await store.complete(snapshot.sessionId, fresh, esim);

  const stale = await store.fail(snapshot.sessionId, slow, "provider outage");
  assert.equal(stale.kind, "superseded");

  const record = await store.get(snapshot.sessionId);
  assert.equal(record?.status, "provisioned", "the customer's eSIM must survive");
  // ...and the order must not have been re-opened for a third purchase.
  assert.equal((await store.claim(snapshot, DEFAULT_LEASE_MS)).kind, "provisioned");
});

The last assertion is the one that matters commercially. It is not enough that the customer's item survives. The order must also not be claimable again, because a claimable order is a third charge waiting for the next retry.

In the application this was extracted from, the suite went from 78 tests to 92 at the fencing commit. The standalone kit is smaller, 59 tests, and every fencing test above is in it.

What this generalises to

The same shape appears anywhere ownership can move while work is in flight:

A job queue with visibility timeouts, where a worker that stalled past the timeout finishes and acks a message another worker already processed.
A distributed lock with expiry, where the original holder's write lands after a new holder took the lock. This is the case fencing tokens were invented for.
A saga or workflow engine where a step's compensating action fires after the step was retried and succeeded.
Any "mark as failed on exception" handler wrapped around a call with no upper bound on its duration.

The question to ask of each is the same. Not "can two owners run at once?" A lease answers that. The question is "what happens to a write from an owner that has already been replaced?" If the answer is "it lands," you have this bug, and your tests almost certainly settle every resource from its current owner.

How it was found

Adversarial review. A second reviewer was given the code with a mandate to find the failure rather than approve the work, reproduced it with the injectable clock, and reported it before it reached production. That is now the standard for anything I write that touches money: the person or tool reviewing it is asked to break it, and the test that reproduces the break is written before the fix.

Try it on your own handler

The sequences above, plus a crash before the purchase and an unpaid-before-paid ordering, are packaged as a zero-dependency tool that runs them against any handler, reports what it actually did, and emits a regression test: second-run (github.com/GG5533/second-run). While I was writing its "correct" example, the retry-after-crash scenario failed against it: a crash between the provider returning and the record being written let the retry buy again. That is the tool doing its job on its own author.

The code

Full implementation, both stores, every test above: github.com/GG5533/stripe-fulfilment-kit

npm install && npm test runs the suite with no network and no Stripe account.

Sami Habbal fixes the failures that only show up on run number two. Charlotte, NC. github.com/GG5533

← Back to the offer · second-run on GitHub