Durable Agent Runtime

45 — Durable Agent Runtime: Crash Recovery and External Effects

This project evaluates journal-based workflow recovery and external-effect idempotency using a controlled refund workflow. It injects failures at defined crash points and measures recovery, duplicate effects and replay behavior.

The principal result concerns the boundary between the runtime and the payment gateway:

The evaluated 40-step workflow has 42 crash points. At 41 points, the recorded journal state determines the replay action. At the remaining point, the gateway may have applied the refund before its outcome is recorded locally. Recovery at that boundary depends on the retry policy and the gateway's deduplication contract.

Everything below is measured by src/experiments.ts and regenerated into docs/results.md on every build.

Problem and scope

A restart can leave a workflow uncertain whether an external effect completed. Idempotency keys address this only when the runtime's retry behavior and the remote system's deduplication behavior form a compatible contract.

This self-directed experiment evaluates that contract by crashing before each journal write, recovering the workflow and counting refunds under different gateway and retry policies. The results describe this controlled workflow and its explicit assumptions.

What came out of it

1. Exactly-once held at all 42 crash points — under exactly one set of assumptions.

Row 2 changes nothing in this repository. It changes the remote system. Exactly-once is a property of the pair — runtime and gateway — and the runtime alone cannot promise it. Row 3 changes one line here: generating a fresh key instead of reusing the recorded one. That line looks correct in review.

2. Nondeterminism inside a step is harmless; between steps it is fatal.

Ten workflows containing patterns I have written without thinking. Four were caught by replay. Two were latent — genuinely nondeterministic, invisible to an in-process replay, and therefore reported clean by a test suite. One diverged silently, where no guard can possibly see it. Three were inert, and that is the useful part: their nondeterminism happened inside a ctx.step body, so it was journalled on the first attempt and replayed verbatim.

That gives a rule you can actually apply in review, which "workflows must be deterministic" does not:

Randomness inside a step is recorded. Randomness between steps changes the sequence of operations and destroys the run.

3. The latent class presents as a flaky test. branch-on-wall-clock was caught when the machine was busy and missed when it was not. That is not a bug report, it is a retry annotation — and then it is load-bearing in production.

4. A resumed run invents a budget overrun that never happened. Work costing 200 cents against a 250-cent limit reports 310 cents after one crash, and aborts. The counter lives on the runtime object, and the runtime object is the one thing that does not survive a crash. It fails hardest on the runs that crashed most.

5. The approval gate survives because it is not a promise. awaitSignal unwinds the stack rather than blocking. Five restarts, zero steps re-executed, zero journal growth. An approval modelled as an unresolved promise passes every test, because tests do not redeploy.

Eight of thirteen predictions written before the experiments were contradicted, and the scoreboard in §7 of the results says which.

Running it

Node 22+ and PowerShell 7. No dependencies, no package.json, no build step — Node executes the TypeScript directly by stripping types, and stage 5 of the test harness fails the build if a non-builtin import ever appears.

pwsh build.ps1     # loads every module, regenerates docs/results.md
pwsh test.ps1      # 6 stages: 143 tests, mutation testing, secrets scan
pwsh demo.ps1      # a five-minute tour that runs the real experiments

test.ps1 includes a mutation stage. Eight single-line edits to load-bearing decisions — removing the sequence-gap check, generating a fresh idempotency key, resetting the budget on resume, ignoring a late signal — and all eight must make the suite fail. 8/8 killed.

The first version of that harness reported 8/8 while every mutant run had actually died on a module-resolution error before executing a single test. It now refuses to score a run whose suite did not start. A mutation score is worth exactly as much as the check that the mutant ran.

How it is built

src/journal.ts       append-only event log; serialise/parse tolerating a torn tail
src/ledger.ts        a refund gateway that can be told to ignore idempotency keys
src/runtime.ts       replay, effects, signals, budget, nondeterminism guards
src/workflows.ts     the refund workflow and the 10-entry nondeterminism corpus
src/experiments.ts   the six experiments
src/predictions.ts   written first, scored mechanically, not edited afterwards
src/report.ts        generates docs/results.md and docs/results-stable.md
tools/mutate.ts      the mutation harness

The load-bearing decisions are in docs/adr/:

1. The journal records decisions, not state
2. Idempotency keys are positional, never generated
3. No build step
4. Suspension is an unwind, not a blocked promise
5. The unknown window gets a policy, not a fix
6. The budget is re-derived from the journal

What this deliberately does not show is in docs/known-limitations.md — no real I/O, no concurrency, an in-memory journal, and one workflow shape. The security posture is in docs/security-review.md.