Durable Execution: How to Make AI Agents Survive Crashes, Timeouts, and Human Delays

An AI agent can reason correctly and still fail operationally. It may draft the right refund, call the right service, and then crash before recording success. When restarted, it calls the service again. The customer receives two refunds.

This is not primarily a model problem. It is an execution problem. Reliable agents need a way to preserve progress across process failures, delayed approvals, network uncertainty, and deployments. That discipline is called durable execution.

The central idea is simple: treat the agent’s work as a recoverable workflow, not as one uninterrupted model session. Decisions, side effects, and waiting states are recorded so execution can resume from a known point rather than begin again.

What Durable Execution Actually Means

Ordinary application code often keeps its immediate state in memory. If the process disappears, that state disappears with it. A durable workflow externalizes enough state to reconstruct what has happened and determine what should happen next.

A useful design separates four things:

• Workflow state: the case identifier, current step, accumulated evidence, and approval status.
• Decision history: the sequence of completed steps and the results they returned.
• Side effects: actions that change the outside world, such as issuing a refund or sending a message.
• Waiting conditions: timers, human approvals, or external events that may arrive long after the original process ends.

Durability does not require keeping a server process alive. Quite the opposite: a workflow may stop while waiting and later resume on another machine. The continuity lives in persisted history, not in the original process.

Why Agents Make the Problem Harder

Traditional workflows usually follow a predefined route. Agents may choose their next step dynamically, based on retrieved evidence or model output. That flexibility introduces three reliability hazards.

Model calls are not stable computations

Repeating the same request can produce a different answer. If a workflow replays after a crash and calls the model again, it may choose a different tool or formulate a different action. Therefore, consequential model outputs should be persisted as step results. Recovery should reuse the recorded result unless deliberate regeneration is part of the design.

External success can be locally ambiguous

A payment service may complete a refund while the agent times out waiting for the response. The agent knows only that the outcome is uncertain. Blindly retrying can duplicate the action; abandoning the action can leave the case unresolved.

Human time exceeds process time

An approval may take minutes or days. Holding a thread, container, or model context open is fragile and wasteful. A durable workflow records that it is waiting, releases compute, and resumes when an approval event arrives.

The Five Building Blocks

An idempotency key deserves particular attention. It is a stable identifier attached to a side-effecting request. If the same request is submitted twice with the same key, the receiving service should return the original outcome rather than perform the action again. A key might combine the workflow identifier and logical action name, such as case-482-refund.

Idempotency must exist at the boundary where the effect occurs. Recording a local flag before calling a payment service can suppress a necessary retry. Recording it afterward cannot prevent duplication if the process crashes between remote success and the local write. The remote service must recognize the key, or the workflow must reconcile uncertain outcomes before trying again.

Worked Example: A Refund Agent That Can Recover

Consider an agent handling refund requests. It gathers the order record, checks policy, drafts a recommendation, seeks approval for exceptional cases, issues the refund, and notifies the customer.

1. Start the workflow. Create a persistent workflow identifier tied to the refund case.
2. Fetch evidence. Call the order and policy systems. Persist their returned data or stable references to immutable records.
3. Ask the model for a recommendation. Require structured output containing the proposed outcome, rationale, confidence category, and evidence references. Persist the complete output.
4. Apply deterministic policy. Code checks whether the amount and reason permit automatic execution. The model may recommend; policy decides whether approval is required.
5. Wait if necessary. Record an approval request and suspend the workflow. Resume only when an event carrying the workflow identifier arrives.
6. Execute the refund. Call the payment service with an idempotency key derived from the case and action.
7. Record the receipt. Persist the payment provider’s refund identifier and status.
8. Notify the customer. Send a message with its own idempotency key, then mark the workflow complete.

Now suppose the worker crashes immediately after the payment service completes step six but before step seven is recorded. On recovery, the workflow sees no local receipt and attempts step six again. Because it uses the same idempotency key, the payment service returns the existing refund rather than creating another. The workflow stores that receipt and proceeds.

If the payment service does not support idempotency, recovery requires a reconciliation step. The workflow queries refunds for the order and verifies whether the intended action already occurred. Only when absence is established does it submit a new refund. This is slower and more complex, but safer than treating a timeout as failure.

Decide What to Persist and What to Recompute

Persisting everything creates storage and privacy burdens. Recomputing everything undermines reproducibility. The right boundary depends on whether a value is stable, expensive, sensitive, or consequential.

• Persist consequential decisions: tool selection, approved action, generated customer message, and policy outcome.
• Persist external receipts: transaction identifiers, delivery identifiers, and approval events.
• Recompute deterministic transformations: formatting or calculations derived entirely from persisted inputs, provided the code version is controlled.
• Reference large immutable artifacts: store an object identifier and integrity marker rather than copying a large document into workflow history.
• Minimize sensitive data: retain identifiers or redacted evidence when full content is unnecessary for recovery or audit.

Versioning matters. A workflow started under one policy or prompt may resume after a deployment. Changing its logic midstream can produce an impossible hybrid. Record the workflow, policy, prompt, and model versions used for consequential steps. Long-running workflows may need explicit migration rules or versioned branches.

Design Retries by Failure Class

“Retry on error” is not a sufficient policy. Different failures imply different responses.

• Transient infrastructure failure: retry with increasing delay and a limit.
• Rate limit: respect the provider’s timing signal or defer the step.
• Invalid request: do not repeat unchanged; repair the input or escalate.
• Ambiguous side effect: reconcile using the idempotency key or remote records.
• Policy denial: stop or route to a human; this is an outcome, not an outage.
• Repeated model-format failure: use constrained output, a repair step, or a deterministic fallback rather than an endless loop.

Place a retry budget around the workflow as well as individual calls. A step that alternates between services can otherwise consume resources indefinitely while never making meaningful progress. When the budget is exhausted, preserve the full state and create an actionable escalation containing the failed step, prior attempts, and known external outcomes.

Where Durable Execution Does Not Help

Durability preserves control flow; it does not make a poor decision correct. It cannot compensate for missing authorization, weak evaluation, inaccurate retrieval, or an unsafe tool interface. Nor does it automatically provide exactly-once effects. That property depends on cooperation between the workflow and the system performing the action.

There is also a complexity cost. Persisted histories, versioning rules, reconciliation paths, and retention controls all require engineering. A read-only assistant that answers disposable questions may not need them. Durable execution becomes valuable when work is long-running, expensive to repeat, dependent on humans, or capable of changing external state.

A Practical Adoption Sequence

Begin with one consequential workflow rather than converting an entire agent platform. Map every step, then mark each as deterministic computation, model decision, read, side effect, timer, or human event. For every side effect, define its stable identity and its reconciliation method.

Next, test failure boundaries deliberately. Terminate the worker before an external call, after remote success, while waiting for approval, and during notification. Confirm that recovery neither loses progress nor repeats an effect. Finally, inspect whether an operator can understand and resume a failed case without reconstructing it from scattered logs.

The decisive shift is to stop treating an agent run as a conversation that must remain alive. Treat it as a sequence of recoverable commitments.

Once those commitments are persisted, identified, and safely repeatable, an agent can survive the ordinary disorder of production: machines vanish, networks hesitate, people respond late, and software changes. Reliability then comes not from hoping the run completes, but from knowing precisely how it will continue when it does not.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.