Durable Workflow Execution and Recovery Explained

Updated on Oct 13, 2026

10 min read

Distributed applications increasingly run processes that outlive a single request: a payment may wait for fraud review, a deployment may pause for approval, and an AI task may call several external services. Durable workflow execution makes this work resumable when a worker restarts or a dependency fails, without requiring application code to keep every step in process memory.

Why Durable Workflow Execution Is Being Discussed

As teams connect APIs, queues, cloud services, and human approvals, a workflow can remain active for minutes or months. A chain of ordinary function calls is easy to write, but a process crash can erase its local variables and leave operators unsure which actions already happened. Retrying the whole request may repeat a payment, send a duplicate notification, or strand work between services.

Durable workflow engines address this gap by storing execution progress outside the worker process. Interest in the model has grown alongside long-running automation and agent-like systems, where work can pause for external input and must continue later. Durability is not a new kind of business logic; it is an execution model for preserving that logic’s progress.

What Is Durable Workflow Execution?

A workflow is a named, stateful process made up of steps, decisions, waits, and external actions. Durable workflow execution means the system records enough information about an execution to reconstruct its state after a worker or machine fails. A replacement worker can continue the same workflow rather than starting a new copy from the beginning.

The Temporal project is one example of this approach. Its workflow documentation describes workflow definitions as code that produces workflow executions. The application defines the process; a workflow service persists its progress; and workers run the code and perform assigned tasks.

Durability applies to orchestration state, not automatically to every system the workflow calls. If a workflow asks a payment provider to charge a card, the engine can remember that it scheduled the activity and record its result. It cannot make an external provider’s database part of its own transaction. The boundary between orchestration and side effects is therefore central to designing safe recovery.

The Problem: Process Memory Is Not Durable State

In a simple application, a function can call service A, keep the response in a local variable, call service B, and return. If the process exits between those calls, the variable disappears. Developers can persist a custom status row, but then they must build and operate the machinery for leases, retries, delayed jobs, status transitions, manual repair, and safe restarts.

Queue-based retry alone does not solve the whole problem. A queue can redeliver a message when a consumer fails, but an application still needs to know which workflow step completed, what result it produced, and whether a side effect happened before the acknowledgment was lost. The same ambiguity occurs when an HTTP timeout hides whether the server completed a request.

Without durable coordination, teams often add ad hoc status fields and scheduled repair jobs. These can be appropriate for simple tasks, but each custom mechanism introduces another state machine to test. The distributed-systems resilience guide covers the broader failure modes: partial failures are normal, and retries need timeouts, backoff, and idempotency rather than a blanket repeat.

How Durable Workflow Execution Works

A typical execution has a client, a workflow service, a persistent history, and one or more workers:

Client starts workflow
        |
        v
Workflow service records start and schedules work
        |
        v
Worker runs workflow code and requests an activity
        |
        v
Activity worker calls an external system
        |
        v
Service records the result and schedules the next step

The service records events such as workflow start, activity scheduling, activity completion, timers, and signals. This event history is the durable record used to recover an execution. When a worker is lost, another worker can replay the workflow code against the recorded history. Completed activity results are supplied from history; the engine does not simply repeat every completed step.

For replay to reconstruct the same decisions, workflow code must be deterministic: given the same prior events, it must make the same choices and emit compatible commands. Calls to external services, random values, and direct wall-clock reads belong in activities or must use the workflow SDK’s deterministic APIs. Otherwise, replay could take a different branch from the one that produced the stored history.

Feature	Hand-written retry loop	Durable workflow engine	Managed state-machine service
Progress storage	Usually local or custom application state	Workflow history and execution state are stored by the orchestration service	State is stored and advanced by the provider’s workflow service
Worker restart recovery	Must be designed and tested by the application	A worker can replay recorded history and continue pending work	The provider resumes the state machine and invokes configured tasks
Workflow definition	Ordinary application code plus custom persistence	Often code-based workflow definitions with separately implemented activities	Often a declarative state machine, service integrations, or provider-specific code
Waiting and timers	Requires queues, schedulers, and persisted wake-up state	Timers and external signals are part of the workflow model	Wait states, callbacks, or service events are provider features
External side effects	Retry behavior is application-defined	Activities can retry, but application-side idempotency is still required	Task retry policies are provided, but downstream effects still need safe handling
Operational ownership	Application team owns the state machine and recovery logic	Team operates or configures the workflow service and workers	Provider manages more of the orchestration infrastructure

These are models, not guarantees shared by every product. For example, AWS Step Functions is a managed state-machine service, while Temporal workers execute workflow code against a separate service. The Step Functions guide explains that service’s workflow types and AWS integrations.

Components and Key Concepts

Workflow definition and execution. A definition describes the process; an execution is one run with its own identity and history. A stable workflow ID lets an application refer to the same logical process, while a run ID can distinguish an individual execution attempt or continuation.

Workflow service and history. The service persists events and coordinates task dispatch. History is more than a log for debugging: it is the input used to rebuild a workflow’s state. Because histories grow, engines provide limits and patterns such as continuing a workflow as a new execution with a fresh history.

Workers and task queues. Workers poll for tasks and run registered workflow and activity code. A task queue routes work to compatible workers; it is not usually the durable record itself. Workers can be scaled or restarted independently, provided the service still has the execution history and the workflow code can replay compatibly.

Activities. Activities perform operations that may be nondeterministic or have side effects: querying a database, charging a payment method, sending an email, or calling an external API. An activity can time out or fail transiently and be retried. Its implementation should use a stable business identifier as an idempotency key where the downstream system supports one.

Timers, signals, and cancellation. A timer lets an execution wait without holding a worker thread. A signal or equivalent message lets an external event update a running workflow, such as a human approval. Cancellation communicates that work is no longer wanted, but activities and downstream services may need explicit cancellation or compensation logic.

Retries and compensation. Retrying is suitable for transient failures; it is not a substitute for correcting invalid input or handling permanent rejection. Some operations cannot be rolled back atomically across services. A workflow may need a compensating action, such as releasing a reservation after a later payment step fails.

Real-World Use Cases

Orders and payments: Reserve inventory, authorize payment, wait for fulfillment, and release the reservation if a later step fails. Idempotency keys help prevent a retry from charging twice.
Provisioning and deployments: Create cloud resources, wait for readiness, run checks, and pause for an approval before exposing a release. A saved execution can resume after an operator action or worker restart.
Account onboarding: Coordinate identity checks, document review, account creation, and notifications, including waits that may last longer than a web request.
Data and model pipelines: Track dependent tasks, preserve results, and retry an individual failed stage instead of rerunning a complete pipeline.
Long-running AI tasks: Track tool calls and approval points across multiple services. The workflow should persist control state and references to data, not treat model output as a trusted or automatically repeatable side effect.

The common requirement is a process whose state and timing matter beyond the lifetime of one server request. For a short, stateless operation, a workflow engine may add more operational complexity than value.

Getting Started: Model Work as a Workflow

Start by separating deterministic orchestration from activities that interact with the outside world. The following TypeScript excerpt uses the Temporal SDK pattern: the workflow coordinates two activities, while activity implementations are registered with a worker. The actual payment and inventory functions should call the application’s services and make irreversible operations idempotent.

Install the SDK packages in a TypeScript project and start a local Temporal development server using the Temporal CLI:

npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity
temporal server start-dev

In workflows.ts, keep external calls behind activity proxies:

import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';

const { reserveInventory, chargePayment } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
  retry: {
    initialInterval: '1 second',
    maximumAttempts: 5,
  },
});

export async function checkout(orderId: string): Promise<void> {
  await reserveInventory(orderId);
  await chargePayment(orderId);
}

The worker must register this workflow and the corresponding activity functions on a task queue, and a client must start checkout with a unique workflow ID. Keep the workflow function deterministic; put network requests, database writes, and other side effects in activities. Test recovery by stopping a worker during a workflow, starting a replacement worker with the same workflow code, and checking that the execution continues. The local development server UI and temporal workflow list can help inspect running executions.

Before production, define activity timeouts, retry limits, workflow cancellation behavior, and alerting for stuck executions. Check whether an operation can be retried safely and what happens when a caller times out after the external system has already completed it. RFC 9110’s definition of idempotent HTTP methods explains why a repeated request should have the same intended effect, but application-level payment or provisioning APIs still need their own idempotency contract.

Common Misconceptions

“Durable means every action happens exactly once.” A workflow service can persist orchestration history, but an activity may reach an external service and then lose its response. Retrying can repeat the request. Use idempotency keys, deduplication, or reconciliation for side effects; do not infer exactly-once behavior from a durable workflow.

“The workflow worker must stay alive.” A worker executes tasks but does not have to hold a workflow’s entire lifetime in memory. The service stores progress and can dispatch tasks to another compatible worker. The service and its persistence still need their own availability, backup, capacity, and upgrade plans.

“Replay means calling every API again.” Replay reconstructs workflow decisions from the recorded history. Completed activity results come from that history, while pending activities may still execute or retry. Workflow code changes must preserve replay compatibility for executions that are still open.

Changelog

Initial publication.