You're Already Building a Workflow Engine

The Pattern You’ve Built Ten Times

Open your codebase. Search for a column called status or workflow_state or processing_status. Found one? Good. You’ve built a workflow engine.

Here’s how it usually goes. You need an access request approval, or a multi-step onboarding flow, or an order that moves through reserve, charge, ship. So you add a state column. Then you write API handlers that advance the state on each event. Then you add a background job for timeouts, because what if the approver never responds? Then you add a reconciliation cronjob, because things fall out of sync and you need something to catch the gaps.

The actual business logic — “wait for A, then B, then provision” — is maybe ten lines of code. The state enum, the transition handlers, the timeout job, the reconciliation logic, the error recovery paths? That’s 200+ lines. And every new workflow needs the same boilerplate from scratch.

If you read my previous post on exactly-once semantics, you know that every platform guarantee stops at the system boundary. What happens when your business logic spans five boundaries and takes three days? You build infrastructure. And it’s the same infrastructure every time.

There’s a pattern that replaces most of it. Four building blocks, same idea across every implementation. It’s called durable execution, and the part that makes it practical — the part that lets your code sleep for a week and wake up when a human clicks “approve” — is durable messaging.

What You Probably Believe vs. What’s True

Before we get into the mechanics, let’s surface some beliefs that steer teams toward the wrong solution.

Checkpointing and Replay: The Core Mechanism

The fundamental problem with crash recovery is amnesia. The process crashes, restarts, and has no idea what already happened. Did the payment go through? Was the inventory reserved? The process doesn’t know, so it either skips everything (leaving the system in a broken state) or re-runs everything (creating duplicates).

Checkpointing solves the amnesia. After each step completes, save its result to a durable store: a database, a journal, an event log. When the process crashes and restarts, it can see exactly which steps finished and what they returned.

But checkpointing alone doesn’t tell you how to resume. That’s where replay comes in. On restart, the runtime re-runs your function from the beginning. But instead of executing completed steps, it returns their saved results and skips ahead. Step 1: checkpoint exists, return saved result. Step 2: same. Step 3: same. Step 4: no checkpoint found. This is where the original execution failed. Execute step 4 for real, save the result, continue.

Your code stays a linear function. No state machine. No switch/case. The runtime handles recovery transparently.

If this sounds familiar, it should. It’s the same idea as a database write-ahead log (WAL), applied at the application layer instead of the storage layer. Postgres writes every change to the WAL before applying it, so it can recover after a crash. Durable execution writes every step’s result to a checkpoint store, so your workflow can recover after a crash.

sequenceDiagram
  participant W as Workflow
  participant DB as Checkpoint Store
  participant API as External APIs
  W->>API: Step 1: Reserve inventory
  API-->>W: OK
  W->>DB: Save result
  W->>API: Step 2: Charge payment
  API-->>W: OK
  W->>DB: Save result
  Note over W: CRASH
  Note over W,DB: Process restarts
  W->>DB: Step 1? Found → skip
  W->>DB: Step 2? Found → skip
  W->>API: Step 3: Notify warehouse
  API-->>W: OK
  W->>DB: Save result
  Note over W: Done. No duplicate charges.

Determinism: The Constraint You Can’t Skip

Replay only works if the function produces the same sequence of steps every time it runs. If the function takes a different code path on replay than it took originally, the checkpoints won’t line up, and the runtime throws a non-determinism error.

This creates a rule: anything that could produce a different value between runs must be wrapped in a checkpointed step. The most common offenders:

time.Now(): the original execution runs at 4:58 PM and takes the “same-day shipping” path. The process crashes. Replay starts at 5:03 PM and takes the “next-day” path. Same checkpoint position, different branch. The runtime panics.

uuid.New(): each call generates a different UUID. If the workflow uses a UUID to create a record, replay generates a different UUID and looks for a checkpoint that doesn’t exist.

rand.Intn(), os.Getenv(), external config reads: anything where the return value could differ between the original execution and a replay.

The fix is consistent across every framework: wrap the non-deterministic call in a checkpointed step. On the first execution, the runtime saves the value. On replay, it returns the saved value instead of calling the function again.

This sounds minor until you hit it in production. Determinism violations are invisible in staging, because staging environments rarely have the worker restarts and long execution times that trigger replay. The bug only surfaces during crash recovery. Exactly when you need the system to work.

Durable Messaging: Where It Gets Interesting

Checkpointing, replay, and determinism give you crash-proof sequential execution. But real workflows aren’t purely sequential. They need to wait for things. A payment webhook. A human approval. A response from another service. Sometimes for days.

Durable messaging is the mechanism that makes this possible. If you know Go channels, you already know the mental model.

A Go channel has <-ch (receive, blocks until a value arrives) and ch <- (send, delivers a value). Durable messaging has Recv and Send. Same shape. Same blocking semantics. One critical difference: a Go channel lives in memory. Process dies, the channel is gone. Durable messaging persists the suspension point to a database. The server can restart fifty times over seven days. When the human finally clicks “approve,” the Send arrives, and the workflow picks up right where it left off.

No thread held. No connection open. No compute consumed. The entire workflow state is a row in a database. When the message arrives, the runtime reconstructs the workflow, replays it to the suspension point (skipping all completed steps), and continues execution.

This is what separates durable execution from “just using a queue.” A queue delivers a message to a consumer. Durable messaging rehydrates the entire workflow context and resumes execution from exactly where it paused.

Pat Helland puts it well in Idempotence Is Not a Medical Condition: for long-running work that arrives over days or weeks, applications need durable state that captures the essence of what happened in earlier steps. That’s what durable messaging provides. The workflow carries its history forward, so when the next message arrives, the function has full context without re-querying or re-computing anything.

sequenceDiagram
  participant W as Workflow
  participant DB as Checkpoint Store
  participant H as HTTP Handler
  participant U as Human
  W->>DB: Save suspension point
  Note over W: Suspended (zero resources)
  Note over W: Server restarts 3 times
  Note over W: 4 days pass
  U->>H: POST /approve
  H->>DB: Send("approved")
  DB-->>W: Wake up
  W->>DB: Replay: skip steps 1-3
  W->>W: Continue from step 4
  Note over W: Resumed with full context

If you’ve followed this far, you have the four building blocks: checkpointing, replay, determinism, and durable messaging. The rest of this post is about applying them in practice: what the code looks like, where they break, and when to reach for something else.

The Architecture Spectrum

These four building blocks appear in every durable execution framework. The difference is in deployment: how much infrastructure sits between your code and the checkpoint store.

At one end, DBOS is a library inside your app that checkpoints to your existing Postgres. No new infrastructure. At the other end, Temporal runs a cluster of four services with its own database. Restate sits in between: a single Rust binary that handles journaling and replay.

The trade-off: less infrastructure means faster adoption. More infrastructure buys you operational features (multi-tenancy, advanced versioning, distributed workers across regions). Pick based on your scale and operational maturity, not on which building blocks you need. The building blocks are the same everywhere.

graph LR
  subgraph cluster ["Cluster · Temporal"]
      A1["Your app"] --> W["Workers"] --> T["Server
(4 services)"] --> D1[("DB")]
  end
  subgraph lightweight ["Single Binary · Restate"]
      A2["App + SDK"] --> R["Restate
binary"] --> D2[("Journal")]
  end
  subgraph embedded ["Library · DBOS"]
      A3["App +
DBOS lib"] --> D3[("Postgres")]
  end

All Four Building Blocks in One Workflow

Here’s a checkout workflow that uses all four building blocks. It reserves inventory (checkpointed), creates an order (checkpointed), waits for a payment webhook (durable messaging), and starts delivery dispatch (child workflow). If the server crashes at any point, it replays from the last checkpoint and continues.

Using DBOS in Go, because it has the least ceremony. The same pattern works in Temporal, Restate, and others.

Checkout workflow with all four building blocks (DBOS, Go)

func checkoutWorkflow(ctx dbos.DBOSContext, _ string) (string, error) {
    orderID, _ := dbos.RunAsStep(ctx, func(c context.Context) (int, error) {
        return createOrder(c)
    })
    _, _ = dbos.RunAsStep(ctx, func(c context.Context) (bool, error) {
        return reserveInventory(c, orderID)
    })
    // Durable messaging: workflow suspends here, zero resources
    payment, _ := dbos.Recv[string](ctx, "payment-status", 60*time.Second)
    if payment == "paid" {
        dbos.RunAsStep(ctx, func(c context.Context) (string, error) {
            return updateOrderStatus(c, orderID, "PAID")
        })
        dbos.RunWorkflow(ctx, dispatchWorkflow, orderID)
    }
    return fmt.Sprintf("order-%d", orderID), nil
}

The Recv call is the durable messaging primitive. The workflow suspends there with zero resource consumption and survives any number of restarts until the payment webhook arrives.

func CheckoutWorkflow(ctx workflow.Context, orderID string) error {
    var reserved bool
    err := workflow.ExecuteActivity(ctx, ReserveInventory, orderID).Get(ctx, &reserved)
    if err != nil { return err }

    err = workflow.ExecuteActivity(ctx, CreateOrder, orderID).Get(ctx, nil)
    if err != nil { return err }

    // Signal channel: durable messaging in Temporal
    var paymentStatus string
    ch := workflow.GetSignalChannel(ctx, "payment-status")
    ch.Receive(ctx, &paymentStatus)

    if paymentStatus == "paid" {
        return workflow.ExecuteChildWorkflow(ctx, DispatchWorkflow, orderID).Get(ctx, nil)
    }
    return fmt.Errorf("payment failed: %s", paymentStatus)
}

Three Ways Durable Execution Breaks in Production

Durable execution is not magic. Here are three production failure modes that catch teams who assume the runtime handles everything.

1. The invisible determinism violation. A fintech team deploys a payment workflow. It works for weeks. Then a worker pod gets rescheduled during a Kubernetes cluster upgrade. The workflow replays. A time.Now() call, not inside a checkpointed step, returns a different value. The workflow takes a different code path. The runtime throws a non-determinism error. The payment is stuck in a half-processed state at 2am on a Saturday.

This is the most common production failure in durable execution. It works perfectly in staging because staging workers rarely restart while workflows are in-flight. The bug only manifests under crash recovery, the one condition you can’t easily simulate.

2. The workflow versioning nightmare. You ship a new version of your approval workflow that adds a step between steps 2 and 3. There are 400 in-flight approval workflows running the old version. On the next worker restart, the runtime tries to replay those workflows with the new code. Step 3’s checkpoint doesn’t match the new step 3. Every in-flight workflow breaks simultaneously.

Temporal solves this with Worker Versioning (pinning workflows to the code version that started them) and workflow.GetVersion() for branching logic. Restate allows compensations to roll forward. DBOS sidesteps some of this by keeping workflow state in regular Postgres rows. But every framework treats this as the hardest operational problem in durable execution.

3. The idempotency boundary you forgot. Your workflow step calls a payment gateway. The gateway processes the charge and returns success. The response times out before reaching your worker. The checkpoint never gets written. On recovery, the step re-executes. The customer is charged twice. Sound familiar? It’s the same partial completion failure from my post on exactly-once semantics. Durable execution doesn’t solve it. The step still has at-least-once semantics for external calls. You still need an idempotency key on the payment request.

Complexity Migrates, It Doesn’t Disappear

There’s a principle sometimes attributed to Larry Tesler: complexity doesn’t vanish, it just moves to another part of the system. Durable execution is a textbook case.

Look at what changes when you adopt it. On the left, the problems you had before. On the right, the problems you have after.

The total complexity didn’t shrink. But look at the difference in character. The “before” column is scattered and implicit: a retry loop absent from one call site, a dead-letter queue nobody checks, a crash recovery path nobody tested. The “after” column is visible. These constraints show up in code review and deployment checklists.

The trade isn’t less complexity. It’s better-located complexity. You moved it from the gaps between your services, where bugs hide for months, to the surface of your code, where you can see it and reason about it.

When to Reach for What

Durable execution isn’t always the right answer. Neither is a queue. Neither is a state machine. The skill is matching the pattern to the problem.

Here’s the gut check: describe the work as a single verb. “Send.” “Process.” “Resize.” If a single verb covers it, it’s a job. Use a queue. If you need “and then” or “wait until” to describe it, it’s a workflow, and that’s when durable execution earns its keep.

graph LR
  A{"Describe the work
in one verb?"} -->|Yes| B["Job queue
(BullMQ, Celery)"]
  A -->|"Needs 'and then'"| C{"Crosses system
boundaries?"}
  C -->|No| D["DB transaction
or savepoints"]
  C -->|Yes| E{"Long waits?
Human approval?"}
  E -->|No| F["Saga with
outbox pattern"]
  E -->|Yes| G["Durable
execution"]

In practice, these patterns compose. A food delivery order might use the outbox pattern for atomic event publishing, a durable execution workflow for the reserve-charge-dispatch saga, Kafka choreography for independent reactions (notifications, analytics), and a plain job queue for generating the receipt PDF. Four patterns, one order. The skill is knowing where one ends and the next begins.

The Cheat Sheet

• Durable execution is four building blocks: checkpointing (save each step’s result), replay (skip completed steps on restart), determinism (same inputs produce the same step sequence), and durable messaging (sleep and wake across crashes).
• Durable messaging is what separates durable execution from “just retries.” It lets a workflow suspend indefinitely, consume zero resources, and resume with full context when an external event arrives.
• The complexity doesn’t disappear. You trade retry loops and state machines for determinism discipline, workflow versioning, and idempotency boundaries. The total is comparable. The location is better.
• Not everything needs it. Single-step jobs belong in a queue. Database-local work belongs in a transaction. Reach for durable execution when your workflow spans system boundaries and involves waits you can’t control.
• Patterns compose. Real systems use outbox + durable execution + choreography + job queues at different layers. The skill is knowing where one ends and another begins.

Further Reading

Start here:

• The Definitive Guide to Durable Execution by Temporal. Clear, framework-agnostic explanation of the concept and why it matters. Best starting point if this post was your introduction.
• Building a Modern Durable Execution Engine from First Principles by Restate. How Restate built their engine from scratch. Excellent technical depth on the journaling and replay mechanics.

Go deeper:

• Life Beyond Distributed Transactions by Pat Helland (ACM Queue, 2016). The paper that explains why entities, not distributed transactions, are the right atomicity boundary. Predicted the need for durable workflow state years before the current tooling existed.
• Idempotence Is Not a Medical Condition by Pat Helland (ACM Queue, 2012). Foundational paper on why durable messaging requires durable state. Dense but worth it.
• Sagas by Garcia-Molina and Salem (ACM SIGMOD, 1987). The original paper that defined the saga pattern. Still the clearest explanation of compensating transactions.
• Solving Durable Execution’s Immutability Problem by Restate. The best treatment of workflow versioning, the hardest operational problem in durable execution adoption.
• Your Queue Is Not a Workflow Engine by DebuggAI. Practical comparison of queues vs. durable execution with clear decision criteria.