Five Ways to Make a Message Survive

The Message That Arrived But Nobody Remembered Why

Your team ships an order processing service. It receives messages from a Kafka topic. Each message triggers a five-step workflow: validate the order, reserve inventory, charge payment, send confirmation, dispatch fulfillment.

The message is durable. Kafka persists it to disk, replicates it across brokers. If a broker dies, the message survives. Your SRE team sleeps soundly. The message will always arrive.

Then the service crashes after step 3 (charge payment) but before step 4 (send confirmation). Kafka redelivers the message. Your consumer picks it up. But the consumer has no idea that steps 1-3 already ran. It starts from scratch. The customer gets charged twice.

The message survived. The context didn’t.

This is the gap that “durable messaging” is supposed to fill. But that phrase means five architecturally different things depending on who’s saying it. A Kafka engineer, a Temporal user, an Orleans developer, and someone running Debezium with a transactional outbox all mean something different when they say “our messaging is durable.”

In my previous post on durable execution, I introduced durable messaging as the fourth building block — the mechanism that lets a workflow suspend and resume. But that was one approach. This post maps all five, shows how each one works under the hood, and surfaces the second-order effects that only become visible six months into production.

The Spectrum: Durable Delivery vs. Durable Context

The fundamental confusion around “durable messaging” exists because people conflate two different problems:

1. Durable delivery: the message itself survives failures. Kafka persists to disk. RabbitMQ with persistent queues writes to Mnesia. SQS stores across availability zones. The message won’t be lost.

2. Durable context: the processing state — what happened before this message, what step we’re on, what decisions were already made — survives failures. When processing resumes, the full context is reconstructed.

Every approach to “durable messaging” sits somewhere on this spectrum. On the left: the message is just bytes that survive a crash. On the right: the entire workflow history is preserved and reconstructed.

Most production failures happen because teams pick a solution from one end of the spectrum when their problem lives at the other end.

graph LR
  subgraph left ["Message Survives"]
    A["Persistent<br/>Broker"] --> B["Transactional<br/>Outbox + CDC"]
  end
  subgraph middle ["Message + Some State"]
    C["Event Log<br/>(Kafka Streams)"]
  end
  subgraph right ["Full Context Survives"]
    D["Virtual<br/>Actors"] --> E["Durable<br/>Execution"]
  end
  left -.->|"More infrastructure,<br/>less app logic"| right
  right -.->|"More app logic,<br/>less infrastructure"| left

What You Probably Believe vs. What’s True

Before mapping the five approaches, let’s surface the beliefs that steer teams into the wrong architectural bucket.

The Five Approaches

The rest of this post maps five architecturally distinct approaches to durable messaging. Each solves a different subset of the problem. Each has second-order effects that only surface in production.

They’re ordered from “message survives” (left of the spectrum) to “full context survives” (right of the spectrum). Most real systems use 2-3 of these at different layers.

graph TD
  A["1. Persistent Broker<br/>(Kafka, Pulsar, SQS)"] --> B["Message bytes on disk"]
  C["2. Transactional Outbox<br/>(+ CDC / polling)"] --> D["Atomic state + message"]
  E["3. Event Log as State<br/>(Kafka Streams, EventStoreDB)"] --> F["Replayable history"]
  G["4. Virtual Actors<br/>(Orleans, Durable Objects)"] --> H["Per-entity durable state<br/>+ sequential messaging"]
  I["5. Durable Execution<br/>(Temporal, Restate, Inngest)"] --> J["Full workflow context<br/>reconstruction"]

Approach 1: The Persistent Broker

The simplest form of durable messaging: write the message to disk before acknowledging it. If the broker crashes, the message is still there when it comes back.

Kafka writes to a partitioned, append-only log replicated across brokers. Pulsar separates brokers from storage (BookKeeper ledgers). SQS distributes messages across multiple availability zones. RabbitMQ persists to Mnesia (with quorum queues for replication since 3.8).

What you get: the message will be delivered at least once to a consumer. With Kafka’s idempotent producer (sequence numbers per partition) and transactions (KIP-98), you can get exactly-once semantics for consume-transform-produce pipelines that stay within Kafka.

What you don’t get: any awareness of what the consumer did with previous messages. The broker is a postal service. It guarantees the letter arrives. It has no idea what the recipient does after opening it.

The mechanism: Kafka’s exactly-once uses a producer ID + sequence number for deduplication within a session, plus a transaction coordinator that atomically commits offsets and output messages together. The consumer reads only committed messages (isolation level: read_committed). This is Atomic Broadcast within Kafka’s topology.

Where it stops: the moment your consumer writes to an external database or calls a third-party API, you’re outside the transaction boundary. The Kafka transaction committed, but did the Postgres write succeed? Did the Stripe charge go through? You’re back to application-level idempotency — Level 2 from my exactly-once post.

sequenceDiagram
  participant P as Producer
  participant K as Kafka (3 brokers)
  participant C as Consumer
  participant DB as External DB
  P->>K: Produce (idempotent, seq #42)
  K-->>P: ACK (replicated)
  Note over K: Message durable on disk
  K->>C: Deliver (read_committed)
  C->>DB: INSERT order
  Note over C,DB: This write is NOT<br/>covered by Kafka's<br/>transaction
  C->>K: Commit offset
  Note over C: If crash between DB write<br/>and offset commit: duplicate

Approach 2: The Transactional Outbox

The persistent broker solves message durability but creates a new problem: the dual-write. Your service needs to update its database AND publish a message. These are two independent systems. If one succeeds and the other fails, your system is inconsistent.

The transactional outbox pattern solves this by eliminating one of the two writes from the application’s critical path. Instead of writing to the database and publishing to Kafka, you write to the database and an outbox table in the same transaction. A separate relay process reads the outbox and publishes to Kafka asynchronously.

Three delivery mechanisms for the relay:

1. Polling publisher: periodically queries the outbox table for unsent messages. Simple but introduces latency and database load.

2. CDC (Change Data Capture): Debezium tails the database’s transaction log (WAL for Postgres, binlog for MySQL). Zero application code changes. But the CDC pipeline becomes a critical dependency.

3. LISTEN/NOTIFY (Postgres): The database itself notifies the relay when new outbox rows appear. Lower latency than polling, no external CDC infrastructure. But NOTIFY is lossy under connection drops.

What you get: atomic consistency between your state change and your message intent. If the transaction commits, the message will eventually be published. If it rolls back, no message exists.

What you don’t get: immediate delivery, ordering guarantees across entities, or consumer-side state management. The outbox guarantees the message will be published. It says nothing about what the consumer does when it receives it.

sequenceDiagram
  participant S as Service
  participant DB as Postgres
  participant R as Relay Process
  participant K as Kafka
  participant C as Consumer
  S->>DB: BEGIN TRANSACTION
  S->>DB: UPDATE orders SET status='paid'
  S->>DB: INSERT INTO outbox (event, payload)
  S->>DB: COMMIT
  Note over DB: Both writes atomic
  R->>DB: SELECT * FROM outbox WHERE sent=false
  DB-->>R: [OrderPaid event]
  R->>K: Publish OrderPaid
  K-->>R: ACK
  R->>DB: UPDATE outbox SET sent=true
  K->>C: Deliver OrderPaid
  Note over C: Consumer still needs<br/>idempotency

Approach 3: The Event Log as State

What if the message log IS the state? Instead of storing current state in a database and separately publishing events, you store all state changes as an immutable sequence of events. The current state is derived by replaying the log.

This is event sourcing. And when you combine it with a streaming processor (Kafka Streams, Flink, EventStoreDB projections), the event log becomes both your messaging system and your state store.

The mechanism: Every state change is an event appended to a log. To know the current state of an order, replay all OrderCreated, ItemAdded, PaymentReceived, Shipped events for that order ID. The event log is the source of truth. Read models (projections) are materialized views derived from the log.

What you get: A complete audit trail. Time-travel debugging (replay to any point). Natural event-driven architecture (the events exist by default, not as an afterthought). No dual-write problem (the event IS the state change, not a side effect of it).

What you don’t get: Simple reads. Current state requires projection. Projections introduce eventual consistency between the write side and read side (CQRS). Schema evolution across events is non-trivial. And the event log grows without bound unless you implement snapshotting.

The key insight: In event sourcing, “durable messaging” isn’t a separate concern. Publishing events to other services is just giving them a read pointer into your log. The message IS the state mutation. There’s no dual-write because there’s only one write.

graph LR
  subgraph write ["Write Side"]
    C["Command"] --> H["Handler"]
    H --> E["Event Store<br/>(append-only log)"]
  end
  subgraph read ["Read Side (projections)"]
    E --> P1["Order Summary<br/>Projection"]
    E --> P2["Analytics<br/>Projection"]
    E --> P3["Search Index<br/>Projection"]
  end
  E --> S["Other Services<br/>(subscribe to events)"]

Approach 4: Virtual Actors

Virtual actors (Orleans Grains, Cloudflare Durable Objects, Akka Persistent Actors) solve durable messaging through a different abstraction entirely: every entity is an addressable, stateful object that processes messages sequentially.

The mechanism: You send a message to an actor identified by a key (e.g., order-12345). The runtime locates the actor (activating it if it’s dormant), delivers the message, and the actor processes it with access to its persisted state. One message at a time. No concurrent access to the same actor’s state. No locks needed.

What you get: Per-entity durable state without explicit database queries. Sequential message processing without distributed locks. Automatic lifecycle management (actors activate on demand, deactivate when idle). Location transparency (the runtime places actors across the cluster; callers don’t need to know where).

What you don’t get: Cross-entity transactions. If “transfer $100 from Account A to Account B” requires atomicity across two actors, you need to build that coordination yourself (saga pattern, compensation logic). Actors are atomic within themselves, never across each other.

The insight for messaging: Virtual actors turn “send a message to an entity” into a first-class operation. The message IS the invocation. The actor’s state IS the durable context. There’s no separate “message store” and “state store” — they’re unified in the actor’s persisted state. When a message arrives, the actor already has all the context it needs.

This is close to what Pat Helland describes in Life Beyond Distributed Transactions: entities as atomicity boundaries, with messaging between them as the coordination mechanism. Actors are a natural implementation of that paper’s vision.

sequenceDiagram
  participant API as API Gateway
  participant R as Actor Runtime
  participant A as OrderActor-12345
  participant S as Storage
  API->>R: Send(order-12345, "PaymentReceived")
  R->>S: Load state (if not cached)
  S-->>R: {status: "pending", items: [...]}
  R->>A: Activate + deliver message
  Note over A: Process sequentially<br/>(no concurrent access)
  A->>A: Update state: status = "paid"
  A->>S: Persist state
  A->>R: Send(fulfillment-12345, "Dispatch")
  Note over A: State survives crashes,<br/>no explicit DB queries

Approach 5: Durable Execution Primitives

The right end of the spectrum. Here, “durable messaging” means something fundamentally different: sending or receiving a message is a checkpointed operation within a durable workflow. When the message arrives, the entire workflow context is reconstructed from its journal and execution resumes from exactly where it suspended.

The mechanism varies by framework, but the shape is the same:

Temporal: The workflow calls workflow.GetSignalChannel("payment").Receive(). The workflow suspends. The Temporal server records the suspension point in the workflow’s event history. Days later, an HTTP handler calls client.SignalWorkflow(workflowID, "payment", data). The server appends a SignalReceived event, creates a workflow task. A worker picks it up, replays the entire workflow from its event history (skipping completed steps), and continues from the suspension point.

Restate: The handler creates a “Durable Promise” or “Awakeable” and suspends. Restate journals the suspension in its append-only log (Bifrost). An external system resolves the promise by ID. Restate journals the resolution and replays the handler from the journal.

Inngest: The function calls step.waitForEvent("payment/received"). Inngest persists the waiting state. When a matching event arrives, Inngest resumes the function from the waiting step.

Cloudflare Workflows: step.waitForEvent({ name: "payment", timeout: "7d" }). The workflow instance is serialized. When an event is sent via the Workers API, the instance is deserialized and execution continues.

AWS Step Functions: A task uses the “Wait for Callback” integration pattern. Step Functions generates a task token, passes it to an external system. The workflow pauses (billing stops). When the external system calls SendTaskSuccess with the token and payload, the workflow resumes.

What makes this different from the other four approaches: None of the other approaches reconstruct the full processing context. A Kafka consumer gets the message but has no memory of previous messages. An outbox guarantees publication but not consumption context. Event sourcing rebuilds entity state but not workflow state. Virtual actors remember per-entity state but not cross-entity workflow progress.

Durable execution reconstructs EVERYTHING: every completed step, every intermediate variable, every decision branch, every prior message that was received. The workflow function resumes as if it never stopped.

sequenceDiagram
  participant W as Workflow
  participant RT as Runtime (journal)
  participant H as External System
  W->>RT: step.waitForEvent("approval")
  RT->>RT: Journal suspension point
  Note over W: Zero resources consumed
  Note over W: Days pass. Deploys happen.<br/>Servers restart.
  H->>RT: sendEvent(workflowID, "approval", {approved: true})
  RT->>RT: Journal event received
  RT->>W: Reconstruct workflow
  Note over W: Replay: skip steps 1-5
  W->>W: Resume from step 6
  Note over W: Full context available:<br/>all variables, all history

The Comparison Matrix

Each approach solves a different subset of the durable messaging problem. Here’s what each one actually guarantees — and what it explicitly doesn’t.

Second-Order Effects: What Surfaces at Month Six

Every approach has consequences that only become visible after you’ve committed to it and run it in production for months. These aren’t bugs. They’re architectural gravity — the system pulls you toward certain patterns and away from others.

Persistent Broker: You Build a State Machine Anyway

Teams start with Kafka because “we just need messaging.” Six months later: the consumer has a processing_status column. There’s a dead-letter queue for failed messages. There’s a reconciliation cron that re-processes stuck items. There’s a retry mechanism with exponential backoff. Congratulations: you’ve built a workflow engine out of Kafka consumers and database columns. The exact infrastructure durable execution replaces.

Transactional Outbox: The Relay Becomes Your Bottleneck

The outbox pattern trades the dual-write problem for a relay dependency. Under load, the relay falls behind. Your “real-time” events arrive with 3-second lag. Product asks why the notification came late. You add more relay instances, which introduces ordering challenges (two relays processing the same outbox rows). You add partition-based assignment, which is now a distributed system that needs coordination. The complexity migrated; it didn’t disappear.

Event Sourcing: Projection Lag Becomes a Product Problem

The write side accepts a state change. The read projection hasn’t processed it yet. The user sees stale data. They retry. You need idempotency on the write path. The team adds “read-your-writes” consistency by falling back to the event store for recent writes. But the event store wasn’t designed for random reads. Performance degrades. You add a cache. The cache has its own consistency issues. You started with “events are the source of truth” and ended with three layers of read paths.

Virtual Actors: Cross-Entity Coordination Is Your Whole Job

The first few months are great. Each entity is clean, isolated, testable. Then product asks for “transfer money between accounts” (two actors) or “reserve inventory across three warehouses” (three actors). Actors can’t do cross-entity transactions. You build a saga. The saga needs its own durable state. You’ve now built a workflow engine on top of your actor system.

Durable Execution: Workflow Versioning Becomes Your Hardest Problem

Workflows can run for days or weeks. You ship a new version that changes the step sequence. There are 2,000 in-flight workflows on the old version. Every one of them breaks on the next replay. You need versioning discipline: Temporal’s workflow.GetVersion(), Restate’s forward-compatible compensations, or blue-green worker deployments. This operational overhead is ongoing and compounds with every workflow you add.

The Dual-Write Problem: Why Everything Else Exists

Every approach on this list exists because of one fundamental constraint: you cannot atomically write to two independent systems.

You can’t atomically write to Postgres AND publish to Kafka. You can’t atomically call Stripe AND update your database. You can’t atomically send a message AND persist its result. The moment two systems are involved, you have a gap where one can succeed and the other can fail.

This is the Two Generals Problem applied to system coordination. And every “durable messaging” approach is, at its core, a strategy for living with this constraint:

• Persistent broker: Accept the gap. Make the consumer idempotent.
• Transactional outbox: Eliminate one write from the critical path. Publish asynchronously.
• Event sourcing: Make the event log the single system. No second write needed.
• Virtual actors: Make the actor the single system. State and messaging are unified.
• Durable execution: Make the workflow runtime the coordinator. It checkpoints everything to one journal, then executes external calls with at-least-once semantics.

None of them “solve” the dual-write problem in the sense of making it disappear. They each relocate it to a boundary they can manage. The outbox relocates it to the relay. Event sourcing relocates it to projections. Virtual actors relocate it to cross-entity coordination. Durable execution relocates it to external call boundaries (the idempotency key on the Stripe charge).

Pat Helland saw this in 2007. His Life Beyond Distributed Transactions paper argues that the entity (what Orleans calls a grain, what DDD calls an aggregate) is the natural atomicity boundary. Messages between entities are inherently unreliable and must be handled with idempotency and retries. Twenty years later, every framework on this list implements some version of that insight.

graph LR
  A["The Dual-Write<br/>Problem"] --> B["Can't atomically write<br/>to 2 systems"]
  B --> C["Persistent Broker:<br/>Accept it, be idempotent"]
  B --> D["Outbox:<br/>Single-system write,<br/>async relay"]
  B --> E["Event Sourcing:<br/>Single system<br/>(event log IS state)"]
  B --> F["Virtual Actors:<br/>Single system<br/>(actor IS entity)"]
  B --> G["Durable Execution:<br/>Single journal,<br/>at-least-once externals"]

When to Reach for What

These five approaches compose. Most production systems use 2-3 at different layers. The skill is matching each layer’s needs to the right approach.

Start with the question: what needs to survive a failure?

If it’s just the message bytes, use a persistent broker. If it’s the atomicity of “state change + message,” use a transactional outbox. If it’s the complete history of state changes, use event sourcing. If it’s per-entity state across messages, use virtual actors. If it’s the full cross-service workflow context, use durable execution.

A realistic system uses multiple layers. A food delivery platform might use:

• Kafka (persistent broker) for distributing order events to independent services (notifications, analytics, search index)
• Transactional outbox on the order service for atomic order creation + event publishing
• Durable execution (Temporal) for the multi-step fulfillment workflow (reserve restaurant, charge customer, assign driver, track delivery)
• Virtual actors (Cloudflare Durable Objects) for real-time driver location tracking (per-driver stateful entity)

Four approaches, one platform. Each at the layer where it earns its complexity.

graph LR
  A{"What needs<br/>to survive?"} -->|"Message bytes"| B["Persistent Broker<br/>(Kafka, SQS)"]
  A -->|"State + message<br/>atomicity"| C["Transactional Outbox<br/>(+ CDC or polling)"]
  A -->|"Complete change<br/>history"| D["Event Sourcing<br/>(EventStoreDB, Kafka)"]
  A -->|"Per-entity state<br/>across messages"| E["Virtual Actors<br/>(Orleans, Durable Objects)"]
  A -->|"Full workflow<br/>context"| F["Durable Execution<br/>(Temporal, Restate)"]

The Cheat Sheet

• “Durable messaging” means five different things. A persistent broker (message survives), a transactional outbox (publish is atomic with state), an event log as state (history IS the messaging), virtual actors (entity state + messages unified), and durable execution (full workflow context reconstructed). Know which one you’re talking about.

• The spectrum runs from “bytes survive” to “context survives.” Most production failures happen because teams pick a solution from the left (broker durability) when their problem lives on the right (workflow context preservation).

• Every approach relocates the dual-write problem, none eliminate it. The outbox moves it to the relay. Event sourcing moves it to projections. Actors move it to cross-entity coordination. Durable execution moves it to external call boundaries. Your idempotency logic lives wherever the boundary landed.

• Second-order effects matter more than first-order features. The persistent broker grows into a state machine. The outbox relay becomes a bottleneck. Event sourcing creates projection lag. Actors need sagas. Durable execution needs versioning discipline. Choose based on which second-order effects you can live with.

• Real systems compose 2-3 approaches at different layers. Kafka for distribution, outbox for atomic publish, durable execution for complex workflows. The skill is knowing where one layer ends and the next begins.

Further Reading

Start here:

• Life Beyond Distributed Transactions by Pat Helland (ACM Queue, 2016). The paper that predicted all five approaches. Entities as atomicity boundaries, messaging between them as the coordination mechanism. Dense but foundational.

• Understanding the Dual-Write Problem by Confluent. The clearest explanation of why you can’t atomically write to two systems, and why the outbox pattern exists.

Go deeper:

• Idempotence Is Not a Medical Condition by Pat Helland (ACM Queue, 2012). Why durable messaging requires durable state. The intellectual foundation for all five approaches.

• KIP-98: Exactly Once Delivery and Transactional Messaging by Apache Kafka. The design document for Kafka’s transactional protocol. Shows exactly where their guarantee starts and stops.

• Building a Modern Durable Execution Engine from First Principles by Restate. How to build journal-based durable execution from scratch. Excellent for understanding approach 5 at the implementation level.

• Change Data Capture That Doesn’t Break by Reliable Data Engineering. Real production failures in Debezium CDC pipelines. Reading this before committing to CDC-based outbox relay could save you a 3-week silent data loss incident.

• Saga Orchestration vs Choreography by Temporal. When centralized orchestration beats distributed choreography, and when it doesn’t.

• Durable Execution: The Missing Layer in Your AI Agent Stack by Reptile Haus (2026). How AI agent workloads are driving the adoption of approach 5 as the default for any non-trivial async work.