Exactly-Once Is a Lie (Sort Of)

The Friday Deploy That Charged a Customer Twice

You ship a payment service. It handles charges, refunds, the usual. Staging looked great. You deploy on Friday afternoon because you’re feeling bold.

A network blip causes a timeout between your API and the payment provider. Your retry logic kicks in. The customer gets charged twice. Your Slack fills up with the kind of messages that start with “Hey, so…”

The post-mortem lands on a familiar recommendation: “We need exactly-once delivery.” Your tech lead nods. Someone opens a Jira ticket. And nobody in the room agrees on what “exactly-once” actually means.

Here’s the thing: there’s a famous blog post that says exactly-once delivery is mathematically impossible. There’s also a major open-source project that shipped it as a feature. They can’t both be right. Can they?

They can. They’re talking about different things. That’s what this post is about.

What You Probably Believe vs. What’s Actually True

Before we untangle this, let’s surface the beliefs that get people into trouble. These aren’t dumb beliefs. They’re reasonable intuitions that happen to be wrong in ways that matter at 3am.

The Three Levels Nobody Distinguishes

The reason smart people keep arguing about this is that “exactly-once” means three different things depending on which layer of the stack you’re standing on. Almost every argument about whether it’s possible is two people talking about different layers.

graph LR
  L1["Level 1:
Transport Delivery"] -->|"Impossible"| A1["At-least-once
+ deduplication"]
  L2["Level 2:
Application Processing"] -->|"Achievable"| A2["Idempotency keys,
optimistic concurrency"]
  L3["Level 3:
System Semantics"] -->|"Achievable"| A3["Kafka transactions,
SQS FIFO, Pub/Sub"]
  A3 -->|"Within boundary only"| L2

Level 1: Transport delivery. Can you send a message over a network and guarantee it arrives exactly once? No. The Two Generals Problem isn’t a design complexity, it’s an impossibility result. You send a message, you get an ack, but did the ack get lost? You can’t know without another ack, and that creates an infinite regress. Tyler Treat’s widely-shared post puts it clearly: “Within the context of a distributed system, you cannot have exactly-once message delivery.”

Level 2: Application processing. Can you ensure that a message’s effect is applied only once, even if the message arrives multiple times? Yes. This is what idempotency gives you. SET x = 5 is idempotent (run it 10 times, x is still 5). INCREMENT x is not. The hard part is making non-idempotent operations behave as if they were.

Level 3: System semantics. Can you build a system where the observable outcome is as if each message was processed exactly once? Also yes. This is Atomic Broadcast (or Total Order Broadcast), and it’s equivalent to consensus. Paxos, Raft, Zab all solve it. This is what Kafka means by “exactly-once semantics.” Jay Kreps’ response to the skeptics is worth reading in full: “If you don’t believe consensus is possible, then you also don’t believe Kafka is possible.”

The catch: Level 3 only works within the system boundary. Kafka can guarantee exactly-once semantics for a consume-transform-produce pipeline that stays inside Kafka. The moment you write to an external database or call a third-party API, you’re back to Level 2.

If you’ve followed this far, you have the framework that resolves most “is exactly-once possible?” debates. The answer is: depends which level you’re asking about. The rest of this post is about Level 2, the one you actually control.

Idempotency: The Actual Engineering

Formally: f(f(x)) = f(x). An operation that produces the same result whether you execute it once or a hundred times. Some operations are naturally idempotent. DELETE FROM users WHERE id = 42: run it twice, same outcome. UPDATE accounts SET balance = 500 WHERE id = 1: same. But UPDATE accounts SET balance = balance - 50 WHERE id = 1 is not. Run it twice, and you’ve deducted 100.

The problem is making non-idempotent operations (charges, transfers, inventory decrements) behave as if they were. Three strategies, for different situations.

graph LR
  A{"Naturally
idempotent?"} -->|Yes| B["No extra
work needed"]
  A -->|No| C{"Control
the client?"}
  C -->|Yes| D["Idempotency
keys"]
  C -->|No| E{"Single or
cross-system?"}
  E -->|Single| F["Optimistic
concurrency"]
  E -->|Cross-system| G["Transactional
outbox"]

Strategy 1: Natural idempotency. Rewrite the operation to be inherently idempotent. Instead of “increment counter,” “set counter to 7 if current version is 6.” This is optimistic concurrency control, and it works when you control both sides of the operation.

Strategy 2: Idempotency keys. The client generates a unique key for each logical request and sends it with every retry. The server stores the key and its result. On duplicate, return the cached result. Stripe popularized this pattern: every mutating API call accepts an Idempotency-Key header. Brandur Leach’s implementation guide is the canonical reference.

Strategy 3: Transactional outbox. For operations that span multiple systems (write to DB, publish to queue, call external API), use a transactional outbox pattern. Write the intent to your database in the same transaction as your state change, then process the outbox asynchronously with at-least-once delivery and idempotent consumers.

Idempotency Keys in Practice

The client generates a key (a UUID or deterministic hash of the request parameters). The server checks if it’s seen that key before. If yes, return the cached response. If no, process the request and store the result.

The tricky part isn’t the happy path. It’s handling the race condition when two concurrent retries arrive with the same key before the first one finishes processing.

Minimal idempotency key middleware (Express/Node.js)

async function idempotencyMiddleware(req, res, next) {
  const key = req.headers['idempotency-key'];
  if (!key) return next();

  const existing = await db.query(
    `INSERT INTO idempotency_keys (key, status, created_at)
     VALUES ($1, 'processing', NOW())
     ON CONFLICT (key) DO NOTHING
     RETURNING *`,
    [key]
  );

  if (existing.rowCount === 0) {
    const cached = await db.query(
      'SELECT response_code, response_body FROM idempotency_keys WHERE key = $1',
      [key]
    );
    if (cached.rows[0]?.status === 'completed') {
      return res.status(cached.rows[0].response_code)
        .json(JSON.parse(cached.rows[0].response_body));
    }
    return res.status(409).json({ error: 'Request is still being processed' });
  }

  const originalJson = res.json.bind(res);
  res.json = async (body) => {
    await db.query(
      `UPDATE idempotency_keys
       SET status = 'completed', response_code = $1, response_body = $2
       WHERE key = $3`,
      [res.statusCode, JSON.stringify(body), key]
    );
    return originalJson(body);
  };
  next();
}

The INSERT ... ON CONFLICT DO NOTHING is doing the heavy lifting. It atomically claims the key: if two concurrent requests race, only one wins the insert, and the other gets rowCount === 0. No check-then-insert, no race window.

The Landscape: What the Platforms Actually Guarantee

Every major messaging platform now claims some form of exactly-once. Here’s what each one actually provides, and where the guarantee stops.

graph LR
  subgraph kafka ["Kafka Exactly-Once Scope"]
    P[Producer] -->|"Idempotent write"| T[Topic]
    T --> C[Consumer]
    C -->|"Transactional"| T2[Output Topic]
  end
  C -->|"YOUR responsibility"| DB["External DB"]
  C -->|"YOUR responsibility"| API["Third-party API"]

Apache Kafka provides exactly-once semantics for consume-transform-produce pipelines that stay entirely within Kafka. The mechanism: idempotent producers (dedup via producer ID + sequence number) plus transactions (atomic writes across multiple partitions). The guarantee stops at Kafka’s boundary. If your consumer writes to Postgres, you’re responsible for the idempotency of that write.

AWS SQS FIFO provides exactly-once processing via content-based deduplication or explicit deduplication IDs. The dedup window is 5 minutes. After that, a duplicate can slip through. And the guarantee only covers message delivery to the consumer. What the consumer does with it is your problem.

Google Cloud Pub/Sub added exactly-once delivery in 2022. It guarantees no redelivery after successful acknowledgment, but only for pull subscriptions, only within a single cloud region, and your subscriber must handle the case where an ack itself fails.

Notice the pattern: every platform qualifies its guarantee with a scope boundary. That’s not a weakness, it’s honesty about what’s achievable.

Where This Breaks: Three Real Failures

These aren’t hypotheticals. These are published post-mortems from companies that had idempotency implementations and still got bitten.

Failure 1: The replica lag trap. Michal Drozd documented a case where idempotency key lookups were routed to a read replica that lagged 2+ seconds behind the primary during traffic spikes. The code was textbook-correct: check if key exists, if not, process. But the replica hadn’t received the key yet. Result: one DB record, two credit card charges. Fix: idempotency key reads MUST hit the primary, or use a strongly-consistent read path. This is the failure mode that catches teams who “did everything right.”

Failure 2: The Kafka consumer rebalance. Depop’s engineering team found their successful-processing counts wildly exceeding production counts. The cause: Kafka consumer rebalances (triggered by autoscaling and aggressive restarts) reset consumer offsets to the last committed position, replaying hundreds of messages per rebalance. Even without physical partition reassignment, a rebalance resets the consumer’s position. If your consumer isn’t idempotent, every rebalance creates duplicates.

Failure 3: The partial completion. Your API charges the customer (external side effect), then crashes before recording the idempotency key. On retry, the key isn’t found, so the system charges again. This is Brandur’s core insight: every foreign state mutation needs its own atomic phase. Record the key before the external call, update it with the result after. If the call fails, the key sits in a ‘processing’ state and retries can detect it.

The Cheat Sheet

• “Exactly-once delivery” over an unreliable network is impossible. The Two Generals Problem is an impossibility result, not a design challenge.
• “Exactly-once semantics” within a system boundary (Kafka, SQS FIFO, Pub/Sub) is achievable and well-understood. It’s Atomic Broadcast, equivalent to consensus.
• “Effectively-once processing” at the application layer is what you actually build. At-least-once delivery plus idempotent processing gets you there.
• The moment your request crosses a system boundary (Kafka to Postgres, your API to Stripe), you need application-level idempotency. No platform saves you here.
• Most “is exactly-once possible?” arguments are definitional disputes, not technical ones. Figure out which level you’re discussing before jumping in.

Further Reading

Start here:

• You Cannot Have Exactly-Once Delivery by Tyler Treat. The clearest explanation of why transport-level exactly-once is impossible. Start here.
• Exactly-once Support in Apache Kafka by Jay Kreps. The counterargument: exactly-once semantics is well-defined and achievable. Read right after Treat to see both sides.

Go deeper:

• Idempotence Is Not a Medical Condition by Pat Helland (ACM Queue, 2012). Dense but worth it. His four principles for message-passing systems are the kind of thing you memorize.
• Implementing Stripe-like Idempotency Keys in Postgres by Brandur Leach. The best practical guide out there. Full state machine, reaper, race condition handling.
• Avoiding Double Payments in a Distributed Payments System by Jon Chew (Airbnb Engineering). How Airbnb built a generic idempotency framework during their SOA migration.
• Designing Data-Intensive Applications, Chapter 11 by Martin Kleppmann. The textbook treatment, from idempotent operations through deterministic retry to transactional processing.
• End-to-End Arguments in System Design by Saltzer, Reed, Clark (1984). The paper that explains why reliability guarantees have to live at the application layer. Predicted this whole debate 30 years early.