All insights

Designing for duplicate events

A practical guide to deciding what should happen only once, keeping retries safe, and testing what happens when processing stops midstream.

Define what should happen once

A distinction in Palash's work on purchase events is useful here: a duplicate delivery is not always the same thing as a duplicate business action. In a hypothetical service that grants a first-time customer credit, two events with different message IDs can still describe the same credit. An event ID can catch redelivery. A business key such as (customer, credit program) protects the credit itself. The right key follows the rule for when a new credit is legitimate.

Put the check where the state changes

When the effect stays within the database, a unique constraint can claim the business key in the same transaction that applies the change. A separate read-before-write can race with another consumer. Committing the claim on its own can leave the change permanently skipped after a crash. The consumer acknowledges only after commit, so if it crashes in between, a retry can see the committed claim without applying the change again.

Treat external effects differently

A database cannot roll back a remote call. Record the intent alongside the local update, then dispatch it separately with a stable idempotency key if the receiver supports one. If the call times out and the receiver cannot confirm what happened, reconcile the outcome before retrying. Keep deduplication records for as long as replays are supported.

Test the failure windows

  • Deliver the same message concurrently and check that only one business effect is recorded.
  • Send distinct messages for the same business key, then send another with a different key to check that legitimate work still happens.
  • Interrupt processing before commit, after commit but before acknowledgement, and after the remote system accepts a request but before its response arrives.

For a separate architectural example, see Architecture.