Failure pattern · 2 documented incidents

Partial success leaves systems out of sync

A multi-step workflow half-completes, or events arrive out of order, and two systems now disagree about what happened.

See it happen

Watch how this failure unfolds, then switch to the same situation with the safeguard in place. Click any step to replay it.

Scroll the diagram sideways to see every system.

12 / 12 s

The customer has paid, but no order exists. The systems disagree.

What goes wrong:

  1. Checkout charges the customer’s card.
  2. Reserving the item fails halfway through.
  3. The workflow stops, and nothing undoes the charge.
  4. The customer has paid, but no order exists. The systems disagree.

With the safeguard:

  1. Checkout charges the customer’s card.
  2. Reserving the item fails halfway through.
  3. Every step has an undo: the workflow refunds the charge automatically.
  4. A reconciler compares the systems on a schedule and flags any gap.

What it is

A process with several steps gets halfway: the card is charged but the item isn’t reserved, or one database accepted a write the other never saw. Now two systems disagree about what happened, and nobody notices until a customer does.

How it happens

  • One business action spans several services or databases
  • A later step fails and nothing undoes the earlier ones
  • Events can arrive late, twice or out of order

What it causes

  • Customers charged for orders that don’t exist
  • Records that contradict each other between systems
  • Manual cleanup that takes days

Warning signs

  • Mismatches between your records and a partner’s
  • Workflows stuck in an in-between state
  • Support tickets about paying without receiving

How to prevent it

Design multi-step workflows to converge. Use an outbox or saga with compensating steps, order and version events so stale ones are ignored, and reconcile against the system of record on a schedule.

  • Every step in a multi-step workflow has a matching undo that runs if a later step fails
  • Events carry versions or timestamps, so stale ones are ignored
  • A scheduled job reconciles the systems and flags any disagreement

Key ideas:Reconciliation

Check your system against the full launch checklist

Real incidents with this pattern

Documented incidents first. Each links to its public source.

Open in the database
9 security advisories and illustrative scenarios with this pattern
Illustrative scenarioDate not recorded

Kafka Consumer Group Rebalance Storm Message Lag

Slow consumer heartbeat during heavy batch processing triggered infinite rebalance loop, halting all event consumption.

Retry amplification and runaway resource consumption

In the standards

CollectSEA's mapping; not an official determination by the standards bodies.