Transactional Outbox: Publish Events Without Losing Database Changes

A service that updates a database and then publishes a message has a dual-write problem. If the process crashes between those actions, state and events disagree. The transactional outbox writes the business change and an event record in one database transaction; a separate publisher later delivers the event.

The practical problem

An order service marks an order paid and must notify fulfillment. The payment transaction also inserts an outbox row containing event ID, aggregate ID, schema version, and payload. A publisher claims rows in short batches, sends them to the broker, and records delivery. It crashes after sending but before recording success, so the broker receives a duplicate. Fulfillment uses the event ID in an inbox table and applies the transition once.

This scenario defines the boundary for transactional outbox pattern: the system of record, the component that enforces the decision, the asset or money at risk, and the condition that ends an exception. If one of those is unknown, discover it before changing production or presenting the number.

A step-by-step workflow

1. Scope the system. Define event semantics and aggregate ordering.
2. Establish the baseline. Insert business state and outbox row atomically.
3. Apply the control. Keep payloads versioned and minimal.
4. Test failure paths. Publish with bounded batches and claim leases.
5. Verify the result. Treat delivery as at least once.
6. Record ownership. Make consumers idempotent with durable deduplication.
7. Close the loop. Separate retryable errors from poison messages.
8. Review the change. Retain enough history for replay and audit.
9. Complete the review. Compact or archive with a tested policy..

Prove the failure paths

Start with a healthy case based on “Define event semantics and aggregate ordering”, then repeat the exercise while deliberately violating the assumption behind “publish with bounded batches and claim leases”. Capture the observable result, the alert that should fire, and the person who decides whether to stop or continue. Next, simulate the most credible failure in this guide: marking delivered before broker acknowledgement can lose events. The test passes only when the team detects the condition, limits its impact, and restores service or the decision model without relying on undocumented knowledge. A final replay should cover one global ordering key limits throughput; this is the check most likely to reveal that a control works in a diagram but not at the actual boundary.

Evidence worth keeping

• Measure oldest unpublished age
• Publish attempts
• Duplicate rate
• Poison-message count
• Consumer lag
• Deduplication hits
• Schema-version usage
• And cleanup backlog. Alert on age rather than only queue length because one stuck early record can matter more than many fresh ones.

Attach each record to the concrete object under review: the relevant asset, release, customer, contract, key, query, or configuration. Preserve timestamps and version identifiers so another reviewer can reconstruct the sequence. The evidence list above should answer three questions: what was true before the change, what changed, and what proves the intended outcome. For this topic, the most useful cross-check is “treat delivery as at least once”. Keep sensitive fields out of screenshots and tickets; retain structured exports or signed records when they provide stronger provenance. Set retention according to incident, audit, privacy, and contractual needs rather than keeping everything indefinitely.

Common mistakes

• Marking delivered before broker acknowledgement can lose events.
• assuming exactly-once transport removes consumer safeguards.
• one global ordering key limits throughput.
• unbounded retries hide poison messages.
• deleting rows immediately removes evidence.
• letting payloads expose internal tables couples every consumer..

A focused operating checklist

• Name the owner of the system and the owner of the decision.
• Verify “insert business state and outbox row atomically” from the actual production or financial source.
• Exercise the failure described by “Marking delivered before broker acknowledgement can lose events”.
• Reconcile the observed result with the evidence list above.
• Put an expiry and rollback condition on every exception.
• Monitor the user, security, reliability, or cash outcome—not only activity.
• Revisit dependencies when the underlying product, contract, or standard changes.

Make this control sustainable

Assign the owner closest to the enforcement point described in “keep payloads versioned and minimal”, plus a backup who can execute the rollback. Review the control after a material release, supplier change, incident, financing event, or policy change that affects its assumptions. An alert is useful only when it names an action: inspect, throttle, revoke, reconcile, fail over, or escalate. Use the evidence measurements above to set thresholds from observed workload or financial behavior. Treat assuming exactly-once transport removes consumer safeguards as a specific review trigger. Close temporary exceptions with an owner, expiry, compensating control, and proof that the normal path was restored.

Primary references and further verification

Document transaction isolation, claim query, broker acknowledgement, and failure boundaries. Test crashes before commit, after commit, after send, and after acknowledgement.

For a current transactional outbox pattern decision, record the publication or effective date of the primary material and the version actually deployed or executed. Recheck vendor, standards-body, regulator, or accounting guidance when the product, protocol, vulnerability, or contract changes. Discovery summaries can point to an issue, but the decision record should link to the authoritative source and the organization’s own reproducible evidence.

Conclusion

The practical outcome is a control the team can explain and test: define event semantics and aggregate ordering, then compact or archive with a tested policy.. Keep the assumptions and exception path visible. That is more useful than a generic checklist because the next reviewer can reproduce the decision, see where it can fail, and reverse it safely.