Enterprise Integration and Consistency

Consider a hypothetical example: an order is placed. The order service commits it, the payment service authorizes the card, the inventory service reserves stock, and the warehouse system — which runs on someone else’s schedule and accepts a file twice a day — is told to ship. Four systems, four databases, and one thing the customer believes happened.

Now suppose the third step fails. There is no shared transaction to roll back: the order is committed and a payment authorization is in place. Authorization reserves payment capacity; it is not capture or settlement. If the order is canceled before capture, the payment action is to void or release that authorization under the provider’s rules. Whatever happens next is something you have to have designed in advance, and this is the subject of the article: what you actually get when you give up distributed transactions, and what you have to build to replace them.

The through-line is a distinction worth stating before the mechanisms. Technical completion means the message was delivered, the call returned 200, the file was picked up. Business completion means the order is actually fulfilled, the payment is actually settled, the two ledgers actually agree. They are different facts, they fail independently, and no amount of messaging infrastructure turns the first into the second.

Why not one transaction

The obvious answer to the scenario above is a transaction that spans all four systems, and the protocol for that exists. Two-phase commit has a coordinator ask every participant to prepare, and commit only when all of them have agreed. It is real, it works, and databases implement it. PostgreSQL’s own documentation is unusually candid about what it costs.

PostgreSQL’s PREPARE TRANSACTION detaches the transaction from its session and durably stores its state for a later commit decision. The feature is intended for an external transaction manager coordinating multiple transactional resources, not for application code to leave transactions prepared without a recovery owner.

The prepared transaction retains its locks. PostgreSQL also warns that long-lived prepared transactions impede vacuum cleanup and, in extreme cases, can force shutdown to prevent transaction-ID wraparound. The cost lasts until the participant learns and applies the final decision.

That is the cost in one sentence: a prepared transaction holds its locks until someone tells it the answer, and the thing that tells it is a coordinator that can itself fail.

The blocked work depends on the lock. PostgreSQL row-level locks allow ordinary MVCC reads of a visible version while conflicting writers or lockers wait. A stronger table-level lock can block additional operations, including plain reads.

A coordinator failure does not normally require a human to invent the outcome. A transaction manager records its decision durably and recovery reapplies it. Narayana assigns this work to a recovery manager using retained transaction and resource information in its object store; the original application need not restart. If the log is lost or a heuristic decision has already occurred, resolution needs investigation. Guessing abort after another participant committed breaks atomicity.

Blocking can remain while recovery waits for the coordinator’s decision, storage, network, or participants. Narayana describes affected resources as potentially inaccessible until recovery finishes. This partial failure needs monitoring and a tested recovery owner; restarting one process is not necessarily the end of the wait.

Two-phase commit is an option when every required resource supports enlistment and the participants share a workable recovery contract. Neither one database vendor nor ownership by one organization is a protocol requirement. External payment APIs and file-based warehouse interfaces often do not offer enlistment; if the interfaces in this example do not, one distributed transaction cannot cover the process. Even with support, hours of prepared-state lock retention may be unacceptable. Evaluate atomicity needs, blocking, failure isolation, and operational cost before choosing it.

Three integration styles, one business process

A business process can combine synchronous calls, messages, and files. Each transport reports progress at a different boundary; the completion contract must say which boundary its response describes.

	What “success” means	How it fails	Where completion is visible
Synchronous call	The response reports the request’s defined outcome; 202 means accepted, not completed	A timeout can leave the outcome unknown	In the response or a durable status resource; interpret the method, status, and body together
Asynchronous message	Broker acceptance confirms the configured transport handover	Processing can fail, repeat, or arrive out of order	A correlated result event, callback, or status query can report processing and business outcomes
File transfer / batch	The file is available at the agreed publication boundary	Wrong cutoff, partial file, skipped or repeated processing	A receipt, processing report, or status interface can distinguish acceptance from completion

Two consequences follow, and the second is the one that ruins weekends.

First, end-to-end latency is the sum along the critical path, not the worst single step — three sequential stages of one, two, and three minutes take six minutes, not three. What a batch window adds on top of that sum is waiting whose size depends on arrival time: work that arrives just before a twice-daily cutoff waits minutes, and work that arrives just after waits most of a half-day. So a batch step usually dominates the worst-case budget and inflates the average, without making the other steps free. A promise of immediate fulfillment in the user interface is a promise about one step of that path.

Without a completion-tracking contract, the order service, broker, and warehouse each know only their own part. A status service can assemble the evidence for this order has been shipped and paid for, provided the identifiers, outcome semantics, and observation coverage are defined. HTTP 202 illustrates the distinction: RFC 9110 treats it as acceptance, and Microsoft’s asynchronous request-reply pattern uses a separate status resource to report later progress.

A note on which style to reach for, because the choice affects the caller’s obligations. Use a synchronous call when the caller cannot proceed without the answer and the answer is fast. Use a message when the work can happen later, provided that delayed completion is acceptable — and note that this buys a delay you must now tolerate, not convergence you have been given. Eventual consistency holds only where propagation is durable, conflicts have a resolution rule, and something detects and repairs what was missed; without those a message-based copy drifts rather than converges. Use a file when the other side only accepts files — which is a real and permanent condition in enterprise estates, not a legacy embarrassment to be designed away.

What a saga actually guarantees

The standard answer to multi-system work is a saga: break the process into local transactions, each committing in one system, and define how to undo the earlier ones if a later one fails. The idea is older than microservices: Garcia-Molina and Salem introduced it in 1987 for long-lived transactions.

In that model, local transactions may interleave with other work. The recovery contract distinguishes a completed sequence from compensation of the completed prefix after a partial execution. Releasing resources between steps makes intermediate effects observable; that is the tradeoff behind the model.

In the paper’s sequential model, backward recovery compensates that prefix in reverse order. That is not a universal order for every business workflow: modern compensating-transaction guidance permits a different order, or parallel compensations, when dependencies and business priorities allow it.

The original model does not automatically notify or abort other transactions that observed intermediate results. That is a limit on its recovery guarantee, not a prohibition on sending cancellation events or notifying affected people.

A compensation cannot erase an earlier observation. If a customer saw a reservation or a downstream report counted it, the workflow may need a correction event, revised report, or user notification. Design and track those actions explicitly; they do not follow automatically from compensating the local database change.

Each local transaction has already committed, so a later failure cannot roll all participants back as one database transaction. Without additional application controls, concurrent sagas can overwrite each other’s changes or observe a process that later compensates. A saga’s exposed intermediate state is distinct from reading an uncommitted local database transaction.

The original paper assumes relatively independent steps that can tolerate interleaving. If later correctness requires unchanged earlier data, a saga alone does not provide that isolation. You must reserve the relevant resources, validate versions and handle conflicts, or choose a stronger transaction boundary. A sequence of independent commits is not a cheaper implementation of one shared snapshot.

Compensation, and the step that has none

A compensating transaction reverses a committed step in business terms. It runs against the state that exists now, which may include other transactions’ work. The saga paper distinguishes this semantic reversal from restoring the database’s earlier contents.

Its seat-reservation example makes the distinction concrete: cancel this reservation rather than restore the old seat count, because other customers may have booked in the meantime. The same reasoning applies to voiding an authorization before capture, refunding a captured payment when permitted, or issuing a credit note. The corrective operation must preserve unrelated work and its audit evidence.

Which means compensations are ordinary business operations and need the same care as the forward path.

• They must be safe to repeat. A compensation that runs twice and refunds twice has turned a failure into a loss, so it needs the same treatment as any retried operation — see idempotency and, for the mechanism, an idempotency key.
• They can fail on their own. Record compensation progress durably and make retries safe. A temporarily unreachable payment provider can then be retried from the recorded position. Use spaced, bounded attempts and escalate with the failure details on a permanent error or deadline; persistence does not mean silently retrying forever.
• They leave a trace, and should. A refund is a real event with real accounting consequences. Compensation that quietly reverses a record destroys the evidence of what happened, which is why the forward step and its compensation both belong in the audit log.

Classify steps by recovery policy. Compensable steps have an acceptable business reversal. The pivot marks the workflow’s transition from backward recovery to forward completion. Microsoft’s saga guidance allows that boundary to be an irreversible action, the last compensable transaction, or the first retryable operation. Retryable steps after it must be designed for safe repetition and eventual completion under stated recovery assumptions.

Locate that boundary deliberately. A disclosure with no withdrawal path may force the decision, while another process can choose to stop compensating after its last reversible reservation. The boundary is a workflow policy, not simply a label attached to a physical action.

Physical irreversibility still differs from business compensation. An allowed refund can compensate a settled payment, and an authorized return can compensate a shipment, even though neither erases the past. Decide whether that reversal belongs to this workflow or to a separate process, and account for its cost and possible failure.

Validate critical conditions and reserve required resources before crossing the pivot where feasible. After it, retain progress and the code needed to finish. Idempotency makes repetition safe; it does not make a permanent rejection succeed. State which transient failures can recover and who owns exception handling or manual completion when forward progress remains impossible.

Choreography lets participants trigger one another through events; orchestration gives an explicit controller responsibility for progression and recovery. For a complex audited process, I would favor an orchestrator because it provides a natural place to inspect workflow state. Choreography can also feed a central status projection using correlation IDs and outcome events. Either view is only as complete and current as the evidence it receives; orchestration also adds a recovery dependency that needs an owner.

Getting the message out of the building

All of the above assumes each step can tell the next one to proceed. That assumption hides a failure that predates any saga logic: committing a row and publishing a message are two operations, and the process can end between them.

AWS describes the dual-write problem as a database change and an event publication that can fail independently. Committing without notification leaves a consumer unaware; notifying without a successful database change can make it act on something that never committed.

The standard fix is to write the message into the same database transaction as the business change and let a separate relay deliver it afterwards, which is the transactional outbox. The commit becomes one atomic act — the order and the intent to notify — and delivery becomes a separate, retryable concern.

What the outbox buys is narrow and important: it closes the dual-write gap, so a committed business change can never be left with no record of the intent to publish it. What it does not buy is end-to-end delivery. Everything after the commit still has to hold — the relay has to come back and resume, the broker has to durably acknowledge what it accepted, and the records have to be retained long enough on both sides for that to happen. An outbox row deleted before a confirmed handover, or a broker retention window shorter than a consumer’s outage, loses the message just as thoroughly as a failed publish would have. Nor does it buy you that the message arrives once. The relay can crash after sending and before marking the record sent, so delivery is at-least-once, and AWS recommends recording processed message identities at the consumer to make repeats harmless. That tracking is event deduplication, and it belongs to the consumer — a sender cannot supply it.

Order is the other thing to state rather than assume. AWS’s ordering requirement needs an explicit source and delivery contract. Three places can disrupt it. A sequence number is allocated when the row is written but becomes visible when the transaction commits, so a relay can legitimately see a later number first. Two relay instances, several broker partitions, or one message retried after a subsequent one succeeded will each deliver out of order. And anything parked in a dead letter queue and replayed arrives after everything published in the meantime.

Choose the required ordering scope. For an entity, serialize its updates or use conditional version changes, and commit the source-assigned version with its event. Keying messages by entity can keep them in one ordered partition, but cannot repair an order already lost by the relay. What the consumer may skip depends on both the payload and the work it promises to perform.

For a latest-state projection, complete state plus a version guard can be enough. Compare versions from the same source entity and generation, and atomically replace only with a newer state. Define how conflicting payloads at the same version are handled. Preserve versioned deletion evidence long enough to reject stale updates that would resurrect a row. This contract permits skipping superseded states; it does not cover consumers required to act on every transition.

For partial changes, a newer message may not contain an older message’s effect. Starting at {status: open, amount: 100}, let v1 carry {status: paid} and v2 carry {amount: 80}. Applying v2 and dropping v1 leaves the status wrong. Such changes need source ordering or a merge rule that preserves their effects. Unconstrained integer increments are different: +3 and +5 commute, so each needs atomic application with duplicate recognition, but not predecessor waiting. Intermediate balance constraints or external actions can make their order significant again.

For order-dependent commands, hold a later message until required predecessors have been applied and recover missing work from retained events, commands, or an authoritative business ledger. Fetching the entity’s current state can repair a state projection, but cannot reconstruct every lost transition or prove that associated emails, charges, or other actions occurred. Missing history or effects need investigation and an explicit correction path. Event-ID deduplication prevents repeated application; it does not supply missing events or restore source order.

Choose full-state messages when the consumer only needs the current projection and may skip intermediate states. A final refunded state can replace an earlier paid state in that projection, but does not establish whether a separate action required by the paid transition happened. Keep that contract distinct from reconstructing a view using source versions, discussed under deterministic ordering.

Use Retry-After and backoff to space retries. After the retry policy is exhausted, a dead letter queue can isolate the failing message for owned recovery. It does not make later dependent commands safe: pause that entity or ordered group when predecessors are required, while allowing independent work to proceed. Replay must respect the ordering or merge contract and duplicate protection.

When the other system is simply down

Some participants are outside your control entirely, and their unavailability is a design input rather than an incident. Three responses exist, and choosing between them is a business decision that engineering should not make alone.

• Refuse the work. Honest and sometimes correct — for a payment you cannot authorize, better than promising. The cost is visible and immediate.
• Accept and defer. Queue work when delayed completion is permitted and the system can retain and recover it. Show the pending state and define a deadline, notification, and cancellation or escalation path instead of presenting acceptance as fulfillment.
• Accept and proceed provisionally, then reconcile. Use this only within explicit risk and business-policy limits, such as an order accepted against a possibly stale stock count. Where a credit decision requires a completed check, defer the decision or refuse it; a different approved policy may permit a bounded provisional action.

The saga paper points out a recovery dependency beyond database state: completing missing steps or running compensations requires the corresponding application code to remain available.

A database recovers from its log with no help from the application. A half-finished business process does not. Resuming it requires the code that knows what step five was and how to compensate step four — which means recovery depends on a deployable, working application, and a saga state store whose records are meaningless without it. That has a blunt operational implication: retiring the service that owned an orchestration does not free you from the sagas it left open.

Checking business completion

An outbox commit proves durable intent within its transaction; a broker acknowledgement proves the configured handover; a workflow record describes observed steps. Those facts are useful but must be connected to the business completion condition. A successful step response may still coexist with a wrong-account update, an unshipped order, or a later overwrite.

Define the completion condition and the evidence that can establish it: a business ledger entry, a shipment confirmation, or correlated outcome records under a stated contract. reconciliation is an important recurring control for checking the relationship between those records. It supports that claim within its scope; two systems sharing an error can agree, and agreement alone does not prove a physical delivery happened. Four decisions make the comparison useful.

1. What is compared, in business terms. Orders accepted against orders shipped. Charges authorized against charges captured. Not message counts — matching message counts are compatible with every order going to the wrong warehouse.
2. As of when. Different cutoffs can create apparent discrepancies when relevant changes occur between them. State a boundary both sides can honor, including late arrivals and permitted in-flight work. If nothing relevant changed, different observation times need not produce different results.
3. How often, and at what depth. Cheap aggregate checks often, full item-level comparison occasionally. The interval is a decision about how long you are willing to have been wrong.
4. Who resolves a break, and by when. This is the step that gets skipped, and skipping it converts reconciliation into a report that accumulates known-wrong rows. Detection is not correction; the two need separate owners and separate deadlines.

A discrepancy narrows an investigation; it does not identify the failed integration by itself. Check cutoffs, definitions, observation gaps, and each system’s evidence. Shared terms such as order, customer, and shipped need agreed mappings between domains, which a data contract and a clear bounded context help make explicit.

What solves what

Mechanism	Solves	Does not solve
Two-phase commit	Atomic commit decisions across enlisted resources with a compatible protocol and recovery contract	Unsupported participants, blocking, or the operational cost of holding prepared state
Saga	Coordinating local steps and their recovery under the workflow’s contract	Automatic isolation or erasure of prior observations; compensation events and notifications need explicit design
Compensating transaction	A business-level reversal of a committed step, resumable from where it failed	Restoring a prior state, and steps your business rules admit no acceptable reversal for
Transactional outbox	Closing the dual-write gap: a commit never leaves the intent to publish unrecorded	Delivery. Relay resumption, confirmed broker handover, and retention still have to hold — plus duplicates and ordering
Idempotent consumer, deduplication	Duplicates being harmless	Messages that were never sent, or business rules applied wrongly
Dead letter queue	Isolating failed work for diagnosis and recovery	Safe progress of dependent commands, ordered replay, or resolution without an owner
Reconciliation	Testing agreement with a defined reference at an aligned boundary	Shared errors, unobserved physical effects, or automatic correction

Each mechanism contributes evidence or a recovery capability under its own conditions. Connect those conditions to the business result rather than treating a green transport dashboard as proof that an order shipped. Local compensation, message handling, and final-state verification answer different questions.

For each cross-system process, write down what counts as business completion, which evidence supports it, and where that evidence is evaluated. Assign a cadence, an owner, and a path for unknown or conflicting outcomes. That makes failures investigable without claiming that one comparison or workflow status proves everything.

References

• “Sagas,” Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data
• PostgreSQL documentation, PREPARE TRANSACTION
• Microsoft Azure Architecture Center, Saga Design Pattern
• AWS Prescriptive Guidance, Transactional outbox pattern
• PostgreSQL Documentation, Explicit Locking — Row-Level Locks
• Microsoft Azure Architecture Center, Compensating Transaction pattern
• Narayana JTA Administration Guide, Failure Recovery Administration
• RFC 9110, HTTP Semantics
• Microsoft, Asynchronous Request-Reply
• Amazon SQS, Using dead-letter queues
• Garcia-Molina and Salem, Sagas (SIGMOD 1987, pp. 249–259; Cornell-hosted paper)