Nothing Rolls Back: Integrating Enterprise Systems Without Distributed Transactions

An order is placed. The order service commits it, the payment service authorizes the card, the inventory service reserves stock, and the warehouse system — which runs on someone else’s schedule and accepts a file twice a day — is told to ship. Four systems, four databases, and one thing the customer believes happened.

Now suppose the third step fails. There is no transaction to roll back, because two of the four already committed and one of them charged a card. Whatever happens next is something you have to have designed in advance, and this is the subject of the article: what you actually get when you give up distributed transactions, and what you have to build to replace them.

The through-line is a distinction worth stating before the mechanisms. Technical completion means the message was delivered, the call returned 200, the file was picked up. Business completion means the order is actually fulfilled, the payment is actually settled, the two ledgers actually agree. They are different facts, they fail independently, and no amount of messaging infrastructure turns the first into the second.

Why not one transaction

The obvious answer to the scenario above is a transaction that spans all four systems, and the protocol for that exists. Two-phase commit has a coordinator ask every participant to prepare, and commit only when all of them have agreed. It is real, it works, and databases implement it. PostgreSQL’s own documentation is unusually candid about what it costs.

Preparing a transaction means “its state is fully stored on disk, and there is a very high probability that it can be committed successfully, even if a database crash occurs before the commit is requested.” Note the phrasing — a very high probability, not a certainty. The purpose is “to allow an external transaction manager to perform atomic global transactions across multiple databases or other transactional resources,” and the documentation adds: “Unless you’re writing a transaction manager, you probably shouldn’t be using PREPARE TRANSACTION.”

The caution that follows names the real cost. “It is unwise to leave transactions in the prepared state for a long time. This will interfere with the ability of VACUUM to reclaim storage, and in extreme cases could cause the database to shut down to prevent transaction ID wraparound.” And: “Keep in mind also that the transaction continues to hold whatever locks it held.”

That is the cost in one sentence: a prepared transaction holds its locks until someone tells it the answer, and the thing that tells it is a coordinator that can itself fail.

Be precise about what those locks block, because it is narrower than it sounds. In PostgreSQL, “row-level locks do not affect data querying; they block only writers and lockers to the same row.” An ordinary read still gets a visible version. What waits is a conflicting write — and where a stronger table-level lock was taken, more than that.

Be equally precise about the coordinator failing, because “a human has to decide” is wrong for a properly deployed transaction manager. Recovery is automatic and log-based: the decision is written durably before phase two, so after a crash the manager replays it. Narayana’s documentation describes the shape — after a machine or network failure “the recovery will not take place until the system or network are restored, but the original application does not need to be restarted,” and “recovery responsibility is delegated to The Recovery Manager,” using information that “is held in the ActionStore.” Manual intervention is the exception, for cases where the log is gone or a heuristic outcome has already occurred, and guessing is actively harmful: if the coordinator logged commit and one participant committed before it died, aborting the other by hand breaks the atomicity the protocol delivered.

What remains true is the blocking. Until the coordinator or its log comes back, the affected rows are held — Narayana notes that “resources affected by a transaction that was in progress at the time of the failure may be inaccessible,” reported by databases as rows held by “in-doubt transactions.” That is a partial failure whose duration is set by how quickly the coordinator recovers, and it is why the coordinator needs an owner rather than a hostname.

Inside one organization with one database vendor and a workload measured in milliseconds, that risk can be managed. Across the order service, a card processor, and a warehouse system that answers twice a day, it cannot — the third participant would have to hold locks for hours, and the payment provider will not enlist in your transaction at all. So the practical constraint in enterprise integration is not that two-phase commit is bad. It is that at least one participant will always be something you do not control.

Three integration styles, one business process

Real estates do not pick one style. They accumulate all three, usually in this order, and each arrives with a different definition of “done.”

	What “success” means	How it fails	Where completion is visible
Synchronous call	A 2xx response	Timeout — the caller does not know whether the work happened	In the caller, immediately, and nowhere afterwards unless it recorded the outcome
Asynchronous message	The broker accepted the message	Consumer fails, retries, duplicates, or reorders; the sender learns nothing	Only in the consumer, later
File transfer / batch	The file was written to the agreed location	Wrong cutoff, partial file, a day skipped, a day processed twice	In tomorrow’s report

Two consequences follow, and the second is the one that ruins weekends.

First, end-to-end latency is the sum along the critical path, not the worst single step — three sequential stages of one, two, and three minutes take six minutes, not three. What a batch window adds on top of that sum is waiting whose size depends on arrival time: work that arrives just before a twice-daily cutoff waits minutes, and work that arrives just after waits most of a half-day. So a batch step usually dominates the worst-case budget and inflates the average, without making the other steps free. A promise of immediate fulfilment in the user interface is a promise about one step of that path.

Second, and more important: no single system can tell you whether the business process finished. The order service knows it published an event. The message broker knows it was consumed. The warehouse knows it received a file. Nobody holds the sentence “this order has been shipped and paid for.” That sentence has to be assembled deliberately, and the fact that it has to be assembled is why the last section of this article exists.

A note on which style to reach for, since the choice is usually made by habit. Use a synchronous call when the caller cannot proceed without the answer and the answer is fast. Use a message when the work can happen later, which is most of the time — and note that this buys a delay you must now tolerate, not convergence you have been given. Eventual consistency holds only where propagation is durable, conflicts have a resolution rule, and something detects and repairs what was missed; without those a message-based copy drifts rather than converges. Use a file when the other side only accepts files — which is a real and permanent condition in enterprise estates, not a legacy embarrassment to be designed away.

What a saga actually guarantees

The standard answer to multi-system work is a saga: break the process into local transactions, each committing in one system, and define how to undo the earlier ones if a later one fails. The idea is older than microservices. Garcia-Molina and Salem introduced it in 1987 for long-lived transactions, and their formulation is precise in a way most retellings are not.

Their definition: a long-lived transaction “is a saga if it can be written as a sequence of transactions that can be interleaved with other transactions,” and the system “guarantees that either all the transactions in a saga are successfully completed or compensating transactions are run to amend a partial execution.”

Read that guarantee closely, because it is narrower than “the transaction is atomic.” It promises that one of two sequences will run: all the steps, or some prefix of the steps followed by their compensations in reverse. It does not promise that the intermediate states were invisible. The authors say so themselves, in a parenthesis that deserves to be the most-quoted sentence in the paper and is not:

Note that other transactions might see the effects of a partial saga execution. When a compensating transaction Cj is run, no effort is made to notify or abort transactions that might have seen the results of Tj before they were compensated for by Cj.

That is the whole difference between a saga and a rollback, stated by the people who invented it. A rollback makes it as though nothing happened. A saga makes a correcting entry and leaves everything that already read the intermediate state exactly as it was. If a customer saw “reserved,” if an operator acted on it, if a downstream report counted it — those happened, and compensation does not unhappen them.

Modern pattern documentation makes the same point in operational language: “Data can’t be rolled back because saga participants commit changes to their respective databases.” And since there is no isolation across participants, the anomalies you would be protected from inside one database are all available — lost updates, “dirty reads,” and “fuzzy, or nonrepeatable, reads” among them.

The condition under which a saga is the right shape is also in the original paper, and it is worth checking against your process before adopting the pattern. A saga fits when the work is “a sequence of relatively independent steps, where each step does not have to observe the same consistent database state.” The authors justify it from the physical world: “In reality, one does not physically lock the warehouse until a purchase order is fully processed.” If your steps genuinely must all see one consistent state — if step four’s correctness depends on nothing having changed since step one — then a saga is not a way to do it more cheaply. It is a different thing that does not do it.

Compensation, and the step that has none

A compensating transaction is not an undo. The paper is explicit: it “undoes, from a semantic point of view, any of the actions performed by Ti, but does not necessarily return the database to the state that existed when the execution of Ti began.”

The example given is the clearest illustration of why. If a step reserves a seat, its compensation “can cancel the reservation” — but it “cannot simply store in the database the number of seats that existed when Ti ran because other transactions could have run between the time Ti reserved the seat and Ci canceled the reservation.” Restoring the old value would erase everyone else’s work. Compensation has to be expressed as a business operation with the opposite effect: cancel the reservation, refund the charge, issue a credit note. Never “put the number back.”

Which means compensations are ordinary business operations and need the same care as the forward path.

• They must be safe to repeat. A compensation that runs twice and refunds twice has turned a failure into a loss, so it needs the same treatment as any retried operation — see idempotency and, for the mechanism, an idempotency key.
• They can fail on their own. Pattern guidance states it plainly: “compensating transactions might not always succeed, which can leave the system in an inconsistent state.” The documented response is durable resumption first, not escalation — “the system should record progress so that it can resume the compensating transaction from the point of failure,” and since “a step might run multiple times when retried,” each step should be “an idempotent command.” A payment network that was briefly unreachable is the case this handles. Escalation is for what durable retry cannot fix: “sometimes manual intervention is the only way to recover from a failed step,” and then “the system should raise an alert that includes detailed information about the reason for the failure.” So: resume with bounded, spaced attempts; escalate on a permanent error or a deadline; never retry forever in silence.
• They leave a trace, and should. A refund is a real event with real accounting consequences. Compensation that quietly reverses a record destroys the evidence of what happened, which is why the forward step and its compensation both belong in the audit log.

Then there is the step that cannot be compensated at all. The useful vocabulary here classifies a saga’s steps into three kinds: compensable transactions, which “can be undone or compensated for by other transactions with the opposite effect”; the pivot transaction, which serves “as the point of no return in the saga” — after it succeeds, “compensable transactions are no longer relevant”; and retryable transactions, which “follow the pivot transaction” and are “idempotent,” so the saga can still reach its final state through repetition rather than reversal.

Finding the pivot in your own process is the most useful hour you can spend on this design. The candidates are usually physical or external — the pallet left the dock, the email went to the customer, the funds settled, the filing was submitted.

But apply the test carefully, because irreversible in the physical world is not the same as non-compensable in the business. Compensation was always new work rather than a restoration, so a permitted refund compensates a settled payment and an authorised return compensates a shipment — neither undoes the past, and both can be perfectly good compensations. What makes a step a genuine pivot is that your workflow’s rules admit no acceptable reversal: a disclosure that cannot be unsent, a regulatory filing with no withdrawal path, a payout to a counterparty with no recall. Classifying every settlement as a pivot by reflex rules out compensations you could have designed; assuming every shipment is reversible ignores that returns cost money, take weeks, and sometimes fail. The question is what the business will accept, and it is answered per workflow.

Two moves follow from knowing where it is. Push the irreversible step as late in the sequence as the business allows, so that most failures happen while reversal is still possible. And treat everything after the pivot as work that must eventually complete rather than work that might be abandoned — which is the original paper’s second recovery direction: “When a failure interrupts a saga, there are two choices: compensate for the executed transactions, backward recovery, or execute the missing transactions, forward recovery,” with the honest qualification that “forward recovery may not be an option in all situations.”

Where the coordination lives is a smaller decision than it looks. Choreography has “services exchange events without a centralized controller”; orchestration puts “a centralized controller, or orchestrator,” in charge of the sequence. Choreography suits short flows and gets hard to follow as steps accumulate — with published trade-offs including that it is “difficult to track which commands each saga participant responds to” and that integration testing needs every service running. Orchestration makes the flow legible and introduces “a point of failure because the orchestrator manages the complete workflow.” For a process that crosses departmental systems and will be audited, I would take the orchestrator: when someone asks where order 4471 stopped, an orchestrator has the answer in one place, and a choreographed flow requires reconstructing it from logs in six.

Getting the message out of the building

All of the above assumes each step can tell the next one to proceed. That assumption hides a failure that predates any saga logic: committing a row and publishing a message are two operations, and the process can end between them.

AWS’s guidance names it the dual write problem — it “occurs in distributed systems when a single operation involves both a database write operation and a message or event notification” — and spells out both directions of damage. “If the database update is successful but the event notification fails, the downstream service will not be aware of the change, and the system can enter an inconsistent state. If the database update fails but the event notification is sent, data could get corrupted.”

The standard fix is to write the message into the same database transaction as the business change and let a separate relay deliver it afterwards, which is the transactional outbox. The commit becomes one atomic act — the order and the intent to notify — and delivery becomes a separate, retryable concern.

What the outbox buys is narrow and important: it closes the dual-write gap, so a committed business change can never be left with no record of the intent to publish it. What it does not buy is end-to-end delivery. Everything after the commit still has to hold — the relay has to come back and resume, the broker has to durably acknowledge what it accepted, and the records have to be retained long enough on both sides for that to happen. An outbox row deleted before a confirmed handover, or a broker retention window shorter than a consumer’s outage, loses the message just as thoroughly as a failed publish would have. Nor does it buy you that the message arrives once. The relay can crash after sending and before marking the record sent, so delivery is at-least-once, and the guidance is explicit about the consequence: “we recommend that you make the consuming service idempotent by tracking the processed messages.” That tracking is event deduplication, and it belongs to the consumer — a sender cannot supply it.

Order is the other thing to state rather than assume. The same guidance asks you to “send messages or events in the same order in which the service updates the database,” which is harder than it sounds and fails in three distinct ways. A sequence number is allocated when the row is written but becomes visible when the transaction commits, so a relay can legitimately see a later number first. Two relay instances, several broker partitions, or one message retried after a subsequent one succeeded will each deliver out of order. And anything parked in a dead letter queue and replayed arrives after everything published in the meantime.

So pick the order you actually need instead of hoping for a global one. Per-entity is usually enough and is achievable: have the producing transaction assign a version to that entity, and key the message by the entity so one partition holds its history. What the consumer may then do with an out-of-order arrival depends on what the message contains, and this is the distinction that decides whether the cheap rule is safe.

If every message carries the entity’s complete current state, a consumer can compare versions and discard anything older than what it holds. Arrival order then genuinely stops mattering, because the newest message already contains the effect of every message before it. The check has to be atomic with the write — compare and replace in one operation, or the two concurrent appliers race and the older one wins.

If messages carry changes rather than states, discarding an older version loses data. Take an order at {status: open, amount: 100}, with v1 = {status: paid} and v2 = {amount: 80}. If v2 arrives first and the consumer then drops v1 as stale, the result is {status: open, amount: 80}. The correct answer is {status: paid, amount: 80}, and nothing about it will look broken — there is no error, just an order that was never paid. The same applies to increments and decrements, which are commutative and therefore order-insensitive but must each be applied exactly once, and to business commands, which are not optional.

For that second contract the consumer needs real ordering rather than a version filter: serialize by entity, hold an out-of-order message until its predecessor has been applied, and have a way to recover a version that never arrives — replay from the source, or fetch the entity’s current state and reconcile. Deduplicating on event ID is a separate requirement and does not substitute for it. Note also that keying by entity does not repair order that was already lost: a single partition preserves the sequence in which the relay published, so if the relay sent v2 before v1, the partition faithfully delivers v2 before v1.

The cheap route, where the domain allows it, is to make every message a complete state. That is a producer-side decision worth taking deliberately, because it is what buys the version-comparison rule. Reconstructing state from source-assigned versions, rather than from the order things landed, is the reading side of the same discipline — see deterministic ordering.

Retries need spacing, not just persistence — a failing downstream system being retried aggressively by four upstream ones is how a slow dependency becomes an outage, which is the practical content of Retry-After and backoff. And messages that cannot be processed after a bounded number of attempts need a destination other than the queue they are blocking: a dead letter queue, which is only useful if someone actually reads it. An unwatched dead letter queue is a place where business transactions go to be forgotten with a clear conscience.

When the other system is simply down

Some participants are outside your control entirely, and their unavailability is a design input rather than an incident. Three responses exist, and choosing between them is a business decision that engineering should not make alone.

• Refuse the work. Honest and sometimes correct — for a payment you cannot authorize, better than promising. The cost is visible and immediate.
• Accept and defer. Queue the work and complete it when the system returns. This is usually right, and it carries an obligation people forget: you have now promised something you have not done, so the promise needs a visible state and a deadline after which someone is told.
• Accept and proceed provisionally, reconciling later. Appropriate when the risk of being wrong is small and bounded — accepting an order against a stock figure that might be stale. Not appropriate for a credit check.

Recovery afterwards has a property that catches teams out, and the 1987 paper identified it: “To complete a running saga after a crash it is necessary to either complete the missing transactions or to run compensating transactions to abort the saga. In either case it is essential to have the required application code.”

A database recovers from its log with no help from the application. A half-finished business process does not. Resuming it requires the code that knows what step five was and how to compensate step four — which means recovery depends on a deployable, working application, and a saga state store whose records are meaningless without it. That has a blunt operational implication: retiring the service that owned an orchestration does not free you from the sagas it left open.

The control that tells you the truth

Everything so far improves the odds. None of it establishes that the business process completed, because every mechanism above reports on itself: the outbox says the message was durable, the broker says it was consumed, the saga store says the steps returned success. All three can be green while an order sits unshipped because a consumer wrote to the wrong account, or a file was processed twice, or a step succeeded and its effect was later overwritten by something else.

The only thing that answers the business question is comparing the systems against each other: reconciliation, run as a standing control rather than as an investigation after someone complains. Four decisions make it real.

1. What is compared, in business terms. Orders accepted against orders shipped. Charges authorized against charges captured. Not message counts — matching message counts are compatible with every order going to the wrong warehouse.
2. As of when. Two systems with different cutoffs always disagree, and most “discrepancies” found by a first reconciliation are two clocks rather than two truths. The comparison needs a stated boundary both sides can honour.
3. How often, and at what depth. Cheap aggregate checks often, full item-level comparison occasionally. The interval is a decision about how long you are willing to have been wrong.
4. Who resolves a break, and by when. This is the step that gets skipped, and skipping it converts reconciliation into a report that accumulates known-wrong rows. Detection is not correction; the two need separate owners and separate deadlines.

Reconciliation is also what makes the rest of the estate legible. A break tells you which integration is failing and roughly when, which is information no amount of per-service monitoring produces. And it is the argument for writing down what each system means by a shared term in the first place — “order,” “customer,” “shipped” — since two systems cannot be reconciled on a field whose definition they never agreed, which is what a data contract and a clear bounded context are for.

What solves what

Mechanism	Solves	Does not solve
Two-phase commit	Atomicity across resources you fully control, with automatic log-based recovery	Participants that will not hold locks for you, and the blocking while a failed coordinator is still down
Saga	Which sequence runs: all steps, or a prefix plus compensations	Isolation. Others see the intermediate state and are never notified
Compensating transaction	A business-level reversal of a committed step, resumable from where it failed	Restoring a prior state, and steps your business rules admit no acceptable reversal for
Transactional outbox	Closing the dual-write gap: a commit never leaves the intent to publish unrecorded	Delivery. Relay resumption, confirmed broker handover, and retention still have to hold — plus duplicates and ordering
Idempotent consumer, deduplication	Duplicates being harmless	Messages that were never sent, or business rules applied wrongly
Dead letter queue	One bad message not blocking the rest	Anything at all, if nobody reads it
Reconciliation	Discovering that the systems disagree	Making them agree — that is a separate job with a separate owner

The column that matters is the second one. Each row is a real guarantee about a mechanism, and none of them is a guarantee about the business process. Reading them together is how you avoid the two errors that make integration work go wrong: believing that a saga gives you a rollback, and believing that green dashboards mean the orders shipped.

If I had to reduce it to one habit, it would be this. For every cross-system process, write down the sentence that means it finished in business terms, name the one place that sentence can be evaluated, and make something check it on a schedule. Teams that can produce that sentence handle integration failures as ordinary work. Teams that cannot find out how many there were during an audit.

References: Hector Garcia-Molina and Kenneth Salem, “Sagas,” Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, pp. 249–259; PostgreSQL documentation, PREPARE TRANSACTION; Microsoft Azure Architecture Center, Saga Design Pattern (documentation dated 2025-02-25); AWS Prescriptive Guidance, Transactional outbox pattern; PostgreSQL Documentation, Explicit Locking — Deadlocks.