Replication and Consistency: Living With Copies That Disagree

A customer updates their shipping address in a retailer’s account settings, sees a confirmation, and immediately reloads the page. The old address is back. Nothing failed. The write went to one copy of the data and the reload was served by another that had not received it yet. A minute later both copies agree and the problem is gone, which is why the support ticket will be closed as unreproducible. The retailer is invented, but the shape of the complaint is not.

Timeouts and retries, the subject of the companion piece on partial failure, are about not knowing whether something happened. This is the other half: once data exists in more than one place, the copies can hold different values at the same moment, and every design has to say what a reader is allowed to see. This article works through why copies exist, what each way of keeping them in step costs, what the models between strong and eventual consistency actually promise, how to use CAP and PACELC without misreading them, when a system genuinely needs consensus, and why clocks cannot be trusted to order events.

Why there is more than one copy

Replication is not a feature someone chose for its own sake. It is the answer to three separate requirements, and which one is driving matters, because it changes what the replicas are for.

• Surviving failure. A machine, a rack, or a data center will eventually be lost. Durability and availability targets that exceed the reliability of one machine require the data to exist somewhere else before the loss, not after it.
• Serving more reads. One node has a ceiling. A read replica raises read throughput without touching the write path, which is why reporting and search workloads are so often pointed at one.
• Being close to the reader. The speed of light is a fixed cost. A user in Singapore reading from a database in Virginia pays a round trip that no amount of tuning removes, so a copy near the reader is the only thing that removes it.

Each of these is a good reason, and each creates the same problem. Abadi’s 2012 analysis states it plainly: as soon as a distributed database replicates data, a trade-off between consistency and latency arises. The trade-off is not caused by failures. It is present during normal operation, every time a write has to reach more than one place.

Three ways to replicate, and what each costs

Abadi makes a useful reduction: an update can be sent to all replicas at the same time, to an agreed-upon master node first, or to a single arbitrary node first. Everything else is a variation on one of these, and each carries its own consistency and latency cost.

Scheme	How it behaves	What it gives up
To all replicas at once, with no agreement step	Replicas may apply concurrent updates in different orders	Consistency: the copies diverge, and only converge if the operations happen to commute
To all replicas at once, through an agreement protocol or a preprocessor that fixes the order	Every replica applies updates in the same order	Latency: the protocol itself costs round trips, and a single preprocessor forces every write worldwide to route through it
To a master node first, replicated synchronously	The master waits before acknowledging. What it waits for is a design choice: every replica, a named subset, or a quorum	Latency: bounded by the slowest replica it waits for, so waiting for all is bounded by the slowest of all, which is painful across a wide-area network. Waiting for a quorum trades some of that latency for weaker guarantees when a lagging replica is later read
To a master node first, replicated asynchronously, reads served only by the master	The write returns before replication finishes; readers still see a single authoritative copy	Latency and availability of reads: a nearby replica cannot answer, and an overloaded or failed master has no substitute
To a master node first, replicated asynchronously, reads served by any replica	Readers may see values older than the last committed write	Consistency: this is the case that produces the disappearing address
To an arbitrary node first	Different writes to the same item may start at different nodes. The receiving node can replicate synchronously or asynchronously before acknowledging, as in the master case	Consistency: with no single node deciding order, concurrent writes have to be reconciled, whether by quorum overlap, by a conflict resolution rule, or by keeping both versions

Two things follow. First, “we replicate asynchronously” is not a complete description of a system. What a reader sees depends on where reads are routed as much as on how writes propagate, and the two are separate decisions.

Second, the label “eventually consistent” is applied more loosely than the table warrants. It fits the master-with-asynchronous-replication row where any replica may answer: a reader can see a value older than the last committed write, and the copies converge afterwards. The arbitrary-node row is not one thing. What it guarantees depends on the protocol layered on top — how many replicas a write and a read touch, how concurrent writes are ordered or merged, and what the system does with versions it cannot order. Some configurations of such systems are eventually consistent; others provide considerably more. The label describes a category of designs, not a guarantee you can rely on without reading the specific system.

Where the copies are allowed to diverge, how long “eventually” lasts is an operational property to be measured, usually as replication lag, and it belongs in monitoring next to data freshness.

Quorum systems, common in the arbitrary-node family, offer a dial rather than a switch. With N replicas, a write acknowledged by W of them and a read answered by R of them, raising R + W buys more consistency at the cost of latency. Abadi notes the limit of that dial: in the systems he is describing, consistency improves as R + W grows, but setting R + W greater than N does not by itself deliver full consistency in the sense Gilbert and Lynch defined. Overlapping quorums guarantee that a read touches a replica that saw the latest acknowledged write; they do not by themselves settle the order of concurrent writes or prevent a reader from seeing an in-progress one. Systems that do offer linearizable operations on top of quorums add protocol to get there. The lesson is not that quorums are weak but that the arithmetic of R and W is not the whole guarantee, and the documentation of the specific system is.

Between strong and eventual there is a map, not a gap

Teams often discuss consistency as a binary: strong, or eventual. Both ends are precise, and almost every real system lives between them.

The strong end has a definition. In their proof of CAP, Gilbert and Lynch used atomic, or linearizable, consistency, which Abadi’s paper restates: there must be a total order on all operations such that each one looks as if it completed at a single instant, so that the distributed store behaves as if it were a single node answering one operation at a time. That is the property that makes a system feel like it has no replicas at all, and it is the one that costs the most to provide.

The weak end is defined by what it does not promise. Eventual consistency says only that if writes stop, the copies converge. It says nothing about what a reader sees before then, including whether they can see their own writes.

The interesting models are in between, and the useful ones for application design are the session guarantees, because they are stated from the point of view of one client rather than of the whole system.

Model	What a reader is promised	What it fixes
Read your writes	A client sees its own previous writes	The disappearing address
Monotonic reads	A client never sees the data move backwards in time	A page that shows a newer value, then an older one on refresh
Monotonic writes	A client’s writes are applied in the order it issued them	An update overwritten by an earlier update from the same client
Causal consistency	Operations that could have influenced one another are seen in that order by everyone	A reply appearing before the message it answers
Snapshot isolation	A transaction sees a consistent point-in-time view of the data	A report whose totals were computed from a half-updated database
Serializability	Concurrent transactions produce a result equivalent to some serial order	Two withdrawals that each check the balance before either deducts
Linearizability	Operations on a single object appear to take effect at one instant, in an order consistent with real time	A read that returns a value older than one the same client already saw acknowledged, on that object
Strict serializability	Serializability plus real-time order, across multi-object transactions	A transaction that reads two accounts and sees them at inconsistent points, or a committed transaction invisible to one that starts after it

The last two rows are often treated as one, and they are not. Linearizability is a guarantee about single objects, so a store can make every individual key linearizable and still allow a transaction spanning two keys to observe them at different moments. Serializability is the multi-object guarantee, and adding real-time order to it gives strict serializability. A system that advertises linearizable reads and writes has told you something about each key, not about anything you do with two of them at once.

It is also worth saying what none of these models buys. They constrain what a reader may observe; they say nothing about whether the values are right. A strictly serializable database will faithfully preserve a balance that was computed with the wrong rule. Consistency in this technical sense and correctness in the business sense are separate properties, and only one of them is the storage layer’s job.

Jepsen’s consistency reference adds the part that turns this map into a design tool: the price of each model in availability during a network partition. Models at or stronger than cursor stability, snapshot isolation, and sequential consistency cannot be totally available in an asynchronous network. Models at or stronger than read your writes can at best be sticky available, meaning available only as long as a client keeps talking to the same replica. Models weaker than that can be totally available.

This is why the choice deserves a conversation rather than a default. Four questions usually settle it for a given operation.

• Who reads it, and do they also write it? A user reading their own profile needs read-your-writes. A dashboard aggregating yesterday’s orders does not.
• What happens if the value is stale by a second, a minute, an hour? Staleness that produces a wrong number on a screen is different from staleness that lets an item be sold twice.
• Is the operation a decision or a display? Decisions that consume a limited resource, such as inventory, seats, or a credit limit, are where weak models cause real losses.
• Can the damage be detected and undone afterwards? If over-selling can be caught and compensated, a weaker model plus a correction process may be cheaper than global coordination.

CAP is a tool for one moment, not a taxonomy

CAP is the most cited and most misapplied result in this area. Eric Brewer, who stated it, spent a 2012 article correcting its reception, and his corrections are more useful than the theorem as usually taught.

He describes the “pick two of three” formulation as misleading, because it oversimplifies the tensions between the properties. What CAP actually prohibits is a small part of the design space: perfect availability together with perfect consistency during a partition. Since partitions are rare, there is little reason to give up either property when the system is not partitioned, and a good design can have both nearly all the time. A system labeled “AP” is not a system that has abandoned consistency; it is one that has decided what to do in an uncommon situation.

Brewer’s operational framing is the part worth carrying into design reviews. A partition, pragmatically, is a time bound on communication. The essence of CAP arrives at a timeout, when a program must make what he calls the partition decision: cancel the operation, which reduces availability, or proceed with it, which risks inconsistency. That is the same timeout discussed in the companion piece on partial failure, seen from the other side. It also means the choice is made many times, at fine granularity: different subsystems can decide differently, and the decision can vary by operation, by data item, or by user.

He then gives the shape of a complete answer, which most systems only implement the first third of.

• Detect the start of a partition. This is a monitoring and timeout question, and it is where systems that “never partition” turn out to have been partitioned repeatedly without noticing.
• Enter an explicit partition mode that may limit some operations. Deciding in advance which operations are refused, which are allowed to proceed on possibly stale data, and which are recorded for later is what makes this a design rather than an accident.
• Recover when communication returns, restoring consistency and compensating for mistakes made while partitioned. Compensation is the part that is almost always missing. If a system allowed two people to claim the last item, someone has to cancel an order and apologize, and that path needs to exist before it is needed.

PACELC: the trade-off that is on all the time

CAP only says something about partitions, which Abadi argues is a significant omission, because the consistency and latency trade-off is present at all times during normal operation and therefore has a more direct effect on how a system behaves day to day.

His reformulation, PACELC, reads: if there is a partition (P), how does the system trade off availability and consistency (A and C); else (E), when running normally, how does it trade off latency (L) and consistency (C)? The second half only applies to systems that replicate data, which is the point of the previous section: replication is what creates the everyday trade-off.

The classification he gives makes the framework concrete. These are his 2012 assessments of each system’s default design, and several of these products have since made the behavior configurable, so treat the table as an illustration of the categories rather than as current product guidance.

Category	Systems named in the paper	Reading
PA/EL	Dynamo, Cassandra, Riak (default versions)	Gives up consistency for availability during a partition, and for latency in normal operation
PC/EC	VoltDB/H-Store, Megastore, BigTable and HBase	Refuses to give up consistency, and pays for it in both availability and latency
PA/EC	MongoDB	Consistent in the baseline case, but a failed or partitioned master leaves unreplicated writes to be reconciled afterwards
PC/EL	PNUTS	Trades consistency for latency in normal operation, but does not degrade further during a partition

The PC/EL row explains a common confusion. PC does not mean the system is fully consistent. It means the system does not drop below its baseline consistency level when a partition occurs; it reduces availability instead. Reading a label as a promise rather than as a description of a change in behavior is how these frameworks get misused.

When you actually need consensus

Some decisions cannot be left to converge later, because acting on two different answers is unrecoverable. Which node is the leader, which nodes are currently members of the cluster, which node owns a partition, whether a transaction committed: if two parts of the system disagree about any of these, the result is two writers who each believe they are authoritative.

Consensus algorithms exist for exactly this class of question. The Raft project describes consensus as the problem of getting multiple servers to agree on values such that, once agreed, the decision is final, and notes that this is what makes replicated state machines possible. Raft itself was published by Diego Ongaro and John Ousterhout in 2014 as a more understandable alternative to Paxos, with equivalent fault tolerance and performance.

Two properties of these algorithms drive the design consequences. They make progress while a majority of servers is available, so a cluster of five continues to operate with two failures. And that same rule means the minority stops: a group that cannot reach a majority cannot make decisions at all. This is a deliberate exchange of availability for the guarantee that there is exactly one answer, and it is why consensus-backed components are typically small and carefully sized. An odd number of members is conventional because six servers tolerate the same two failures as five while requiring one more machine.

The rule that follows is about scope, not about which data deserves consensus. Routing everything through one global consensus group is the mistake: every write then pays that group’s round trip and inherits its majority requirement, and the group’s throughput becomes the system’s throughput. Keeping a small amount of critical state there — leadership, membership, partition assignment, the commit pointer of a catalog or table format — is the familiar arrangement precisely because that state is small and rarely written.

Business data is a different question, and the answer is not automatically no. Spanner shards data across many Paxos groups, one per shard, so ordinary rows are replicated by consensus while no single group carries the whole system. That is the design point the scope rule actually implies: partition the data so that each consensus group covers a subset, and accept the cost where the guarantee is worth it. Three things decide whether it is worth it for a given dataset — what consistency the operations on it genuinely require, whether it partitions cleanly so that most transactions stay inside one group, and whether its write latency and volume can absorb a majority round trip. Data that fails the second or third test is usually better served by a weaker model plus reconciliation than by forcing it into a shape consensus can handle.

Consensus also does not remove the need for the defenses described in the piece on partial failure. A leader that has been replaced can still be running, with in-flight requests, and unaware that it lost its role. Preventing its late writes from landing is what a fencing token is for.

Clocks do not tell you what happened first

Ordering feels like it should be a solved problem, because every machine has a clock. It is not, and the reason is older than any current system.

Leslie Lamport’s 1978 paper defines the happened before relation from three rules: within one process, earlier events precede later ones; sending a message precedes receiving it; and the relation is transitive. Two events where neither happened before the other are called concurrent. The consequence he emphasizes is that this is a partial ordering of the events in a system: for many pairs of events, the causal relationships the system can observe do not determine which came first, and problems arise because people are not aware of this. Those events may well have happened in some order in the physical world. The point is that nothing inside the system recorded it.

Logical clocks extend that partial order to a total one by counting, under two conditions: a process’s clock advances between its own successive events, and a message carries the sender’s clock value so the receiver’s clock ends up ahead of it. Counting alone is not quite enough, because events in different processes can end up with the same number; a tie-break such as an arbitrary but fixed ordering of process identifiers turns the result into a genuine total order.

Three kinds of order are now in play, and keeping them apart prevents most confusion. Causal order is what happened-before captures, and it is partial. A logical total order is consistent with causal order but chooses arbitrarily between concurrent events. Real-time order is what an outside observer saw, and the system has no access to it unless something tells it.

Lamport’s own caution follows from that gap. A total order built this way can differ from the order a user perceived, and then the system behaves anomalously from that user’s point of view — his example is a request that was issued first, by a person who then told someone else, arriving second. He offers two ways out. One is to convey the external ordering information into the system explicitly, so the missing edge becomes a message the algorithm can see. The other is to bring in physical clocks, synchronized well enough that their error is smaller than the delay between related external events. Physical clocks are not the only remedy, but they are the one that scales when the external events cannot be routed through the system.

Physical clocks on separate machines drift, and the usual response is to pretend they do not. Google’s Spanner took the opposite approach, and its 2012 paper is worth knowing about even for teams who will never run it, because it quantifies the problem. Spanner’s TrueTime interface returns an interval rather than a timestamp: a call gives an earliest and a latest bound, and guarantees the true time of the call lies between them. In the production environment described in the paper, that uncertainty was a sawtooth varying from roughly 1 to 7 milliseconds over each polling interval, so about 4 milliseconds most of the time, with a 30-second poll interval and an applied drift rate of 200 microseconds per second. Keeping it under about 10 milliseconds required GPS receivers and atomic clocks in every data center.

What Spanner does with that interval is the instructive part. To guarantee that a transaction committing before another one starts gets a smaller timestamp, the coordinator holds the commit invisible until the uncertainty interval has passed, a rule the paper calls commit wait. The system deliberately waits out the clock error rather than assuming it away. Two conclusions follow for everyone else. Ordering across machines by comparing wall-clock timestamps is unreliable unless the uncertainty is known and handled, and buying reliable global ordering means paying latency for it.

For data work this shows up in a familiar form. The time an event happened, the time it was written, and the time a pipeline processed it are three different values, they arrive out of order, and choosing which one a window is based on determines what the result means. That choice is what late data and lookback handling is about, and a system that sorts by arrival time has silently chosen the one that is easiest to compute and hardest to explain.

Deciding who wins: concurrency control

Ordering tells you what happened. Concurrency control decides what to do when two operations want the same thing at once. The options differ mainly in when they detect the conflict.

Approach	How it works	Suited to	Failure mode
Pessimistic locking	Take a lock before touching the data; others wait	High contention on the same rows, short transactions	Waiting, deadlock, and a lock held by a process that has stopped responding
Optimistic concurrency	Read, compute, then commit only if nothing changed underneath; otherwise retry	Low contention, or writers that touch different data	Repeated retries and starvation when contention is higher than assumed
Multi-version reads	Readers see a consistent snapshot while writers create new versions, as in MVCC and snapshots	Mixed read and write workloads; long analytical reads	Storage for old versions, and write-write conflicts still need one of the rows above
Last writer wins	A conflict resolution rule: of the conflicting versions, the one with the highest timestamp is kept and the others are discarded	Data where discarding the losing update is acceptable, such as a cache or a status flag	The surviving write is chosen by timestamp, which may not reflect intent or real-time order, so information in the discarded write is lost without a trace
Merge on read or write	Keep both versions and resolve them by a rule, or by asking the application	Values with a meaningful merge, such as sets or counters	Rules that are wrong for some cases, and application code that must handle conflicts

Last writer wins deserves its own note because it is so often the default. It is a real conflict resolution strategy, and it does what it promises: the replicas converge, because every replica applies the same rule and reaches the same answer. Cassandra, for example, documents a last-write-wins model in which every mutation is timestamped, including deletes, and the latest version is the winning value, applied per column. What it does not promise is that the surviving write is the one anyone wanted. The timestamps come from client or coordinator clocks, so Cassandra’s documentation says plainly that its correctness depends on those clocks; where they disagree, the write that wins can be the one from the machine whose clock ran ahead rather than the one issued later. And whatever loses is gone, with no record that a choice was made. When the discarded write was a customer’s, nobody finds out until a repair process or a complaint surfaces it.

Optimistic concurrency is worth understanding in particular because it is how table formats on object storage keep concurrent writers correct. The shared pattern is that a writer prepares its new data files against a known starting version, then attempts to publish a new version, and the attempt succeeds only if nothing else has published in the meantime; otherwise it re-reads and retries. How that check is implemented varies — one format may contend for the next numbered entry in a commit log, another may make a new metadata file authoritative — and the details matter when tuning, but the guarantee is the same. That is atomic publication and optimistic concurrency working together, and it explains a common operational surprise: the approach is efficient when writers rarely collide and degrades sharply when many jobs write the same table at the same time.

Where this shows up in data and platform work

• Streaming platforms. A producer’s acknowledgement setting is a replication decision in the sense described above: waiting for a leader only, or for the replicas that are currently in step, changes both write latency and how much data a failure can lose, which is the subject of in-sync replicas and acknowledgements.
• Analytical tables. Table formats give concurrent writers isolation through snapshots and optimistic commits, which is why a table can be read consistently while a load is running, and why two concurrent loads to the same partition can make each other retry.
• Change capture. A database’s write-ahead log, the mechanism behind write-ahead logging, records changes in the order they were written, which interleaves concurrent transactions and includes work that was later rolled back. What a downstream consumer sees is not the log but a decoder’s output, and the ordering it offers — typically each transaction’s changes emitted together once it commits — is a property of that decoder and its configuration rather than of the log. Whatever ordering it does provide stops at the boundary of that one database. Merging streams from two sources restores the partial-order problem, and events from different systems are frequently concurrent in Lamport’s sense.
• Reporting on replicas. Pointing dashboards at a read replica is usually correct and occasionally wrong: the replica reflects the source as of some moment in the past, so a figure compared against an operational screen is being read from a different point in time, and how far the two numbers diverge depends on what changed during the lag. Finance reconciliations are where that gets noticed.
• Ingestion. Delivery is at-least-once in most pipelines, so consumers need idempotency, and the ordering guarantee usually holds only within a partition or key rather than across the whole stream, which matters as soon as related records are keyed differently. The same constraint shapes database ingestion and change data capture.

Across all of these, the constant is that reconciliation is not an admission of failure. In a system where copies are allowed to disagree for a while, something has to establish whether they converged, and that check has to be designed rather than assumed.

How thorough the check needs to be depends on what a mismatch would cost. Comparing totals as of a common point in time is cheap and catches gross divergence, but it proves less than it appears to: a missing row and a duplicated one can cancel out, leaving equal totals over different data. Stronger checks compare row counts, then keys, then values, at increasing cost. Storage systems that expect replicas to diverge build this in — Cassandra’s repair has replicas compute hierarchical hash trees, Merkle trees, over their data and compare them to identify which ranges differ, so that only the mismatched ranges are transferred. Whatever the method, the result should be reported as what it is: matching totals is evidence about totals, not proof that two datasets are identical.

A short checklist for a replicated system

Question	If the answer is unclear
Why does this data have more than one copy: failure, read volume, or distance?	The replication design is being chosen without knowing what it is for
Where do reads go, and what is the worst staleness a reader can see?	Stale reads appear as unreproducible bugs and disputed numbers
Which consistency model does each important operation need, stated per operation?	The system pays for the strongest requirement everywhere, or fails the strictest one everywhere
What happens during a partition: which operations are refused, and which proceed?	The behavior exists anyway, chosen by whichever timeout fires first
How are the mistakes made during a partition detected and compensated afterwards?	Double bookings and lost updates are discovered by customers
Which state genuinely requires consensus, and how large is that group?	Either split-brain writes, or consensus latency paid on data that did not need it
What orders events: event time, arrival time, or a machine clock?	Results change when a job is rerun, and nobody can explain why
What happens when two writers conflict, and who finds out?	Last writer wins by default, and the losing write is gone silently

Questions to explore further

• For your most-used read path, what is the current replication lag distribution, and does anyone alert on it?
• Which of your operations would cause real loss if two of them ran concurrently on different replicas, and what prevents that today?
• If your primary region were cut off for ten minutes, which writes would you want refused and which would you want accepted and reconciled later? Is that written down?
• Where in your pipelines does ordering depend on a machine clock rather than on an event field or a log position?

References

All sources were checked on September 15, 2026. The system classifications and clock measurements cited are from papers published in 2012 and describe those systems as they were then.

• Eric Brewer, CAP Twelve Years Later: How the “Rules” Have Changed (IEEE Computer, 2012)
• Daniel J. Abadi, Consistency Tradeoffs in Modern Distributed Database System Design (IEEE Computer, 2012, PDF)
• Jepsen, Consistency Models
• Leslie Lamport, Time, Clocks, and the Ordering of Events in a Distributed System (Communications of the ACM, 1978, PDF)
• James C. Corbett et al., Spanner: Google’s Globally-Distributed Database (OSDI 2012)
• The Raft Consensus Algorithm (Diego Ongaro and John Ousterhout, 2014)
• Apache Cassandra Documentation, Dynamo (conflict resolution and anti-entropy repair)