Two Versions at Once: Progressive Release and Change Compatibility

A release goes badly and someone presses the rollback button. It works: the previous version is redeployed in ninety seconds, exactly as rehearsed. The previous version then fails to start, because the migration that shipped with the new release dropped a column it reads. The outage is now longer than it would have been without the rollback, and the only way out is forward.

Everything in this article follows from why that happened. Canary releases, blue-green cutovers, feature flags, and rollback are usually presented as four separate techniques with four separate tools. They share one precondition: the old version and the new version have to be able to run at the same time. And that capability is not a property of your deployment tooling. It is a property of your schema, your interfaces, and your data — which means most of the work of release management happens before anything is deployed.

The pattern literature says this outright. Canary and blue-green deployment “are applications of the parallel change pattern where you have both old and new versions of the code deployed side by side.” Once you see the techniques as one pattern at different layers, the sequencing questions answer themselves. Descriptions below follow the parallel change, blue-green, and feature toggle write-ups on martinfowler.com and DORA’s metric definitions, checked in September 2026.

Deploying is not releasing

The first separation to make is between putting code on a server and letting users reach the behaviour. A feature flag is the mechanism, and the published framing is precise about what it buys: release toggles “allow incomplete and un-tested codepaths to be shipped to production as latent code which may never be turned on,” and using them this way is “the most common way to implement the Continuous Delivery principle of ‘separating [feature] release from deployment.'"

That separation is what makes small, frequent deployment compatible with features that take three weeks to finish. It is also what makes the phrase "feature flag" ambiguous, because four different things wear the name and they have different lifespans and different failure modes.

Kind	Purpose	Expected lifetime
Release	Ship unfinished code dark; turn it on when ready	"Transitionary by nature," and they "should generally not stick around much longer than a week or two"
Experiment	A/B or multivariate testing by cohort	"Long enough to generate statistically significant results" — and no longer
Ops	Degrade or disable behaviour under load; kill switches	Short for a new feature; a few may stay indefinitely
Permissioning	Entitlements — premium, alpha, beta	Possibly years, "at the scale of multiple years"

Conflating these is the most common way flag systems become unmanageable, because a release toggle that outlives its two weeks stops being a delivery tool and becomes a permanent branch in the code. The long-lived ops variety is a legitimate exception, described as allowing "operators of production environments to gracefully degrade non-vital system functionality when the system is enduring unusually high load" — and read that way a kill switch is a manually operated form of the same idea as automatic throttling, deliberately placed under a human's control.

Two costs deserve to be stated before anyone adopts flags enthusiastically.

Testing multiplies. The problem is structural rather than a matter of discipline: "in order to validate all codepaths which may end up live in production we must perform test our artifact in both states: with the toggle flipped On and flipped Off," and "with multiple toggles in play we have a combinatoric explosion of possible toggle states." You cannot test every combination, so you have to decide which combinations are real — usually the current production configuration and the one you are moving to — and state that decision rather than leaving it implicit.

Flags are inventory. "Savvy teams view the Feature Toggles in their codebase as inventory which comes with a carrying cost and seek to keep that inventory as low as possible." The documented tactics are all about forcing removal: a removal task added to the backlog when the toggle is created, expiration dates, or "time bombs" that fail a test once a flag has outlived its window. Any of them beats intending to clean up later, and the cautionary example offered — Knight Capital — is a reminder that stale flag state is not merely untidy.

Two shapes of progressive deployment

With release decoupled from deployment, the deployment itself can be made gradual. Two shapes cover most cases and they answer different questions.

Blue-green deployment keeps "two production environments, as identical as possible," with one live. You deploy and do final testing on the idle one, then "switch the router so that all incoming requests go to the green environment — the blue one is now idle." Its advertised benefit is reversal speed: it "gives you a rapid way to rollback — if anything goes wrong you switch the router back to your blue environment." A side effect worth collecting is that the mechanism resembles a hot standby, so "this allows you to test your disaster-recovery procedure on every release."

A canary deployment instead exposes a limited share of real production work to the candidate while the current version handles the rest, so that evidence accumulates before the blast radius grows. The two are not competitors: blue-green optimizes for a fast, total switch, and canary optimizes for learning under partial exposure. A team that wants both runs a canary and keeps the previous version warm.

Both have the same honest caveat, stated in the blue-green write-up and applying just as much to a canary: "there's still the issue of dealing with missed transactions while the green environment was live." Switching traffic back does not un-write what the new version wrote. Whether that residue is harmless or is the actual incident depends on what the new version changed — which is the compatibility question again, arriving from a different direction.

One distinction to keep straight when a flag is used for gradual exposure: "a canary released feature is exposed to a randomly selected cohort of users while a Champagne Brunch feature is exposed to a specific set of users." Both are gradual, and the difference is what you can conclude from each — which is not the same as one being a canary and the other not.

Two purposes get conflated here, and separating them is the practical point.

• Safety observation. Is the new version breaking anything? A canary answers this by comparing the candidate against the current version on the same kind of work. Non-random cohorts are perfectly normal here — starting with one region, one cluster, internal users, or a small set of consenting customers is standard practice, and the comparison is against a control running the old version rather than against a hypothesis.
• Effect estimation. Does the change improve the metric, and by how much? That is an experiment, and what licenses the causal reading is random assignment within whatever group you are studying — in the standard design "the levels of the primary factor are randomly assigned to the experimental units." Note that this is a separate question from how the group was chosen. Giving the new version to a hand-picked set and comparing them with everyone else supports no causal reading, because whatever made them selectable may also explain the difference. But randomizing old and new inside that same hand-picked set is a valid experiment about that set. What selection costs you is generalization, not the estimate.

And randomization alone does not make a canary informative. Google's SRE guidance is explicit that "a representative canary process requires decisions across many dimensions" — size and duration ("it should be sizeable and last long enough to be representative of the overall deployment"; terminating after "just a handful of queries doesn't provide a useful signal"), traffic volume, and even time of day, since "performance defects typically manifest only under heavy load" and therefore "deploying at an off-peak time likely wouldn't trigger performance-related defects." The same guidance advises "running only one canary deployment at a time," partly because overlapping canaries risk "signal contamination."

So state three things separately: how the group was chosen, how versions were assigned inside it, and which population you intend the answer to apply to. A cohort that receives the new version wholesale is a legitimate canary and no experiment; the same cohort split randomly between versions is an experiment about the cohort, and still needs enough traffic, a clean observation window and no interference between the two arms before the number means anything. A random cohort that is too small or observed at the wrong hour is a poor canary whatever its sampling. And where the question is how the new version behaves under production traffic without affecting anyone, that is shadow testing rather than either of these.

Expand, migrate, contract

Here is the pattern all of the above depends on. Expand and contract — also called parallel change — is "a pattern to implement backward-incompatible changes to an interface in a safe manner, by breaking the change into three distinct phases: expand, migrate, and contract."

1. Expand. "You augment the interface to support both the old and the new versions." Nothing is removed; existing consumers are untouched.
2. Migrate. "You update all clients using the old version to the new version." This "can be done incrementally and, in the case of external clients, this will be the longest phase" — which is the sentence to quote when someone asks why the old field is still there.
3. Contract. "Once all usages have been migrated to the new version, you perform the contract phase to remove the old version."

The property that matters for delivery is stated directly: the pattern "is particularly useful when practicing ContinuousDelivery because it allows your code to be released in any of these three phases." Every intermediate state is shippable, which is exactly what lets a change cross several small deployments instead of one large one.

For databases this is not an analogy but the standard method: database refactoring is "a key component to evolutionary database design. Most database refactorings follow the parallel change pattern, where the migrate phase is the transition period between the original and the new schema, until all database access code has been updated to work with the new schema."

And the blue-green write-up gives the operational recipe, including the words that explain the opening failure: "the trick is to separate the deployment of schema changes from application upgrades. So first apply a database refactoring to change the schema to support both the new and old version of the application, deploy that, check everything is working fine so you have a rollback point, then deploy the new version of the application. (And when the upgrade has bedded down remove the database support for the old version.)"

Which yields a rule worth enforcing mechanically, because it is easy to state and easy to violate by accident: never ship a schema change and the code that requires it in the same deployment. The steps below are ordered, and what governs each one is an entry condition rather than an elapsed number of days.

Step	Schema	Code	Entry condition
1 — expand	Add the new column; keep the old. New column nullable or defaulted	Unchanged	None. The old code never sees the addition
2 — write both	Unchanged	Every writer writes old and new; readers still read old	Every writer is deployed, including batch jobs, admin tools, and anything outside the main service
3 — backfill	Fill the new column for existing rows	Unchanged	Step 2 is complete everywhere, and the backfill itself cannot overwrite a concurrent write — see below
4 — switch reads	Unchanged	Readers read the new column; writers still write both	Backfill finished and verified — the two columns agree for every row
5 — stop writing the old	Unchanged	Writers write only the new column	You have decided you will no longer roll back to a version that reads the old column
6 — contract	Drop the old column	No remaining reference to it anywhere	Every writer and job from step 5 is confirmed deployed. This step is irreversible

Six steps rather than three looks like ceremony until you see what each of the extra ones prevents. Three hazards sit between the phases, and all three are invisible in a passing test suite.

A writer you forgot will silently desynchronize the columns. If the old version still writes only the old column while the new version reads the new one, then any row the old version touches after the backfill has a stale new column and a current old one. Nothing errors; the reader just returns the value from before that write. This is why "write both" has to be complete before the backfill and why "every writer" includes the ones that are not the application — a nightly job, an admin console, a support script, a replication process.

The backfill itself can overwrite live writes, and dual writing does not prevent it. This is the part that is usually left as a warning rather than a condition, so here is the interleaving. The backfill selects a row and reads old = 10. A normal writer then updates that row, setting both columns to 20, and commits. The backfill, still holding the value it read, writes 10 into the new column. The row is now (old, new) = (20, 10), and it will stay that way until something touches it again. Every writer being correct did not help, because the loss happened inside the backfill. Nor does putting the read and the write in one transaction fix it by itself: at the default isolation level a plain SELECT "sees only data committed before the query began," so the stale value is still what gets written.

What makes it safe is to stop carrying the value outside the database. For a copy like this one, the backfill should be a single statement that reads and writes in the same operation — the shape of UPDATE t SET new = old WHERE new IS NULL, applied in batches by primary key. PostgreSQL documents why that is sufficient: when a concurrent transaction has already updated a row the second updater "will wait for the first updating transaction to commit or roll back," and then "it will attempt to apply its operation to the updated version of the row," with the WHERE clause "re-evaluated to see if the updated version of the row still matches the search condition." So the statement copies the current old, and a row a dual writer has already filled fails the condition and is skipped. The documentation gives the same reasoning for a balance transfer, where "letting it see the updated version of the row does not create any troublesome inconsistency."

Where the new value cannot be computed in SQL — a format change, a call to another service — one statement is not available, and you need one of three explicit contracts instead. Lock the row while you work on it (SELECT … FOR UPDATE, which returns "the updated version of the row" after waiting). Or make the write conditional on what you read, in the shape of UPDATE … WHERE id = ? AND old = ?, and re-read and retry when it affects no rows. Or let the database keep the columns in step for you: GitLab's migration guidance takes that route for a column rename, where a regular migration "is used to create a new column with a temporary name along with setting up some triggers to keep data in sync," with cleanup deferred to a later post-deployment migration. If you choose triggers, say which direction they synchronize and confirm that a trigger firing on a live write beats a slower backfill writing an older value — otherwise you have moved the race rather than closed it.

The distinction to hold on to: batching and transactions control how long the backfill runs and how much it locks; they do not decide whether a concurrent write can be lost. That is decided by whether the value being written was read inside the same atomic operation, or is checked before it lands.

Dropping the column and removing the code that writes it are two deployments, not one. If the DROP lands while instances still writing both columns are running, those instances fail on every write. And the window in which you need dual writes is not "while two versions are live" — it is as long as you might still roll back to a version that reads the old column. That window closes when you decide it does, which is what makes step 5 a decision rather than a cleanup.

GitLab's own procedure spreads a column drop across three releases — ignore the column, drop it, then remove the ignore rule — and states the reason plainly: "dropping a column is a destructive operation that can't be rolled back easily." It also names a detail that catches frameworks specifically: the application "caches the tables schema when it boots even if the columns are not referenced," so code that never mentions the column can still break when it disappears. The lesson to carry across stacks is that "no code references it" is a claim to verify at runtime, not to infer by reading the diff.

For a large table the migrate phase is long enough to be its own operational state, and the guidance says so: if you are renaming a large table, "carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet," because the system can run that way "for a significant amount of time." Plan for the intermediate state to be the normal state for a while.

The same three phases apply to an API and to a published data model, with the migrate phase's duration set by who the consumers are. Renaming a field for an internal caller you deploy yourself is a short migration; doing it for external clients is measured in quarters, which is why the removal step belongs to a stated process — see deprecation for how to announce it and breaking change for deciding whether it needs one at all. For an API the expand phase is also where tolerance helps: "following Postel's Law is a good technique to avoid consumers breaking when the payload is expanded." Where a version number is genuinely needed, semantic versioning gives the signal its meaning; for event and analytical schemas the compatibility direction is the governing choice, covered under schema evolution and compatibility, and the consumer-facing side of changing a published model is worked through in Changing a Data Model Without Breaking the People Using It.

When rollback is not available

Rollback is redeploying a retained version, and it is "useful only if the chosen version can still operate correctly with current data, interfaces, and access requirements." That condition fails more often than teams expect, and it fails in four recognizable ways.

• The schema moved past it. A column dropped, a constraint tightened, a type narrowed — the old code cannot run against the current database. This is the opening failure, and the expand-migrate-contract ordering is what prevents it.
• New data exists that the old version cannot read. Rows written with a new enum value, a new document shape, a field the old parser rejects. The data is valid; the old code is not prepared for it.
• Something irreversible already happened. Messages published, emails sent, a payment captured, a file handed to a partner. Redeploying does not retract any of it — but note that this is a different problem from the three around it, and it is the one most often misfiled. See the paragraph below.
• The artifact is gone. Retention expired, or the tag was overwritten, so there is nothing to roll back to — which is why artifact retention is a recovery decision, as covered under build artifact.

The third item needs separating out, because "we cannot roll back" is used for three different situations and only two of them block a rollback.

Question	What it means	When irreversible effects have occurred
Can the old version run?	Deploy the retained artifact and have it work against current data and interfaces	Usually yes — sent emails do not prevent the previous version from running
Can what happened be undone?	Retract the effect itself	No — this is the actual irreversibility, and code deployment was never the mechanism for it
Can further damage be stopped?	Prevent the next occurrence	Usually yes — roll back, or turn the flag off, and the sending stops

So an already-sent batch of wrong emails is a reason to act immediately rather than a reason rollback is off the table: stopping the sender is available, and the emails already out need compensation — a correction notice, a reversal entry, a call to the partner — which is a separate workstream from the deployment decision. Filing it under "rollback unavailable" produces exactly the wrong response, which is to leave the new version running while people discuss remediation.

Where rollback genuinely is unavailable, the remaining option is to fix forward: ship a corrective change through the normal path. Notice that the standard metrics are deliberately neutral about which you choose — failed deployment recovery time is "the time it takes to recover from a deployment that fails and requires immediate intervention," and a change failure's intervention is described as "likely resulting in a rollback of the changes or a 'hotfix' to quickly remediate any issues." What is measured is how long the problem lasted, not which manoeuvre you used.

So the decision is about expected time to safety rather than about preference, and it comes down to three questions. Is the previous version actually able to run right now, against current data? If not, rollback is off the table regardless of how it feels. If it is, can your pipeline deliver a fix faster than the rollback plus the re-fix you will need anyway — which for a pipeline measured in minutes is often yes, and for one measured in hours is almost never. And is the failure understood? Rolling back a symptom you do not understand is the right move; fixing forward on a guess is how a bad release becomes two.

Whichever you choose, the deciding factor is usually not the tooling but whether you can tell the release is failing at all. A SLO with a stated threshold turns "does this look worse?" into a decision anyone on call can make at three in the morning, and a canary is only as useful as the signal you compare against.

A release checklist that follows from all this

1. Can the previous version run against the state this release leaves behind? If not, split the release until it can.
2. Is any schema change in its own deployment, ahead of the code that needs it? Expand first, contract last, days apart.
3. Which toggle states will exist in production, and have those been tested? Not all combinations — the ones that are real.
4. What signal says this release is failing, and what is its threshold? Decided before the deployment, not during it.
5. Who decides to reverse, and is the target artifact still there? Both answers should exist before you need them.
6. When does the contract phase happen, and whose task is it? Unfinished contractions and expired flags are the same debt with different names.

The sixth item is the one that quietly decides whether any of this stays affordable. Expand is fun, migrate is work, and contract is nobody's priority — so systems accumulate half-finished parallel changes and a codebase of flags whose purpose has been forgotten, until every change has to be compatible with three historical shapes of the same data. A release gate can enforce the cheap version of this by refusing to pass a change that adds a flag with no removal date.

And the summary worth carrying: continuous delivery is often described as a tooling achievement, and the tooling is the smaller half. What makes releases safe is that at every moment two versions of your system could both be correct — a property you design into schemas and interfaces, deployment after deployment, and one that no amount of routing cleverness will supply on your behalf. Promotion moves the artifact; compatibility is what makes the movement reversible, and the promotion machinery it rides on is covered under environment promotion.

References: Danilo Sato, Parallel Change (martinfowler.com, 2014); Martin Fowler, Blue Green Deployment (2010, updated 2015); Pete Hodgson, Feature Toggles (aka Feature Flags) (martinfowler.com); DORA, DORA's software delivery performance metrics; GitLab, Avoiding downtime in migrations; PostgreSQL documentation, Transaction Isolation; NIST/SEMATECH e-Handbook, Completely randomized designs; Google, The Site Reliability Workbook, “Canarying Releases”. All checked September 2026.