Configuration Drift

Configuration drift is the gap between the state a system is declared to have and the state it actually has. The word is also used in machine learning for a completely different phenomenon — changes in the distribution of data or in the relationship being modelled — and the two have nothing to do with each other. This entry is about infrastructure: the server that no longer matches its definition.

There is no standards body definition of the term; it is the name infrastructure tooling gives to what its drift detection features look for.

Which fixes the baseline, and the baseline is where the word gets misused. Drift is measured against that system’s own declaration — the definition its team maintains and applies. It is not distance from a central standard. A team that took a shared template, adapted it deliberately, and runs exactly what its adapted definition says has not drifted, even if the result no longer resembles the template it came from. That situation is worth discussing, but it is a question about whether the standard or the adaptation should change, and it is detected differently: drift is found by comparing a system to its own definition, whereas divergence from a standard requires knowing what the current standard is and which version each team adopted.

Three places the gap opens

Most discussion treats drift as one thing — code versus reality — which makes the confusing cases hard to explain. There are three states, and the gaps between them have different causes and different fixes.

The middle row is the one people forget, and it explains an otherwise puzzling situation: a fleet can be perfectly consistent with its images and still not match what the repository says. This is why image pipelines have a validation step at all — a managed pipeline “ensures that all of the customizations work as expected” before the image is used, precisely because a recipe and its output are not the same thing.

So the diagnosis order that saves time is bottom-up: compare the running host against the artifact it came from, then the artifact against the declaration. Jumping from a symptom straight to the code is how an afternoon disappears.

Why fixing the host is not fixing the problem

The tempting response to a drifted host is to correct it, confirm the service is healthy, and close the ticket. The next server built from the same artifact has the same problem, which is how a single incident becomes a recurring one that nobody recognizes as the same issue.

Which order to use depends on where the defect is, and that has to be decided before anything is edited — otherwise you either leave a recurring problem or promote a mistake into the standard.

• The declaration or the artifact is wrong. Change the declaration, rebuild, replace the host. Fixing only the host leaves the next machine built from that artifact with the same problem.
• The live host was changed without authorization and the declaration is correct. Restore the host to the declaration and leave the declaration alone. Folding the live state back into it here is how an incident workaround becomes the fleet standard.
• The change was a legitimate emergency fix. It belongs in the declaration after review — and until it is there, recorded somewhere the next rebuild will pick it up, with a note saying why. An undocumented manual fix is a time bomb whose fuse is the next deployment.

Two habits keep the gap small, and the second is more effective than the first.

• Detect on a schedule. Something should periodically compare live state against the declaration and report the differences. Drift found by a weekly report is cheap; drift found while diagnosing an outage is expensive, because it arrives as a surprise about the thing you were relying on.
• Reduce interactive access. Drift is created by people changing hosts by hand, so a fleet where many people can do that will drift regardless of tooling. Narrowing who can log in is the root control — a least privilege decision — and an audit log is what lets you find which change caused a difference after the fact.

Replacement rather than repair is what makes both habits sustainable, since a host that lives for weeks has less time to accumulate differences than one that lives for years — which is the practical argument for the golden image approach over patching in place.

Not every difference is drift

Chasing every discrepancy produces noise and, eventually, a report nobody reads. Three kinds of difference are legitimate and should be excluded deliberately rather than argued about each time.

• Things the declaration never claimed. Runtime state — process lists, cache contents, log volume — is not drift. Deciding what is in scope is part of writing the declaration.
• Values that are supposed to vary. Instance identifiers, addresses, generated names. Comparing them creates permanent false positives.
• Changes made by the platform. Autoscaling counts, provider-managed tags, rotated certificates. These are correct and will be reported unless excluded.

What is left after those exclusions is the set worth acting on, and it should be small enough that each entry gets a decision. A drift report with two hundred entries is a report that will be ignored, which is worse than not having one — it converts a known gap into a documented and unexamined one.

How drift fits with support dates, patching, and the limits of standardization is worked through in The Server That Outlived Its Support Window.

Reference: AWS, What is EC2 Image Builder?.