Configuration Drift

Configuration drift is the gap between the state a system is declared to have and the state it actually has. The word drift is also used in machine learning, where data drift and concept drift mean changes in the distribution of data or in the relationship being modeled. Those are different phenomena from configuration drift. This entry is about infrastructure: the server that no longer matches its definition.

Infrastructure tools define drift against the state they manage. For example, CloudFormation compares supported resource properties with template expectations. State which baseline and properties a drift check covers; one tool’s result is not a complete assessment of the host.

Within that managed scope, drift is measured against that system’s own declaration — the definition its team maintains and applies. It is not distance from a central standard. A team that took a shared template, adapted it deliberately, and runs exactly what its adapted definition says has not drifted, even if the result no longer resembles the template it came from. That situation is worth discussing, but it is a question about whether the standard or the adaptation should change, and it is detected differently: drift is found by comparing a system to its own definition, whereas divergence from a standard requires knowing what the current standard is and which version each team adopted.

Locate the gap between declaration, artifact, and host

Compare the intended declaration, the built artifact, and the live host including approved launch configuration. These states can differ for different reasons, so the location of the defect determines the correction.

State	How it diverges	Fix
What is declared — code, templates, image recipes in version control	It does not diverge; it is the reference. But it can be wrong, or incomplete about things it never described	Review, and widen coverage to what matters
What the artifact contains — the image actually built	A base image moved, a package version floated, a build step failed without stopping the build	Pin versions, and validate the built artifact rather than trusting the recipe
What is running — the live host	Manual edits, automatic updates, another controller, or external configuration can alter it	Compare with the intended state including approved boot settings; correct the responsible change path

An image can differ from its intended build inputs even when every host matches that image. AWS Image Builder separates customization, validation, and testing; the selected checks still need to cover the properties that matter.

Comparing the host against its artifact and approved launch configuration, then the artifact against its declaration, can help locate the defect. Incident evidence may justify starting elsewhere. An expected boot-time difference is not automatically a defect in the image.

Correct the source of the difference

If the artifact contains a defect, repairing only one host leaves the next instance exposed to the same defect. If the artifact and declaration are correct and only the live host was changed incorrectly, restoring that host may be the right correction. Investigate the change path as well as the observed value.

Which correction to use depends on where the defect is, and that has to be decided before anything is edited — otherwise you either leave a recurring problem or promote a mistake into the standard.

• The declaration or artifact is wrong. Correct an incorrect declaration. If the declaration is right but the output is wrong, fix the build process, input selection, or validation instead. Rebuild and replace affected hosts; do not change the intended state merely to match a defective artifact.
• The live host was changed without authorization and the declaration is correct. Restore the host to the declaration and leave the declaration alone. Folding the live state back into it here is how an incident workaround becomes the fleet standard.
• The change was an authorized emergency fix. Record its owner, reason, and temporary scope. Review whether it should become part of the declaration or be reverted. Make the interim decision visible to subsequent rebuilds so they do not silently remove a fix still needed.

Detection and control of change paths work together:

• Detect against declared expectations. Compare relevant live properties on a schedule and after important changes, allowing for expected dynamic values. Triage differences by their effect on security, service behavior, and recoverability.
• Control who and what can change state. Interactive users, agents, scripts, controllers, and external configuration services can all create differences. Apply least privilege to those paths. An audit log helps trace changes only where the relevant actions and actors are recorded.

Replacement can limit accumulated host-specific changes, which is a reason to consider the golden image approach for suitable workloads. In-place configuration convergence and verification can also control drift. Neither method eliminates the need to define and observe the intended state.

Not every difference is drift

Define permitted variation precisely instead of treating all differences as errors or excluding broad categories. Keep the changing value separate from the conditions that must remain true.

• Properties outside the declared scope. Ordinary cache contents or log volume may not belong in a configuration comparison. An omitted security setting or a property the tool cannot inspect is an observation gap, not evidence of compliance. Add a suitable check where that property matters.
• Values designed to vary. Instance identifiers or certificate serial numbers may change legitimately. Check the conditions around them, such as authorized membership, certificate issuer, names, and remaining validity, instead of requiring identical values everywhere.
• Controller-managed changes. Document which controller owns each property and its allowed range or rule. Autoscaling within approved limits can be expected; an incorrect scaling bound or unauthorized tag change still needs action. Do not exclude a property solely because a platform changed it.

Record the comparison scope and distinguish violations, expected variation, and unchecked properties. Assign risk-based priorities and owners to actionable differences. A large report calls for grouping and triage, not for assuming that everything remaining is either safe or too numerous to investigate.

How drift fits with support dates, patching, and the limits of standardization is worked through in The Server That Outlived Its Support Window.

References

• AWS, What is EC2 Image Builder?
• AWS CloudFormation — Detect unmanaged configuration changes