Infrastructure as Code: State and Drift

A reviewed pull request changes one line of infrastructure code. A pipeline runs, and ten minutes later a database that held production data is gone, because the change renamed the resource and the tool understood the rename as “delete the old one, create a new one.” Nothing malfunctioned. The tool did exactly what its inputs said. The inputs were the code, the tool’s record of what it had already built, and the live environment, and the reviewer had only looked at the first.

This post is about those three inputs and how they disagree. It uses Terraform as the worked tool, because its documentation is explicit about state, plans, and locking, and because it runs locally without a cloud account for most of what we need. After reading it you should be able to explain what the state record is and where to keep it, read a plan as a comparison and not as a promise, describe how locking prevents concurrent writers, tell drift from an intentional change and decide which side to correct, keep secrets out of the places Terraform writes, test and promote a change across environments, and describe what rollback does and does not mean for infrastructure.

This post assumes the account, identity, and permission material from Cloud Accounts, Identity, and Landing Zones: that an account is an isolation boundary, that automation should obtain short-lived credentials through federation instead of holding long-lived keys, and that policy layers combine into one decision. Here that background decides where state lives, who may read it, and which credentials a pipeline uses. Everything about state, plans, and drift is explained from first principles below.

The local setup behind the examples

Most observations below come from a deliberately small local setup that needs no cloud account.

• Environment. Terraform 1.15.5 on macOS. The resource type is terraform_data, which comes from Terraform’s built-in provider, so terraform init downloads nothing and no cloud credentials are involved. The Terraform documentation describes terraform_data as a resource that implements the standard resource lifecycle but takes no other actions, with an input argument stored in state and an output attribute that reflects it.
• Input. The base configuration shown below, kept as main.tf in an otherwise empty directory. The password is a placeholder, not a secret.
• Observed result. terraform apply -auto-approve ended with Apply complete! Resources: 2 added, 0 changed, 0 destroyed., and a file named terraform.tfstate appeared.
• Limit of the sandbox. terraform_data cannot observe anything outside Terraform, so it cannot demonstrate drift from a real API. The drift section uses a table and a model instead of a live demonstration, and says so.

terraform {
  required_version = ">= 1.10"
}

variable "db_password" {
  type      = string
  sensitive = true
  default   = "example-not-a-real-secret"
}

variable "replicas" {
  type    = number
  default = 2
}

resource "terraform_data" "service" {
  input = {
    name     = "orders"
    replicas = var.replicas
  }
}

resource "terraform_data" "database" {
  input = {
    name     = "orders-db"
    password = var.db_password
  }
}

output "service" {
  value = terraform_data.service.output
}

The later examples modify this configuration or copies of it, after one terraform init and one terraform apply -auto-approve.

Desired state and state storage

Declared intent and reconciliation

Infrastructure as code defines infrastructure in versioned, machine-readable files instead of by hand. The files in this post are declarative: they describe the desired state, meaning the resources that should exist and their properties, and a tool works out the steps to get there. That differs from a script of commands, where the author states the steps and the result depends on the starting point.

A declarative tool needs three things to do its job: the declaration, knowledge of what currently exists, and a way to connect the two. Connecting them is the hard part. A declaration says “a database named orders-db,” and the cloud provider says “database identifier db-7f3a exists.” Something has to record that the first refers to the second, or the tool cannot tell an update from a creation.

That record is the state. Terraform’s documentation puts its purpose this way: the primary purpose of state is to store bindings between objects in a remote system and resource instances declared in your configuration. The documentation also says state tracks metadata and improves performance for large infrastructures by avoiding the need to query every resource before each operation. The glossary entry on the infrastructure state file describes the same record in more depth, including recovery with imports.

A property worth stating now is idempotency: applying the same configuration to an environment that already matches it should change nothing. In the local setup, a second terraform apply -auto-approve with no changes reported Apply complete! Resources: 0 added, 0 changed, 0 destroyed. and did not modify the resources. Idempotency is about repeating an operation, not about undoing it: a delete that you run twice has still deleted the data once. Keep that distinction in mind in the rollback section.

What the state holds

State is a JSON document. A small script run after the first apply prints its top-level structure and one resource.

import json
s = json.load(open("terraform.tfstate"))
print({k: v for k, v in s.items() if k != "resources"})
r = s["resources"][0]
print(r["type"], r["name"], r["provider"])
print(sorted(r["instances"][0].keys()))
print(sorted(r["instances"][0]["attributes"].keys()))

The output with this configuration shows these facts. The document has a format version of 4, the terraform_version that wrote it, a serial, a lineage identifier, and outputs. In this run the serial was 2 after the first apply; treat that number as an observation of this run only. Each resource has a type, a name, the provider that manages it, and a list of instances. Each instance has attributes, here id, input, output, and triggers_replace, along with a sensitive_attributes list that records which attributes were marked sensitive.

The id is the binding: it is the identifier the provider assigned, which the tool uses to find the object next time. The serial increases as state is written, and the lineage identifies the history that the state belongs to. Terraform’s documentation on pushing state mentions safeguards against lineage mismatch and serial conflicts, which is a hint at their purpose: they let the tool refuse to overwrite a state that is not an ancestor of the one it holds. Only the presence and values of these fields were confirmed, not their full semantics.

Two consequences follow. First, state is derived from reality yet authoritative about it. If the binding is lost, the tool does not know the database exists. The glossary entry’s worked case is a plan that unexpectedly proposes creating a database that already exists, which usually means the wrong backend or workspace was selected and not that the database disappeared. Second, state holds attribute values, including whatever you passed in. That includes the placeholder password in the sandbox, as the secrets section shows.

Where state lives

By default Terraform keeps state in a local file, using what its documentation calls the local backend. That is fine for a sandbox and wrong for a team: nobody else can see it, it can be lost with a laptop, and two people can run the tool at the same time against different copies. Terraform’s documentation recommends storing state in HCP Terraform or a remote backend instead, so teams can collaborate and avoid data loss.

A backend is the component that stores state and provides the API for locking. Backends differ, and not all support locking, so check the one you pick. The S3 backend is a common choice on AWS, and its documentation covers several points that matter in practice.

• Locking. Setting use_lockfile to true enables S3-based locking. The documentation says DynamoDB-based locking is deprecated and will be removed in a future minor version, so new setups should use the S3 mechanism. The documentation lists extra permissions when use_lockfile is on: get, put, and delete on the lock file, which is named with a .tflock suffix.
• Recovery. The documentation strongly recommends enabling bucket versioning, to allow state recovery after accidental deletion and human error.
• Encryption. encrypt turns on server-side encryption, and kms_key_id selects a KMS key, which requires the identity to have kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey.
• Permissions. The core permissions are list on the bucket and get and put on the state object’s path.

A backend block looks like this. The bucket, key, and region are placeholders, and it was not run, because it needs an S3 bucket.

terraform {
  backend "s3" {
    bucket       = "example-org-terraform-state"
    key          = "orders/prod/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

Because state can contain secrets, the permission to read state is a permission to read those secrets. Treat the state bucket as sensitive: restrict who can read it, encrypt it, log access, and keep the bucket out of the account that workload teams administer. This is least privilege applied to a file that people tend to treat as plumbing. It is also where the previous post’s account design pays off: a state bucket in a dedicated account, reachable only through a federated role for the pipeline, is a smaller target than a bucket in a shared workload account.

How many states you keep is a design choice with a blast-radius consequence. One state for everything is simple to plan and makes every change risk everything, and it forces unrelated teams to wait on one lock. Many small states limit each change to its own scope and make cross-state references and ordering your problem.

Granularity	What it gives you	What it costs
One state for a whole organization	One plan shows everything	Every change can affect everything; one lock serializes all teams; a corrupted state hurts all
One state per environment and component	A change touches a bounded set of resources; locks rarely collide; permissions can be per state	Cross-state dependencies need explicit outputs or data lookups; changes spanning components need ordering
One state per resource	Smallest radius	Management overhead and dependency tangles dominate

The usual compromise is one state per environment and per component that changes at a similar rate, which is a blast radius decision more than a Terraform one. Terraform’s CLI workspaces are not a substitute for this split. The documentation says workspaces keep multiple state files for one configuration and backend, and it says workspaces are not appropriate for system decomposition or deployments requiring separate credentials and access controls. If production and development need different credentials, as they should, give them different backends and credentials, not different workspaces on one backend.

Plans

What a plan compares

A plan answers one question: what would the tool do to make the real environment match the configuration? To answer it, Terraform refreshes its record from the real objects, then compares the configuration with the refreshed record. The refresh is why a plan can notice that something changed outside Terraform. Terraform’s documentation describes -refresh=false as disabling the synchronization of state with remote objects before checking for configuration changes, which makes planning faster and can produce incomplete or incorrect plans because external changes are ignored.

Planning the sandbox configuration with the replica count changed from 2 to 3 looks like this.

terraform plan -var replicas=3

The plan listed one resource, terraform_data.service, to be updated in place. It showed replicas = 2 -> 3 under input, and showed the output as known only after apply, followed by Plan: 0 to add, 1 to change, 0 to destroy. Reading the plan means checking three things in order: the counts, whether any line says destroyed or must be replaced, and which attributes force a replacement.

For automation, terraform plan -detailed-exitcode encodes the result in the exit status. The documentation defines 0 as success with an empty diff, 1 as an error, and 2 as success with a non-empty diff. In the sandbox, the exit status was 0 for a plan with the current values and 2 for a plan with a changed replica count. A scheduled job can use the exit code 2 as a signal, which the drift section builds on.

Saved plans

A plan printed to the terminal is only a preview. Terraform’s output says so: without -out, it cannot guarantee that a later apply takes exactly the planned actions. Saving the plan with -out=FILE and applying that file removes the gap between what was reviewed and what runs.

A saved plan is tied to the state it was made from. The sequence below makes a plan, changes the state with a different apply, and then tries to apply the old plan.

terraform plan -var replicas=3 -out=p1.tfplan
terraform apply -auto-approve -var replicas=4
terraform apply p1.tfplan

The last command failed with Error: Saved plan is stale, explaining that the plan file can no longer be applied because the state was changed by another operation after the plan was created. This is the behavior you want: the plan reviewed by a person is either applied exactly or rejected, never silently re-computed against a changed world.

Two properties of saved plans matter operationally. First, the file is a zip archive. In the sandbox it contained entries named tfplan, tfstate, tfstate-prev, tfconfig/…, and .terraform.lock.hcl, found by running unzip -l on it. Second, Terraform’s documentation says the file contains the full configuration, all values associated with planned changes, and all plan options including input variables, and that sensitive data is saved in cleartext even if the terminal output obscures it. The sandbox plan was extracted, and the placeholder password string was found in it. Treat plan files as secrets: store them where only the apply stage can read them, and delete them after use.

A pipeline built on this has a clear shape: a plan stage produces a saved plan and a human-readable rendering; a reviewer approves that exact plan; an apply stage consumes that file. Because a stale plan is rejected, a second change merged between plan and apply forces a re-plan and a new review. The extra review is the cost of the guarantee. It also fits separation of duties: the identity that plans can be read-only, and the identity that applies is used only on an approved plan file.

Policy checks on a plan

A plan can be machine-read. terraform show -json FILE renders a saved plan as JSON, and each entry in resource_changes carries a list of actions, such as create, update, delete, or both delete and create for a replacement. That makes it possible to enforce a rule that people forget under pressure. This script, written for the sandbox, rejects any plan that deletes a protected resource. It is a small instance of policy as code.

import json, subprocess, sys

PROTECTED = {"terraform_data.service"}

plan = json.loads(subprocess.check_output(["terraform", "show", "-json", sys.argv[1]]))
violations = [
    (rc["address"], rc["change"]["actions"])
    for rc in plan.get("resource_changes", [])
    if rc["address"] in PROTECTED and "delete" in rc["change"]["actions"]
]
for addr, actions in violations:
    print("BLOCKED:", addr, actions)
sys.exit(1 if violations else 0)

Given a saved plan that updates the protected resource in place, the script exits 0. After the resource is renamed in the configuration and a plan is saved, Terraform plans to destroy the old address and create the new one, and the script printed BLOCKED: terraform_data.service ['delete'] and exited 1. That second case is the opening story in miniature. The script does not know whether the delete is intended. It only makes sure a person has to decide.

A plan is still a prediction. It does not catch errors that the provider’s API reports only when it is asked to make the change, such as a permission denial, a quota, or a name collision, and it cannot show effects outside the resource, such as a service restarting. The plan tells you what the tool will ask the API to do. It does not tell you what the API will accept.

Locking and drift

Locking

Two applies running at once against the same state could each read the same starting record and write conflicting results. Terraform prevents this with a lock: Terraform’s documentation says state locking happens automatically for all operations that could write state, and that backends are responsible for supporting locking where possible. Disabling it with -lock=false is possible, and the documentation does not recommend it.

The lock is visible in the sandbox when a configuration whose only resource runs a slow local command is applied in one terminal and applied again in a second terminal while the first is still running.

resource "terraform_data" "slow" {
  input = "v1"
  provisioner "local-exec" {
    command = "sleep 6"
  }
}

The second command failed at once with Error: Error acquiring the state lock, followed by lock information: an ID, the path, the operation (OperationTypeApply), who holds it, the Terraform version, and the time. With the local backend, a file named .terraform.tfstate.lock.info existed in the directory while the first apply ran, containing the same fields as JSON. The “who” field held a local username and hostname, which are left out here. When the first command finishes, the lock is released and the second can proceed.

Locks create their own failure: a process that crashes can leave a lock behind. terraform force-unlock exists for that, and the documentation says to be very careful with it, because using it while another operation really holds the lock allows concurrent writers. Before forcing, confirm from the lock information and your pipeline logs that the holder is dead. Treat a forced unlock as an event worth recording.

What drift is

Drift is a difference between the declared state and the real one. The entry on configuration drift makes a point that matters for tooling: drift is measured against a system’s own declaration, not against a central standard, and a tool detects it only for the objects and properties it manages. Terraform compares what it knows about managed resources, so a resource created outside Terraform is invisible to it, and so is a property the provider does not read.

The sandbox cannot demonstrate drift, because terraform_data does not read anything outside its own state. To make the mechanism concrete, trace the refresh-then-compare procedure by hand with three inputs: the declaration, the recorded state, and the actual objects (the same logic, written as a small toy model below, gave identical traces; it is an illustration, not Terraform’s implementation). The model refreshes each recorded object from what the API reports, notes any difference as drift, and then compares the declaration with the refreshed record. The table shows the results for a declaration of a bucket with versioning on and a queue with four days of retention.

def plan(desired, state, actual, refresh=True):
    # Step 1: refresh. Copy what the API reports into the working state.
    working = {}
    drift = []
    for name, rec in state.items():
        if refresh:
            if name not in actual:
                drift.append((name, "deleted outside the tool"))
                continue
            if actual[name] != rec:
                drift.append((name, f"changed outside the tool: {rec} -> {actual[name]}"))
            working[name] = dict(actual[name])
        else:
            working[name] = dict(rec)
    # Step 2: compare desired with the refreshed state.
    actions = []
    for name, want in desired.items():
        if name not in working:
            actions.append((name, "create"))
        elif working[name] != want:
            actions.append((name, f"update {working[name]} -> {want}"))
    for name in working:
        if name not in desired:
            actions.append((name, "delete"))
    return drift, actions

What happened outside the tool	Plan with refresh	Plan without refresh
Nothing	No drift, no actions	No drift, no actions
Someone changed the queue’s retention to 14 days in a console	Drift reported on the queue; action: update retention from 14 back to 4	No drift seen, no actions; the live value stays 14
Someone deleted the bucket	Drift reported (deleted outside the tool); action: create the bucket	No drift seen, no actions; the plan believes the bucket exists

Without refresh, the plan compares the declaration with the stale record, which agrees with the declaration, so it reports nothing. That is the practical danger of -refresh=false: it is fast and blind. Terraform also has a refresh-only mode, which its documentation describes as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside Terraform. It records what is true now without changing anything.

Deciding which side is wrong

Detecting drift is the easy half. A difference can mean the declaration is wrong, the live object was changed without authorization, or an authorized emergency fix has not been recorded. The correction differs.

Situation	Correct by	Why
The declaration is wrong or incomplete	Edit the declaration, then apply	Applying the old declaration would reintroduce the defect
Someone changed the live object without authorization, and the declaration is right	Apply the declaration to restore the object	Folding the live value into the declaration turns a workaround into the standard
An authorized emergency change was made	Record owner, reason, and expiry; then either update the declaration or revert after review	Otherwise the next apply silently removes a fix that may still be needed
A property is legitimately managed elsewhere, for example by an autoscaler	Tell the tool to ignore that property, deliberately	Otherwise every plan reports noise that hides real drift

For the last row, Terraform’s ignore_changes argument lists attributes to ignore during updates, and the documentation says it applies to update operations and not creation. Use it narrowly, with a comment about who else manages the attribute, because an ignored attribute is invisible to drift detection.

To make detection routine, run terraform plan -detailed-exitcode on a schedule against each environment, using a read-only role, and alert on exit code 2 from a run with no pending change. The pairing of that scheduled check with the least-privilege access from the previous post is natural: the identity that looks for drift needs to read, not write. A scheduled plan also reports changes made on purpose in code that has not yet been applied, so decide how to separate “pending merge” from “unexplained difference” before alerts begin.

Changing addresses without drift

State binds addresses in the configuration to real objects, so renaming an address looks like deleting one object and creating another. In a copy of the sandbox, renaming terraform_data.service to terraform_data.orders_service and planning gave Plan: 1 to add, 0 to change, 1 to destroy., with the old address “not in configuration” and the new one to be created. For a stateless resource that is harmless. For a database it is the opening story.

A moved block tells Terraform that the address changed and the object did not. Terraform’s documentation says it renames the existing object in state to the new address, and that the block should remain until everyone’s state has been updated. With this block added to the renamed configuration, planning again gives a different result:

moved {
  from = terraform_data.service
  to   = terraform_data.orders_service
}

The plan then reported that terraform_data.service has moved to terraform_data.orders_service and ends with Plan: 0 to add, 0 to change, 0 to destroy.

The opposite error is also visible. In the sandbox, terraform state rm terraform_data.database removed the binding without touching anything real, and the next plan proposed to create the database, Plan: 1 to add. This is the “wrong backend” symptom: a missing binding looks like a missing object. An import block binds an existing object to an address. Terraform’s documentation says the block needs a to address with a matching resource block and an identifier known at plan time. In the sandbox, importing the removed database showed Plan: 1 to import, 0 to add, 1 to change, 0 to destroy, because the built-in resource has no way to read its input back from an identifier. With a real provider, the attributes it can read come from the remote object. Whatever you import, the following plan tells you how close your declaration is to the object you adopted, so read it before applying.

Manual state commands are sharp tools. Terraform’s documentation warns that modifying state outside normal plan and apply operations can lead Terraform to lose track of managed resources, and says to keep backups if you modify it by hand. Prefer moved and import blocks, which are reviewed code, over interactive state surgery.

Secrets

Where a secret ends up

Passing a secret to a resource usually means the tool writes it down. In the sandbox, the database resource takes var.db_password, a variable marked sensitive. Terraform’s behavior with it shows the difference between hiding and not storing.

• The terminal output redacts it. In terraform show, the input attribute displays password = (sensitive value).
• The state file contains it. The placeholder string appears in the stored input value of terraform.tfstate, and the sensitive_attributes list records that the attribute is marked sensitive.
• A saved plan contains it. Searching the extracted plan archive for the same string found it, in line with the documentation’s statement that sensitive data is saved in cleartext in plan files.

Terraform’s documentation on sensitive data says the same: state and plan files can contain values such as initial database passwords or API tokens, and the sensitive argument redacts values from CLI and HCP Terraform displays. Marking a value sensitive protects a screen. It does not protect a file. The entry on secrets management frames the general rule as “configuration references a secret and never contains one,” with a qualification the sandbox demonstrates: if the tool reads the value during a run and passes it to a persistent attribute, the tool now holds it.

Values that are never stored

Terraform has a mechanism for values that should not reach state. The documentation describes ephemeral values: they exist during a run and are omitted from state and plan files, and they can be declared with an ephemeral argument on a variable, through ephemeral resource blocks, or as write-only arguments on managed resources. Ephemeral values may be referenced only in limited contexts, such as locals, provider blocks, and provisioners.

The sandbox shows the guardrail. A variable declared with ephemeral = true and used as the input of a terraform_data resource made Terraform refuse to plan, reporting Invalid use of ephemeral value, and explaining that the value is not valid for input because that argument is not write-only and must be persisted to state. The tool is protecting you from a configuration that would have stored the value.

The route to a real secret is a write-only argument, which the documentation describes as an attribute, typically with a _wo suffix, that a resource accepts during the operation and then discards without storing. Whether a given resource has one depends on the provider. The documentation’s examples are AWS provider attributes for a database password and a secret string. No provider-specific write-only attribute was used in the sandbox, so the behavior cannot be shown the behavior beyond the documentation. If your provider has no write-only attribute for a field, the value will be stored, and the control moves to protecting the state.

A practical ordering for secrets in infrastructure code:

1. Prefer not to need the secret. Federated, short-lived credentials for the pipeline itself, as in the previous post, mean there is no provider key to store.
2. Let the service generate its own credential and keep it in a secret manager, so Terraform handles a reference, not a value.
3. If a value must pass through Terraform, use an ephemeral value with a write-only argument where the provider supports it.
4. Otherwise accept that the value will be in state and plan files, and protect those: encrypted backend, restricted reads, short retention for plan files, and rotation of any credential that was in a file that leaked.

Rotation matters more than cleanup. A secret that has been in a state file that was readable by the wrong people has to be considered exposed, and deleting the file does not change that.

Testing, promotion, and rollback

Layers of testing

Testing infrastructure code does not mean one thing. Each layer finds a different class of error at a different cost.

Layer	What it finds	What it cannot find
Format and validate (terraform fmt, terraform validate)	Syntax and internal consistency	Anything about real providers or values
Variable validation and checks in the configuration	Inputs outside the allowed range	Combinations the author did not think of
terraform test in plan mode	Logic errors in expressions and conditions, without creating resources	What the provider will do at apply time
Plan inspection and policy checks	Dangerous actions such as deletes of protected resources	Whether the intended action is correct
terraform test in apply mode, in a sandbox account	Whether real resources can be created, with the assertions you wrote	Behavior under production data, scale, and permissions
Post-deploy verification	Whether the live result meets requirements	Anything you did not write a check for

Terraform has a built-in test framework. Its documentation says test files use the .tftest.hcl extension and contain run blocks. A run in plan mode checks logic without creating infrastructure, and a run in apply mode, the default, creates real resources and destroys them afterward. Mock providers arrived in Terraform 1.7.0 for tests that need provider-shaped data without real resources. The example below uses a configuration with a validated variable.

variable "replicas" {
  type = number

  validation {
    condition     = var.replicas >= 2 && var.replicas <= 10
    error_message = "replicas must be between 2 and 10."
  }
}

resource "terraform_data" "service" {
  input = {
    name     = "orders"
    replicas = var.replicas
  }
}

output "replicas" {
  value = terraform_data.service.input.replicas
}

and a test file named service.tftest.hcl:

run "accepts_normal_value" {
  command = plan
  variables {
    replicas = 3
  }
  assert {
    condition     = terraform_data.service.input.replicas == 3
    error_message = "replica count did not flow into the resource input"
  }
}

run "rejects_single_replica" {
  command = plan
  variables {
    replicas = 1
  }
  expect_failures = [var.replicas]
}

run "apply_and_check_output" {
  command = apply
  variables {
    replicas = 4
  }
  assert {
    condition     = output.replicas == 4
    error_message = "output mismatch"
  }
}

terraform test reported that all three runs passed: Success! 3 passed, 0 failed. The second run is a negative test: expect_failures makes it pass only because the validation rejected the value, which is the more important direction, since a validation that never fires protects nothing. These tests check the logic of your configuration. They do not check that a real database accepts the settings, which needs the apply-mode layer in a sandbox account, where the cleanup after the run is part of the contract. The test passing also says nothing about whether the thresholds in the validation are the right ones; that is a requirements question.

Promotion

Promotion moves a change through environments, and for infrastructure the thing that moves is a version of the code. The environment promotion entry describes moving one artifact through environments with configuration supplied separately. The infrastructure analog is one version of the configuration, applied to each environment with that environment’s own inputs, its own state, and its own credentials.

Three items define that version, and Terraform handles two of them differently.

• Provider versions. Terraform records selected provider versions and checksums in .terraform.lock.hcl. The documentation says to commit it so changes to external dependencies can be reviewed, and that terraform init -upgrade selects newer versions within your constraints. The file belongs to the configuration as a whole.
• Module versions. The same documentation says Terraform does not remember version selections for remote modules, so it always selects the newest available version that meets the constraints. Pin module versions explicitly in the configuration, since the lock file will not do it for you.
• Inputs. Variables per environment, passed through files or the pipeline, are the only intended difference between environments. Anything else that differs, such as a resource present in one environment and not another, is a difference you should be able to justify.

Promotion order follows risk: apply to a non-production environment first, verify, then apply the same version to production. Two cautions apply. A saved plan is specific to one environment’s state, so it cannot be “promoted”; each environment is planned and reviewed on its own, and what you promote is the code. And promotion shows only what the earlier environment exercised. If staging has smaller data, a different network layout, or a missing integration, a successful staging apply does not establish that production will behave the same, so list those differences when you decide how much to trust it.

Safe change patterns

Several habits reduce the chance that a reviewed change becomes an incident.

• Make changes small and independently reversible. A change that renames, resizes, and rewires at once is hard to review and hard to recover from.
• Treat destroys and replacements as the main review item. Read them first. The plan guard above makes that mechanical.
• Use lifecycle protection for objects that cannot be recreated. prevent_destroy makes Terraform reject a plan that would destroy the object. The sandbox shows it: with the flag on a resource, terraform destroy plans the destruction and then fails with Error: Instance cannot be destroyed, saying the plan calls for the resource to be destroyed while lifecycle.prevent_destroy is set. The documentation names a caveat, and the sandbox confirms it: if you delete the resource block, including its lifecycle block, the flag is gone. A plan made after deleting the whole block proposes to destroy the object with no error. prevent_destroy protects against replacement and against destroy, not against removing the code, so pair it with review rules on deletions and with backups.
• Replace only on purpose. -replace=ADDRESS plans a replacement of one resource, which the documentation describes as useful when an object has become degraded. create_before_destroy builds the replacement before removing the original, and the documentation warns that many remote objects have unique-name constraints that must allow both to exist at once.
• Apply what was reviewed. Use the saved-plan flow, so a late merge or a changed environment forces a new review.

These patterns reduce risk without removing it, and they need an owner. Someone has to decide who may approve a plan that deletes something, and under what conditions an emergency change is allowed to skip the pipeline. The identity that applies infrastructure changes can modify what the policies of the previous post protect, so its use should be reviewed and logged like other highly privileged activity.

Rollback is usually roll-forward

Application deployments sometimes support a rollback to a retained earlier version. For infrastructure, that idea is weaker, because an apply is a sequence of provider API calls and there is no transaction wrapping them. The sandbox can show what a failed apply leaves behind. The configuration has three resources in a chain, and the middle one runs a command that fails.

resource "terraform_data" "network" {
  input = "net-v1"
}

resource "terraform_data" "service" {
  input = terraform_data.network.output
  provisioner "local-exec" {
    command = "echo simulated failure; exit 1"
  }
}

resource "terraform_data" "dns" {
  input = terraform_data.service.output
}

terraform apply -auto-approve creates network, begins service, runs the command, and stops with Error: local-exec provisioner error and Error running command 'echo simulated failure; exit 1': exit status 1. terraform state list then shows terraform_data.network and terraform_data.service, and not terraform_data.dns. The first resource stays created, the second is recorded in state, and the third was never attempted. The next plan reports terraform_data.service is tainted, so must be replaced and proposes Plan: 2 to add, 0 to change, 1 to destroy.

Nothing was undone. Terraform recorded what it had done, marked the half-finished resource for replacement, and left the decision to you. The recovery options follow from that:

1. Fix the cause and apply again. This is the default. The state shows how far the first attempt got, and idempotent resources can be reconciled the rest of the way.
2. Revert the configuration and apply. This restores the earlier declaration by changing the intended state back. It is itself a new change, with its own plan, and anything the earlier apply changed has to be reversed by providers’ own update operations. It works for a reversible change and fails for objects that were already destroyed.
3. Restore state from a retained version if the state record is wrong. This is what bucket versioning is for. It repairs bindings, not the objects they refer to; the glossary entry on the infrastructure state file says recovering bindings does not restore database contents.
4. Restore data from backups, which Terraform does not manage. A destroyed database comes back from a backup that was tested before it was needed, not from a code revert.

The entry on rollback makes a point that applies here: a past action stays a past action. A deleted resource, a rotated key, and an email sent by a provisioner are not reversed by returning to earlier code. Before an apply that destroys or replaces something with data, decide the recovery path: whether a backup exists and was restored in a test, whether prevent_destroy should guard the resource, and who can stop the pipeline. Where an apply is risky and cannot be undone, the better design is often a forward fix prepared in advance, or a smaller step that is safe to leave half-done.

Failure modes in summary

Failure	Mechanism	Prevention or detection
A rename deletes and recreates a resource	State binds addresses, so a new address is a new object	Plan guard on deletes; moved blocks; prevent_destroy for irreplaceable objects
A plan looks safe but applies something else	Plan made from a different state or not saved	Apply the saved plan; stale plans are rejected
Two writers corrupt state	No locking, or locking disabled	Backend with locking; avoid -lock=false; careful, recorded force-unlock
Drift goes unnoticed	Refresh skipped, or no scheduled plan	Scheduled -detailed-exitcode plan with a read-only role; avoid -refresh=false for checks
Drift is “fixed” in the wrong direction	Live value copied into code without a decision	Decide per the correction table; record emergency changes
A secret leaks through state or plan	Sensitive marking only redacts display	Ephemeral and write-only values; restricted, encrypted state; plan files treated as secrets; rotation
State lost or wrong	Local file, or wrong backend selected	Remote backend with versioning; check backend and workspace before trusting a plan
A provider or module changes between environments	Unpinned modules; lock file not committed	Commit .terraform.lock.hcl; pin module versions
Half-applied change	No transaction across provider calls	Small changes; read the next plan; recovery path decided in advance
Data lost by a destroy	Destroy is idempotent and irreversible	Backups restored in tests; prevent_destroy; review of every delete

Some limits apply to everything above. The experiments ran on one machine with one Terraform version using a built-in resource, so they show Terraform’s mechanisms and message formats, not how a particular cloud provider’s API behaves under failure, quotas, or eventual consistency. The drift table comes from a model, not from a live provider. Terraform’s behavior and documentation change between versions, so confirm details against the version you run. The next post, Packaging Services and Building Reproducible Artifacts, applies the same ideas of recorded state and controlled change to the thing your infrastructure runs: the build output.

References

• Terraform documentation: State
• Terraform documentation: State locking
• Terraform documentation: Manage sensitive data
• Terraform documentation: Ephemeral values
• Terraform documentation: Backends
• Terraform documentation: S3 backend
• Terraform documentation: CLI workspaces
• Terraform documentation: terraform plan command
• Terraform documentation: terraform_data resource
• Terraform documentation: moved block
• Terraform documentation: import block
• Terraform documentation: Tests
• Terraform documentation: lifecycle meta-argument
• Terraform documentation: Dependency lock file
• Terraform documentation: Manipulating state