Infrastructure as Code in Practice

Consider a hypothetical review: an engineer changes one line of infrastructure code, and the plan proposes replacing a database. The reviewer pauses to diagnose the replacement before approving it. If that investigation never happens, the repository can gradually stop describing production.

One useful starting point is to distinguish three answers to the question “what infrastructure do we have.” Their disagreement can explain unexpected plans, alongside provider constraints and intended changes.

The declaration — what your code says should exist.
The record — what the tool believes exists, and which real object corresponds to which declared resource.
The reality — what the provider actually has, including whatever someone created in the console last Thursday.

This article uses those three views to explain declarations, state handling, change review, execution boundaries, and secret exposure. Terraform and configuration management are examples; their behavior depends on the operation, provider, and supported version.

What “declarative” commits you to

Infrastructure as code uses versioned, machine-readable definitions to provision and manage infrastructure. Microsoft Learn presents repeatable environments as its goal. Achieving that goal requires controlled provider, module, image, package, and input versions, explicit handling of external data, and verification of the result. The same source alone does not guarantee the same build or deployment, and equivalent environments need not have identical generated resource identifiers.

Use reviewed code changes as the normal path for managing environment drift. An authorized emergency console change can still be necessary: record its owner, reason, and temporary scope, then decide the intended state. Either encode the accepted change or deliberately restore the declared state after checking impact. A refresh-only operation updates Terraform’s record; it does not update the configuration.

The mechanism underneath is usually called idempotency — “a deployment command always sets the target environment into the same configuration, regardless of the environment’s starting state.” What is worth noticing is that the published definition names two ways to achieve it: “idempotency is achieved by either automatically configuring the existing target, or by discarding the existing target and recreating a fresh environment.”

Idempotency means that repeating an operation has the same effect as performing it once. A one-shot operation that maps every supported starting state to a fixed target is idempotent, but repeated reconciliation can converge without each step being idempotent: for nonnegative integers, f(x) = max(x – 1, 0) eventually reaches zero while f(f(2)) differs from f(2). Neither property establishes reversibility. Preserving unmanaged attributes is a separate contract, not a consequence of idempotency.

Whether an operation is safe to retry depends on whether repeating it can produce a second effect — which is what an idempotent contract rules out, since “an idempotent operation is one where a request can be retransmitted or retried with no additional side effects.” What supplies that contract varies: a caller-supplied request identifier, a natural key the service itself refuses to duplicate, or an operation whose result is the same however many times it lands. So the absence of a request identifier does not by itself make a create unsafe to repeat — you have to know what the API promises.

Whether a result can be undone depends on whether the previous state still exists somewhere, and on what else happened in between. Deleting what you just created does not restore the world if the creation already had effects outside the resource — a notification sent, a record written elsewhere, an identifier handed to another system, a meter started.

Retrying a delete is safe under conditions worth stating rather than assuming: that the target is named by something stable, and not by a name that may since have been reused. A delete aimed at the resource currently called X is a different operation from one aimed at the resource with identifier 4471, and if a later run recreated X, the retry destroys the new one. So the reflex worth having is not “re-run after creates, never after deletes” but four questions: what exactly does this operation target, what does the API promise about repetition, what is the current state, and — separately from all of that — is there a restore path if the answer turns out to be wrong.

Those two routes are the real fork in this subject, and most tool arguments are actually arguments about which one you have chosen.

	Converge in place	Replace
What a run does	Inspects the target and changes what does not match	Builds a new instance from a known artifact and swaps it in
Typical tooling	Configuration management — Ansible and its peers	Image pipelines plus provisioning
Failure mode	Undeclared drift can escape detection	Unpreserved instance-local state is lost on replacement
Where the truth accumulates	Definitions plus observed host state	Image, launch configuration, and external dependencies

The second column is the immutable approach, where the fleet is kept current by building a new image and replacing hosts rather than repairing each one in place — worked through under golden image. The first is the one people mean by configuration management, and its characteristic weakness is that convergence only covers what the definitions mention. A package installed by hand two years ago may remain if neither a declared package baseline nor an observation rule covers it. If it violates a managed expectation, that gap between declared and actual is configuration drift, and it is the reason a converging tool needs a detection habit rather than only a correction habit.

Most estates run both, which is fine as long as the boundary is written down: the image owns what every host has, and the configuration layer owns what varies. Where that boundary is unstated, the same setting gets managed in two places and the last run wins. Replacement also requires a preservation plan for valuable instance-local data and explicit treatment of external volumes, databases, and other dependencies; those are not automatically lost with an instance.

The record, and why it is an operational asset

A declarative tool needs a way to connect configured addresses with the cloud objects it manages. In Terraform, the infrastructure state file stores those bindings, metadata, and information used to improve performance for large infrastructures. Terraform describes its primary purpose as “to store bindings between objects in a remote system and resource instances declared in your configuration.” The record covers the workspace’s managed infrastructure and configuration; code alone does not supply those bindings.

State is Terraform’s normal authoritative record of resource bindings, but an import declaration can also identify an existing object and its intended address. If a plan unexpectedly proposes creation, stop writes and verify the backend, workspace, and resource address first. Restore a verified state snapshot or reconstruct supported bindings through import and review the resulting plan. A missing binding can lead to creation or a provider conflict; it does not guarantee a second database. Recovering bindings does not restore database contents.

One boundary before the consequences, because it is easy to read the above as a law of declarative tooling. The record has to exist; a file you own and protect is Terraform’s answer rather than the only one. Azure’s Bicep lists the opposite arrangement as a feature — “no state or state files to manage: Azure stores all states” — and describes its preview operation as one that “checks the current state of your environment and eliminates the need to manage this state.” Everything in this section is about the case where the tool hands you the record, which is worth knowing when you compare tools: where does the binding live, and who is responsible for it decides whether these five consequences are yours.

The bindings make state an operational asset distinct from source code: protect concurrent changes, recovery, and access using the following handling rules from Terraform’s state documentation.

Do not put it in version control. Use storage with state locking and secure access control. Unsuitable storage risks both data loss and disclosure of secrets in state.
Do not hand-edit it. State is readable JSON, but use Terraform CLI state commands for supported modifications rather than editing the file directly. The CLI handles the state format as it evolves.
Require locking. A backend that supports locking prevents concurrent writers during operations that could change state. Terraform stops if lock acquisition fails. Check backend support; not all backends provide it. Force-unlock is a last resort because removing another active writer’s lock permits multiple writers.
Treat it as sensitive. State and saved plans can contain resource details and secrets such as initial database passwords or API tokens. Local state is plaintext and includes secret values supplied in configuration.
Prepare recovery. Back up state securely and test restoration of verified snapshots. If restoration is unavailable, supported imports may reconstruct bindings, but require identity and ownership checks. State recovery and recovery of application data are separate procedures.

Review who can read state and saved plans. They may contain credentials and sensitive infrastructure details, although the contents depend on resource, provider, and ephemeral or write-only support. Repository access should not automatically grant artifact access; apply least privilege to these records too.

Reviewing a change to real infrastructure

The distinguishing feature of this kind of code is that you can ask what a change would do before doing it. A plan “creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure,” by reading the current state of existing remote objects, comparing configuration to prior state, and then proposing “a set of change actions that should, if applied, make the remote objects match the configuration.” Critically, “the plan command alone does not actually carry out the proposed changes.”

Review both the configuration diff and the proposed plan. If one line unexpectedly replaces a database, inspect the attribute requiring replacement, the binding, and the selected backend and workspace before approving anything. Share a reviewed, redacted human-readable summary or a restricted artifact link; a saved plan is a separate potentially sensitive artifact.

What a plan does not tell you is worth stating, because a green plan reads like safety.

Whether a replacement is acceptable. The plan says a resource will be destroyed and recreated. Whether that means thirty seconds of downtime or the loss of a volume is domain knowledge the tool does not have.
How long the intermediate state lasts, and what happens if it fails there. A plan does express ordering — dependencies determine sequence, and a replacement is shown as a destroy and a create whose order you can influence, since by default Terraform “destroys the existing object and then creates a new replacement object” while create_before_destroy reverses that. What it cannot tell you is the wall-clock window, whether the new object is actually ready when it reports created, or what state you are left in if the run dies midway. Note also that create_before_destroy is “an opt-in behavior” precisely because two objects of the same kind often cannot coexist — so reading the order in the plan is part of judging safety.
Whether anything outside this configuration depends on what is changing. A security group nobody declared may be the only thing letting a legacy job connect.

A plan normally refreshes managed objects before proposing changes, but refresh can be disabled. Terraform can reject a saved plan when its recorded state is stale. That check does not establish that external infrastructure is unchanged: provider or API checks may detect some intervening changes, but not all. Re-plan after material delay or suspected drift, and assess the final plan before applying.

Include a reviewed, redacted plan summary or a restricted artifact link in the pull request, alongside the diff; branch and merge mechanics support this review. Examine destroys, replacements, and other changes with irreversible external effects. Retry safety depends on the API contract and stable target identity; restoration depends on retained data and a tested recovery path. Record consequential decisions in an architecture decision record.

Modules and environment boundaries

Modules are usually introduced as a way to avoid repetition, which is the least interesting reason to have them. But before the criteria, a distinction that the word “boundary” blurs and that decides whether any of this works: a module is a unit of code, not a unit of execution.

Splitting a configuration into child modules under the same root leaves you with one plan, one state, and one apply. The refactor improves readability and reuse and changes nothing about what a single bad run can touch. What separates execution is a separate root configuration with its own state and its own credentials — and then the coupling between them becomes an explicit contract (an output read as an input, a data lookup, a parameter passed in) rather than a shared graph.

So read the three criteria below as reasons to consider a separate root where they bite hardest, and as reasons for child modules where they do not.

Rate of change. A network that changes twice a year and a service definition that changes weekly are awkward together, because the slow thing is refreshed and evaluated on every fast change and each evaluation is a chance to propose something unintended. Note that moving the network into a child module does not stop this — it is still in the same plan. Only a separate root stops it.
Ownership. A unit that two teams both need to change is a coordination cost on every change. Boundaries that follow ownership let each team move without waiting.
Blast radius. Everything sharing one state and one apply can be damaged by one bad run — and “sharing one state and one apply” is the operative phrase rather than “being in the same module.” This is the argument for keeping the database’s definition in a different root configuration from the thing that redeploys most often, with its own credentials so that the fast-moving pipeline could not destroy it even if the plan asked to.

Keep state and execution credentials separate across environments. Within an environment, multiple root configurations and states can follow ownership, lifecycle, or blast-radius boundaries. Give each managed object one authoritative owner and make dependencies between roots explicit. Separate states without appropriately scoped credentials do not establish the intended access boundary.

Two related notes. The isolation this buys is the same reasoning as workload isolation and tenant isolation, applied to the tooling that changes things rather than to the things themselves. And the account and organizational structure this sits inside — who may create what, and where the guardrails live — is a separate design covered under landing zone and policy as code.

Secrets leak through this machinery by default

Secret values may persist in Terraform state and saved plan artifacts. Those artifacts are distinct from human-readable terminal output, which can redact marked sensitive values. JSON exports and debug output can expose values again. Protect artifact storage and inspect log-producing paths rather than assuming every archived plan is a CI log.

The obvious fix — reference the secret instead of writing it — is necessary and not sufficient, and this is the point at which most treatments of the subject stop too early. Two designs get called “referencing a secret” and they differ in exactly the way that matters.

The runtime resolves the reference. Your configuration passes an identifier — a secret name, an ARN, a path — and the application or service fetches the value itself when it runs. Terraform never holds the value, so it cannot store it.
Terraform resolves the reference. You read the secret from a store during the run and pass the value to a resource argument. Terraform now holds the value, and by default it goes into state and plan just as a hardcoded one would. The source of the value changed; the exposure did not.

Marking things sensitive does not close the second case either. Terraform still saves sensitive values in state and plan files; a reader with artifact access can obtain them. The sensitive marker redacts terminal output rather than preventing storage. Saved plans can contain those values in cleartext even when the terminal display hides them, so protect the plan artifact itself.

What does omit values is a separate, newer mechanism: ephemeral values, which “are available at runtime, but Terraform omits them from state and plan files entirely.” It comes with version and provider conditions worth checking before you plan around it — the sensitive argument needs Terraform 0.15 or later, the ephemeral argument and block need 1.10 or later, a write-only argument on a managed resource needs 1.11 or later, and support is not universal because “each provider defines any available ephemeral blocks and write-only arguments on their managed resources.”

Prefer runtime resolution when Terraform does not need the plaintext. Where it does, use supported ephemeral or write-only paths, or protect state and saved plans with encryption, access controls, and retention limits. Rotate credentials after actual or suspected unauthorized exposure and address retained copies. Legitimate storage in protected state alone does not establish a compromise.

With that established, secrets management is the surrounding discipline, and five of its published principles matter most here.

Standardize how teams use secret services. OWASP allows multiple solutions where needs differ. Define maintainable access, delivery, and incident procedures across them; this does not require a single interface per team.
Limit access. Give people and workloads only the secret access their responsibilities require.
Automate handling. Reduce manual interaction with plaintext through controlled issuance, delivery, and rotation.
Prefer short-lived credentials and verify enforcement. Short-lived credentials can limit reuse, but the current TTL is not an unconditional exposure bound. Check renewal permissions and maximum lifetime, actual expiry enforcement, and whether target-side revocation succeeds. Restarting an application does not itself revoke its previous credential. Vault can fail to revoke a database credential when it cannot reach the database. Monitor failed revocations and verify that the target rejects new authentication; test existing-session behavior separately. Expiry cannot undo data already accessed.
Plan the whole lifecycle. Define and test creation, rotation, revocation, and expiration rather than stopping at issuance.

Protect audit records against tampering and correlate events without logging secret values. Vault records requests and responses to its own API, including issuance, reads, renewals, and revocations. Database or API credential use, authentication failures, and attempts to reuse expired credentials require downstream service logs as well. Correlate credential or lease identifiers and timestamps, document collection gaps, and define behavior when logging fails. An audit log at the secret store alone cannot prove how an external service was used.

One operational requirement that gets discovered at the worst time: this store is on your recovery path. In an incident, “having to wait for credentials could impact the responsiveness of the operations team.” A secrets service that is down during an outage extends the outage, so it needs its own availability answer and a documented break-glass route.

An environment variable is a delivery mechanism, not a sensitivity classification. A hostname may be ordinary configuration or sensitive operational information; a credential requires secret handling. Version non-sensitive configuration, restrict sensitive information according to its classification, and automate credential detection before commits and in pipelines.

Four ways this is not application code

Teams that treat an infrastructure repository exactly like an application repository hit the same four surprises, and each follows from something above.

Changes operate on external state. Infrastructure operations combine declarations, recorded bindings, and live resources. Application deployment can also depend on external state; a source build alone is a narrower activity.
Some changes are irreversible. Re-running a failed deployment is normal, and re-running a destroy usually just confirms the absence — provided it targets a stable identifier rather than a name something else may now hold. What re-running cannot do is undo, so for anything that removes state the question is not whether to retry but whether a tested restore path exists, and whether anything outside the resource already depended on it.
Reverting code does not restore data. A reverted commit proposes another infrastructure change whose feasibility depends on current state and retained recovery material. As with environment promotion, a retained build artifact can restore software but does not by itself restore changed data. Choose a compatible reverse change, tested restore, or forward repair.
Protect intermediate artifacts. Infrastructure secrets can appear in state and saved plans. Application pipelines can also leak secrets through build output, logs, or artifacts, so inspect both paths.

Start by verifying bindings, ownership, and intended state. Separate environments and execution scopes, protect and back up their records, and review redacted plans before changes. Give each object one authoritative owner while allowing multiple roots and states within an environment. Verify which components receive and retain secrets.

References

Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Infrastructure and Configuration as Code