Deploying the Same Artifact Across Environments

A team tests a service in a staging environment, sees green results, and deploys to production. The sentence “we deployed what we tested” is only true if several things hold at once: the bytes that reached production are the bytes that were tested, the settings that differ between environments are limited to settings that are supposed to differ, the credentials were supplied without being baked into anything that travels, and someone can return to the previous state if the new one misbehaves. Each of those conditions can fail quietly, and a pipeline that never checks them produces deployments that look like promotions but behave like fresh, untested releases.

This article is about the mechanics that make “the same artifact in every environment” a checkable statement. It covers environment configuration, promotion gates, secrets, rollback, reproducibility, and deployment validation. By the end you should be able to design a promotion path in which one immutable artifact moves through environments, each environment’s differences are explicit and reviewable, each step records what was deployed, and a rollback is something you have planned the conditions for rather than something you hope works.

Prerequisites: the preceding article on designing CI pipelines and trust boundaries introduced stages with different levels of authority, the practice of keeping verification and release paths apart, and an evidence record that ties a test result to the digest of a specific artifact. This article starts from that point: given a published artifact and evidence about it, how does it move into running environments without becoming a different thing? It also builds on the earlier article on packaging services and building reproducible artifacts, which established that an artifact has a content-derived identity and that builds should be deterministic where practical. General programming knowledge and basic command-line use are assumed.

The code in this article is original and was written for it. The one program included, a packaging script, is a small demonstration of a mechanism and not a production tool. The deployment procedure described in the text is a design to apply with your own tooling, not a program that was run. All hostnames use reserved example domains, and secret values are placeholders.

What has to be the same, and what is allowed to differ

Environment promotion means moving one build artifact through a sequence of environments, such as development, staging, and production, without rebuilding it. The alternative, building separately for each environment, is easy to set up and makes the central claim unprovable. Two builds from the same source revision are not guaranteed to produce the same output, and a build that fetches dependencies at build time can pick up different versions on different days. A test result for one build then describes a different object from the one that was deployed.

DORA’s deployment automation guidance names the practices that make promotion repeatable. Use the same deployment process for every environment, including production. Use the same packages in every environment, and keep environment-specific configuration separate from the deployable artifact. Make it possible for anyone with the necessary credentials to deploy any version of the artifact to any environment on demand in a fully automated fashion. Make it possible to recreate the state of any environment from information stored in version control. And perform automated checks after deployment that verify that external services are reachable and the system works. The remaining sections take these in turn and add the mechanics the guidance leaves to you.

It helps to write the split down. Items that must be identical: the executable code and bundled assets, the dependency set it ships with, and the identity used to refer to it. Items that may differ by environment, and must be supplied from outside the artifact: endpoints, capacity settings, log verbosity, references to credentials, and the state of operational switches. Items that differ whether you want them to or not: data volume, traffic mix, the infrastructure in front of the service, third-party systems that are stubbed in test and live in production. The third group cannot be eliminated, so promotion has to be paired with an explicit statement of what a passing result in one environment does and does not tell you about the next.

Artifact identity: tags, digests, and an immutable store

A promoted artifact needs a name that always refers to the same bytes. Two naming styles exist and they behave differently. A tag such as 1.4.2 or latest is a label that the registry owner can point at different content over time. A digest is the output of a collision-resistant hash over the content. The OCI image specification describes a descriptor’s digest as a content identifier of the form algorithm:encoded, states that implementations should use SHA-256, and says consumers should verify received content against the digest before accepting it from untrusted sources.

Kubernetes’ image documentation spells out the operational consequence. Tags can be reassigned at the registry, which can leave a mix of pods running old and new code under the same tag. Digests pin a specific image. When both a tag and a digest are given, only the digest is used. The pull policy interacts with this: the documented defaults are Always when the tag is latest or omitted and IfNotPresent for other tags, so a node can reuse a locally cached image that no longer matches what the tag currently points to. A deployment record that says only “tag 1.4.2” cannot settle which bytes ran. A deployment record that says “digest sha256:…” can.

A build artifact store therefore needs three properties, and the choice of product matters less than whether it provides them. Immutability: a published identity always maps to the same bytes, and the store refuses to overwrite it. Retention: the artifacts you may need to return to, with the configuration and dependencies to run them, stay available for at least the rollback window. Deployment records: for every environment, which artifact and which configuration are deployed now and what was deployed before. For container artifacts, a container image is a manifest that references its configuration and layers by digest, so referring to the manifest digest ties a deployment to the full content.

A reference deployment procedure used in this article keeps artifacts in a store named by digest with read-only files. Building the same content again returns the existing identity and never replaces a stored file. A deployment verifies that the bytes it is about to install still hash to the digest being promoted. The next sections explain each part of that procedure and the checks it makes.

Reproducibility: when a rebuild can stand in for the artifact

Promotion avoids rebuilding, but rebuilds happen: a retained artifact expires, a hotfix branch needs a clean build, an auditor wants independent confirmation. A reproducible build is one where the declared source, environment, and instructions let another party recreate every specified output byte. The claim is scoped to specified outputs under stated conditions, and logs are usually outside the specified set.

Most nondeterminism does not come from the source code. The Reproducible Builds documentation lists these categories of variance to manage, among others: timestamps, timezones and locales, build paths, stable ordering of inputs and outputs, archive metadata, randomness, value initialization, and version information. Typical examples are embedded build timestamps and archive modification times, absolute build paths leaking into outputs, and unstable ordering of directory iteration and archive entries. A standard mitigation for time is the SOURCE_DATE_EPOCH environment variable: Reproducible Builds describes it as a standardized variable that build tools can consume to produce reproducible output, representing a fixed timestamp in seconds since the Unix epoch rather than the current time.

The script below packages the same three files twice. The “naive” function stores whatever modification times the filesystem reports, walks directories in the order the operating system returns, and stamps the compressed file’s header with the current time. Between the two runs the script touches every file, as a fresh checkout would. The “normalized” function sorts entries, fixes modification times, owners, and permissions, and zeroes the compression timestamp. The source content is identical in every run. The script was run locally with Python 3.12 and its output follows. The naive digests vary from run to run because they depend on the current time.

# Package the same source tree twice and compare digests.
import tarfile, gzip, io, os, hashlib, time, shutil, sys

SRC = "src_tree"
shutil.rmtree(SRC, ignore_errors=True)
os.makedirs(f"{SRC}/app")
open(f"{SRC}/app/main.py", "w").write("print('hello from the service')\n")
open(f"{SRC}/app/util.py", "w").write("def add(a, b):\n    return a + b\n")
open(f"{SRC}/VERSION", "w").write("1.4.2\n")

def sha256(b): return hashlib.sha256(b).hexdigest()

def naive(out):
    # tar with whatever mtimes the filesystem has, directory-walk order, gzip stamped with current time
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w") as t:
        for root, dirs, files in os.walk(SRC):
            for f in files:
                t.add(os.path.join(root, f), arcname=os.path.relpath(os.path.join(root, f), SRC))
    gz = io.BytesIO()
    with gzip.GzipFile(fileobj=gz, mode="wb", mtime=int(time.time())) as g:
        g.write(buf.getvalue())
    open(out, "wb").write(gz.getvalue()); return sha256(gz.getvalue())

def normalized(out, epoch=1700000000):
    # sorted entries, fixed mtime/owner/mode, gzip header mtime fixed
    names = []
    for root, dirs, files in os.walk(SRC):
        for f in files: names.append(os.path.relpath(os.path.join(root, f), SRC))
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w", format=tarfile.PAX_FORMAT) as t:
        for n in sorted(names):
            ti = t.gettarinfo(os.path.join(SRC, n), arcname=n)
            ti.mtime = epoch; ti.uid = ti.gid = 0; ti.uname = ti.gname = ""; ti.mode = 0o644
            with open(os.path.join(SRC, n), "rb") as fh: t.addfile(ti, fh)
    gz = io.BytesIO()
    with gzip.GzipFile(fileobj=gz, mode="wb", mtime=0) as g:
        g.write(buf.getvalue())
    open(out, "wb").write(gz.getvalue()); return sha256(gz.getvalue())

a = naive("a.tgz"); time.sleep(2); 
for p in ("app/main.py","app/util.py","VERSION"): os.utime(f"{SRC}/{p}", None)   # a fresh checkout would touch every file
b = naive("b.tgz")
print("naive build 1      :", a[:16])
print("naive build 2      :", b[:16], "(same source content)")
c = normalized("c.tgz"); time.sleep(2)
for p in ("app/main.py","app/util.py","VERSION"): os.utime(f"{SRC}/{p}", None)
d = normalized("d.tgz")
print("normalized build 1 :", c[:16])
print("normalized build 2 :", d[:16])
print("naive equal:", a == b, "| normalized equal:", c == d)

naive build 1      : 4290d2b4b77ca3e4
naive build 2      : efda82b93da504b4 (same source content)
normalized build 1 : 2fc1d5ebc38c048f
normalized build 2 : 2fc1d5ebc38c048f
naive equal: False | normalized equal: True

The naive builds differ although no byte of source changed, so a digest recorded after the first build would not match the second. The normalized builds match. This is a demonstration of the mechanism on a small archive, not a general result: a real build also has compilers, linkers, embedded paths, and dependency resolution, and the effort needed to reach bit-for-bit reproducibility depends on the build system. When it holds, a rebuild whose digest equals the retained digest can preserve the link to the earlier test evidence. When it does not, the rebuild is a new candidate that needs its own validation.

Two cautions keep this honest. Matching bytes show that declared inputs produced those bytes. They do not show who ran the original build, when, or who approved it, which is what authenticated build provenance records. And pinning dependencies (as the earlier article described, with a lock file and hashes) shrinks the set of uncontrolled inputs without enumerating all of them: a build that reads the clock, draws a random value, or fetches over the network can still differ. Differing bytes are the signal that tells you something uncontrolled was involved.

Environment configuration

Configuration is everything that varies between deployments of the same code. The Twelve-Factor App states this as “everything that is likely to vary between deploys” and requires strict separation of config from code. Its litmus test is whether the codebase could be made open source at any moment without compromising any credentials. The same document argues against grouping settings into named environments in code, because named groups multiply as deployments multiply. The practical reading for a promotion pipeline is that the artifact should contain no environment names and no environment-specific values, and the environment should supply a complete set of values at deployment time.

Four kinds of setting

Sorting settings by purpose makes review easier, because each kind has a different risk if it is wrong.

Kind	Example	If it is wrong	How to check
Identity and endpoints	The URL of a dependency, a queue name	The service talks to the wrong system, possibly a production system from a test environment	Validate the format at startup; test that the environment can reach its own dependencies and not others
Capacity and limits	Replica count, pool sizes, timeouts	Overload, starvation, or wasted capacity	Compare against measured load; keep staging limits stated as scaled versions of production
Behavior switches	A feature enabled or disabled	A different code path runs than the one tested	Record the switch states with the deployment and compare with the states tested
Credential references	The name of a secret, not its value	A missing or wrong credential at startup, or a value leaked into a log	Resolve at deployment; fail clearly when absent; never print values

Operational switches deserve separate attention. A feature flag state is part of which version of the system is running, so a configuration comparison that omits it can miss a difference that decides which code path executes. Feature flags are covered in more depth in the later article on progressive delivery.

Keep the shape the same, and record the pairing

Environments will differ in values. They should not differ in which settings exist. A setting that exists only in production is a code path staging never exercised. A shape check that compares the set of keys across environments catches this. It is intentionally a check of presence, not of values, because the values are meant to differ.

Suppose such a check reports that production has a key the other environments lack, for example an extra behavior switch. A missing key causes startup failures. An extra key may mean someone set a behavior switch directly in production and no test has seen it. Compare the extra key with production’s approved declaration before calling it configuration drift. An intentional environment-specific setting may warrant a test-coverage review without violating that declaration. If the live change is valid and approved, incorporate it into the reviewed configuration and test the affected behavior. If it is wrong or unauthorized, restore the declared state. Record an emergency change and review whether to keep or revert it; copying every live setting into code would make a mistake the new baseline.

Configuration changes are releases too. Version the configuration and record, at each deployment, both the artifact digest and a hash of the configuration applied. A deployment record containing the digest, a short configuration hash, and the previous digest lets an investigation begin from an identified release. A rollback target is the pair of artifact and configuration that ran before, and recovering only one half can produce a combination that never ran.

Validate configuration where it is consumed

A service that reads an optional setting and silently falls back to a default turns a typo into a behavior change. Prefer startup validation that rejects missing required values and unknown keys with a clear error, and distinguish an unset value from an explicitly empty string according to the application’s contract, because the two are not interchangeable. Environment variables are one delivery mechanism, and an environment variable is a string supplied when a process starts. Changing a variable for a future process does not change a running process, so a configuration change usually requires a new deployment or a reload mechanism the application implements.

Secrets

A secret is a value whose disclosure would enable authentication, decryption, or access to protected information. The rule that secrets management reduces to is that configuration references a secret and never contains one. Classify the value, not the delivery mechanism: an environment variable or mounted file can carry either ordinary configuration or a secret.

Referencing is not the same as never handling. Whatever component resolves the reference holds the plaintext during that time and may persist it. A configuration file can hold a reference such as "db_password": "env:DEMO_DB_PASSWORD"; the deployment resolves it from the process environment and reports only that the value was resolved, never the value. When the reference cannot be resolved, the deployment should fail with the name of the missing secret instead of starting a service without it.

Three design questions apply wherever secrets are stored. Who may read them: Kubernetes’ Secrets documentation states that Secrets are stored unencrypted in etcd by default, that base64 encoding is not encryption, and that anyone authorized to create a Pod in a namespace can read the Secrets that Pod can reference. Its recommended steps include enabling encryption at rest, applying least-privilege RBAC rules, restricting access to specific containers, and considering an external secret store. When can they be read: deployment platforms can scope secrets to an environment and withhold them until protection rules pass. GitHub’s environment feature, for example, makes environment secrets available only to jobs that reference the environment and only after its protection rules, such as required reviewers, are satisfied. And how they are replaced: a secret that cannot be rotated or revoked without a redeploy of everything that uses it is a recovery risk.

Secrets also interact with promotion in a way that is easy to miss. The artifact must not contain them, because an artifact is retained, copied, and promoted, and anything inside it travels with it. A secret copied into a container image layer remains recoverable even if a later layer deletes the file. Likewise a secret inside a configuration file stored in version control is in the history for good, so the response to such a leak is rotation, not deletion. The least privilege principle also applies per environment: a staging deployment should not be able to read production credentials, because the point of staging is to be a place where failures are survivable.

Revocation deserves rehearsal. Access revocation removes previously allowed access, and its effect depends on every path that authorizes requests: sessions, caches, and tokens that may outlive the change. If your recovery plan for a leaked deployment credential is “revoke it,” test that revocation on a non-production credential and note how long the old credential keeps working.

Promotion gates

A gate is a decision point where a candidate must meet stated requirements before moving on. A release gate can combine automated checks with human approval, and its requirements should be written before the candidate’s results are known. Two kinds of gate are common in a promotion path, and they answer different questions.

An evidence gate asks whether this exact artifact has the required record of success. A deployment procedure can refuse to install a digest into an environment unless a passing record exists for that digest in the previous environment. An attempt to deploy straight to production is then blocked, and the same digest has to be promoted through each environment in order.

An approval gate asks whether an accountable person or system accepts the risk of moving on. GitHub’s deployment environments illustrate the mechanics: up to six required reviewers, where one approval lets the job proceed, an optional setting that prevents the person who triggered the deployment from approving it, a wait timer that delays a job by a configured number of minutes, and restrictions on which branches or tags may deploy to the environment. Each of these is a control with a purpose. A reviewer requirement puts a human decision before production. A wait timer gives time for automated signals to appear. A branch restriction ensures only code that passed review can reach the environment. A control with no stated purpose is friction. A promotion path that takes a day and three approvals makes batching changes rational, and larger batches are harder to diagnose, so each gate should be able to say what failure it prevents.

What a gate certifies

Approval must name its scope: which candidate, which evidence, which environment. Approval for a limited trial is not approval for unrestricted production, and a changed dependency or configuration invalidates evidence collected for the previous candidate. A passing staging run is bounded by its inputs and assertions. It does not establish performance at production data volume, behavior behind production infrastructure, real latency and error behavior of third parties that were stubbed, the effect of feature switch states nobody compared, or contention under real concurrency. Writing down how the environment before production differs from production gives you a specification of what must still be verified some other way, which is the job of the progressive delivery techniques in the next article, progressive delivery and compatible changes.

Gates also have to be placed where the change cannot edit them. The earlier article showed a change that edits the list of required checks. The same applies to promotion: the rule “production requires a passing record from staging” belongs in the deployment system’s configuration, owned by whoever owns production, and not in a file the change under review can modify.

Deployment validation

Moving files is not deploying. The deployment itself is an operation with success conditions that the artifact’s tests cannot cover, because they depend on the target environment. DORA describes a deployment test that verifies external services are accessible and the system functions. Without such assertions, a successful file transfer does not establish that the application can run.

Useful assertions fall into four groups. Identity: the running version corresponds to the intended digest, which means the process reports it and the check compares it with the deployment record, not with a tag. Configuration: required settings were loaded and validated, and the set of enabled behavior switches matches what was approved. Connections: the service can reach its dependencies using the environment’s own credentials, and it cannot reach those that belong to another environment. Compatibility: the service can operate against the current state of data and interfaces. Compatibility deserves explicit attention because it is the case rollback usually fails on. One way to check it is for each artifact to declare the range of database schema versions it supports, and for the deployment test to compare that range with the environment’s current schema version.

In Kubernetes, a readiness probe reports whether a container can serve, and that result feeds routing decisions. It is a routing input and not a firewall, and failing readiness does not itself restart the container. A Deployment’s rolling update replaces pods gradually under maxSurge and maxUnavailable settings, both defaulting to 25 percent in the documentation. The progressDeadlineSeconds field marks a rollout as failed when it stalls but does not automatically roll back, so a failed deployment needs a person or an automated policy to decide what happens next. A deployment that the platform reports as complete is evidence that the platform finished its steps. Whether the service meets its requirements is a separate question that the deployment test, and then monitoring, must answer.

Rollback

Rollback returns a system to a previously retained version as part of recovery. It is useful only if that version can still operate correctly with current data, interfaces, and access requirements. “We cannot roll back” is used to mean three different things that have different answers: whether the old version can be deployed and run, whether effects that already happened can be undone, and whether further damage can be stopped. The last is usually available even when the middle one is not.

The first meaning has two independent preconditions. The retained artifact must be identifiable and still available, which depends on the store’s immutability and retention. And the old version must be able to run against the world as it is now: the schema, data written in a newer shape, interface versions, access requirements. Having the artifact says nothing about the second condition, and it is the condition that fails more often.

A hypothetical example shows the problem. A first release, 1.0.0, declares support for database schema version 1. Version 2.0.0 requires schema 2 and is promoted to production after the schema moves forward. Rolling back to the retained 1.0.0 artifact then fails the deployment test, because the artifact is intact but its declared compatibility range does not include the database’s current schema. A later 1.0.1 build that declares support for schemas 1 through 2 could be deployed without difficulty.

Two lessons follow. First, a rollback procedure should check compatibility before switching, so that a failed attempt leaves production on the version that was running, and should verify afterward which version is actually serving. Second, the route that works in this example is not “go back” but “go forward to a build compatible with both schemas”, which is what the discipline of expand and contract makes possible: add the new form while the old continues to work, migrate consumers, and remove the old form only after the rollback window closes. The next article develops that sequencing for schemas and APIs. Where a schema change cannot be reversed safely, a forward fix or a temporary suspension may be preferable to a rollback, and the decision belongs in the plan before the release.

Rollback mechanics in common platforms illustrate what the platform does and does not retain. Kubernetes keeps a revision history for a Deployment, ten revisions by default, and kubectl rollout undo returns to a previous one. The documentation states that a rollback is triggered only by changes to the pod template, and that scaling replicas does not create a revision. So the platform’s history records pod template changes, not your database schema, your external configuration store, or flag states. Rolling back the workload object alone does not return those. Retention limits matter too: if the revision history or artifact retention is shorter than your rollback window, the option is gone when you need it.

Rollback also does not undo past actions. A message already sent, an order already accepted, or a record already migrated stays that way, and those outcomes need separate investigation and, where appropriate, authorized corrective or compensating actions. Recovery success means the service again meets its requirements, not that a deployment command reported success.

Putting the pieces together

The reference procedure is a sequence of checks, each corresponding to a section above. First the evidence gate: has this digest passed the previous environment. Then identity: do the stored bytes still hash to the digest. Then configuration and secrets: load the environment’s values and resolve references. Then deployment validation: is the artifact compatible with the environment’s current state. Only after those does it switch the running version, write the record, and mark the digest as having passed in that environment. Each failure message should name the condition that failed, which is what an operator needs at the moment of a failed deployment.

The procedure has limits you should keep in mind when applying it. It describes a single deployment step and says nothing about concurrency, partial rollout, or network failure. A compatibility check based on a declared range is only as good as the declaration and the tests behind it. Gates implemented in the same pipeline as the change raise the question of who owns the rule. Resolving secrets from environment variables is a delivery choice with its own exposure. The model below implements the reference procedure; its limits are the ones just listed, and it was run locally.

"""Illustrative promotion model (written for this article, not a production tool)."""
import hashlib, json, os, shutil, tarfile, io, gzip, sys

ROOT = "world"
shutil.rmtree(ROOT, ignore_errors=True)
for d in ("store", "config", "envs", "evidence"): os.makedirs(f"{ROOT}/{d}")
ORDER = ["dev", "staging", "prod"]

def sha(b): return hashlib.sha256(b).hexdigest()

def build(version, schema_supported):
    """Build a deterministic artifact; the version and supported schema range are baked in, environment settings are not."""
    files = {"app.json": json.dumps({"version": version, "schema_supported": schema_supported}, sort_keys=True).encode(),
             "main.py": b"# service entry point placeholder\n"}
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w", format=tarfile.PAX_FORMAT) as t:
        for n in sorted(files):
            ti = tarfile.TarInfo(n); ti.size = len(files[n]); ti.mtime = 1700000000; ti.mode = 0o644
            t.addfile(ti, io.BytesIO(files[n]))
    gz = io.BytesIO()
    with gzip.GzipFile(fileobj=gz, mode="wb", mtime=0) as g: g.write(buf.getvalue())
    data = gz.getvalue(); digest = sha(data)
    path = f"{ROOT}/store/{digest}.tgz"
    if os.path.exists(path): return digest          # immutable: same content, same name
    open(path, "wb").write(data); os.chmod(path, 0o444)
    return digest

def write_config(env, **kv): json.dump(kv, open(f"{ROOT}/config/{env}.json", "w"), indent=1, sort_keys=True)

def config_keys_agree():
    sets = {e: set(json.load(open(f"{ROOT}/config/{e}.json"))) for e in ORDER}
    base = sets["dev"]
    return {e: sorted(base ^ s) for e, s in sets.items() if s != base}

def resolve_secrets(cfg):
    out = {}
    for name, ref in cfg.get("secret_refs", {}).items():
        kind, key = ref.split(":", 1)
        val = os.environ.get(key)
        if val is None: raise RuntimeError(f"secret '{name}' not available from {ref}")
        out[name] = "<resolved, %d chars>" % len(val)   # never print the value
    return out

def evidence(env, digest): return os.path.exists(f"{ROOT}/evidence/{env}-{digest}.ok")

def deploy(env, digest, db_schema):
    tag = f"deploy {env:7s} {digest[:12]}"
    # 1. gate: the same digest must have passed the previous environment
    i = ORDER.index(env)
    if i > 0 and not evidence(ORDER[i-1], digest):
        return print(f"{tag}: BLOCKED (no passing record for {ORDER[i-1]})")
    # 2. verify the bytes in the store still match the identity being promoted
    data = open(f"{ROOT}/store/{digest}.tgz", "rb").read()
    if sha(data) != digest:
        return print(f"{tag}: REFUSED (stored bytes hash to {sha(data)[:12]})")
    # 3. install next to existing releases and read environment configuration separately
    rel = f"{ROOT}/envs/{env}/releases/{digest}"; os.makedirs(rel, exist_ok=True)
    with tarfile.open(fileobj=io.BytesIO(gzip.decompress(data))) as t: t.extractall(rel, filter="data")
    cfg = json.load(open(f"{ROOT}/config/{env}.json"))
    try: secrets = resolve_secrets(cfg)
    except RuntimeError as e: return print(f"{tag}: FAILED ({e})")
    app = json.load(open(f"{rel}/app.json"))
    # 4. deployment test: the checks promotion cannot infer
    lo, hi = app["schema_supported"]
    if not lo <= db_schema <= hi:
        return print(f"{tag}: FAILED deployment test (app supports schema {lo}-{hi}, database is at {db_schema})")
    cur = f"{ROOT}/envs/{env}/current"
    prev = os.readlink(cur) if os.path.islink(cur) else None
    if os.path.islink(cur): os.remove(cur)
    os.symlink(os.path.abspath(rel), cur)
    json.dump({"env": env, "digest": digest, "config_sha": sha(open(f'{ROOT}/config/{env}.json','rb').read())[:12],
               "previous": prev and os.path.basename(prev)}, open(f"{ROOT}/envs/{env}/deployed.json", "w"))
    open(f"{ROOT}/evidence/{env}-{digest}.ok", "w").write("ok")
    print(f"{tag}: ok  version={app['version']} config={cfg['api_base']} secrets={secrets}")

if __name__ == "__main__":
    os.environ["DEMO_DB_PASSWORD"] = "placeholder-value"
    for e, base, lvl in [("dev","https://api.dev.example.test","debug"),("staging","https://api.staging.example.test","info"),("prod","https://api.example.test","warn")]:
        write_config(e, api_base=base, log_level=lvl, replicas=1 if e!="prod" else 3, secret_refs={"db_password":"env:DEMO_DB_PASSWORD"})
    v1 = build("1.0.0", [1, 1]); v1b = build("1.0.0", [1, 1])
    print("rebuilding identical inputs gives same identity:", v1 == v1b, v1[:12])
    print("\n-- promote in order, with one skipped step attempted")
    deploy("prod", v1, 1)
    for e in ORDER: deploy(e, v1, 1)
    print("\n-- config shape drift between environments")
    cfg = json.load(open(f"{ROOT}/config/prod.json")); cfg["feature_x_enabled"] = True; json.dump(cfg, open(f"{ROOT}/config/prod.json","w"))
    print("keys that differ from dev:", config_keys_agree())
    del cfg["feature_x_enabled"]; json.dump(cfg, open(f"{ROOT}/config/prod.json","w"))
    print("\n-- tampering with stored bytes")
    v2 = build("2.0.0", [2, 2])
    os.chmod(f"{ROOT}/store/{v2}.tgz", 0o644); open(f"{ROOT}/store/{v2}.tgz","ab").write(b"x"); 
    deploy("dev", v2, 1)
    shutil.rmtree(f"{ROOT}/store"); os.makedirs(f"{ROOT}/store"); v1=build("1.0.0",[1,1]); v2=build("2.0.0",[2,2])
    print("\n-- missing secret")
    del os.environ["DEMO_DB_PASSWORD"]; deploy("dev", v2, 1); os.environ["DEMO_DB_PASSWORD"]="placeholder-value"
    print("\n-- release v2 (needs schema 2) and the schema moves forward in prod")
    deploy("dev", v2, 2); deploy("staging", v2, 2); deploy("prod", v2, 2)
    print("\n-- rollback to retained v1 artifact after schema 2 migration")
    deploy("prod", v1, 2)
    print("\n-- a v1.0.1 that tolerates schema 1-2 (the compatible-change route)")
    v1p = build("1.0.1", [1, 2])
    for e in ORDER: deploy(e, v1p, 2)
    print("\n-- deployment records")
    for e in ORDER: print(e, open(f"{ROOT}/envs/{e}/deployed.json").read())

rebuilding identical inputs gives same identity: True d6d23958775a

-- promote in order, with one skipped step attempted
deploy prod    d6d23958775a: BLOCKED (no passing record for staging)
deploy dev     d6d23958775a: ok  version=1.0.0 config=https://api.dev.example.test secrets={'db_password': '<resolved, 17 chars>'}
deploy staging d6d23958775a: ok  version=1.0.0 config=https://api.staging.example.test secrets={'db_password': '<resolved, 17 chars>'}
deploy prod    d6d23958775a: ok  version=1.0.0 config=https://api.example.test secrets={'db_password': '<resolved, 17 chars>'}

-- config shape drift between environments
keys that differ from dev: {'prod': ['feature_x_enabled']}

-- tampering with stored bytes
deploy dev     0ebef42bc43d: REFUSED (stored bytes hash to 1afd7f65b470)

-- missing secret
deploy dev     0ebef42bc43d: FAILED (secret 'db_password' not available from env:DEMO_DB_PASSWORD)

-- release v2 (needs schema 2) and the schema moves forward in prod
deploy dev     0ebef42bc43d: ok  version=2.0.0 config=https://api.dev.example.test secrets={'db_password': '<resolved, 17 chars>'}
deploy staging 0ebef42bc43d: ok  version=2.0.0 config=https://api.staging.example.test secrets={'db_password': '<resolved, 17 chars>'}
deploy prod    0ebef42bc43d: ok  version=2.0.0 config=https://api.example.test secrets={'db_password': '<resolved, 17 chars>'}

-- rollback to retained v1 artifact after schema 2 migration
deploy prod    d6d23958775a: FAILED deployment test (app supports schema 1-1, database is at 2)

-- a v1.0.1 that tolerates schema 1-2 (the compatible-change route)
deploy dev     532a99f93683: ok  version=1.0.1 config=https://api.dev.example.test secrets={'db_password': '<resolved, 17 chars>'}
deploy staging 532a99f93683: ok  version=1.0.1 config=https://api.staging.example.test secrets={'db_password': '<resolved, 17 chars>'}
deploy prod    532a99f93683: ok  version=1.0.1 config=https://api.example.test secrets={'db_password': '<resolved, 17 chars>'}

-- deployment records
dev {"env": "dev", "digest": "532a99f9368399d88ec6e31be3d3f9b8165d173d63618d1fb2480484869ee582", "config_sha": "5d05f444bf77", "previous": "0ebef42bc43d88f0669636029ef86ebbe2a11c2f38f3a88b80d6ea933715beca"}
staging {"env": "staging", "digest": "532a99f9368399d88ec6e31be3d3f9b8165d173d63618d1fb2480484869ee582", "config_sha": "b7f5918f679c", "previous": "0ebef42bc43d88f0669636029ef86ebbe2a11c2f38f3a88b80d6ea933715beca"}
prod {"env": "prod", "digest": "532a99f9368399d88ec6e31be3d3f9b8165d173d63618d1fb2480484869ee582", "config_sha": "ba5467beb8ed", "previous": "0ebef42bc43d88f0669636029ef86ebbe2a11c2f38f3a88b80d6ea933715beca"}

Reading the output in order: rebuilding identical inputs returns the same identity, so the store keeps one file. Deploying to production first is blocked because no passing record exists for staging, and the same digest then passes through development, staging, and production. Adding a key to production’s configuration shows up as a shape difference from development. A stored file altered after publication is refused because its bytes no longer hash to the digest being promoted. An unresolved secret reference fails with the name of the secret and not its value. Version 2.0.0, which supports only schema 2, deploys once the database is at schema 2. Rolling production back to the retained 1.0.0 artifact then fails the deployment test even though the artifact is intact, because it supports only schema 1. A 1.0.1 build declaring schemas 1 through 2 deploys through every environment, and each deployment record holds the digest, a short configuration hash, and the previous digest. The model has one process, local files, and no concurrency, so it shows the order and content of the checks and not how a real platform behaves under failure.

A short procedure follows from all of this, and it applies whether the tools are a managed service or scripts.

1. Publish one artifact per commit to an immutable store and record its digest with the evidence for it.
2. Deploy to each environment by digest through the same process, and refuse any digest that lacks a passing record in the previous environment.
3. Keep configuration versioned and the same in shape across environments, supply credentials by reference, and record the configuration applied beside the digest.
4. Validate each deployment against identity, configuration, connections, and compatibility, and treat a platform’s “complete” status as one input.
5. Write down how the last environment before production differs from production, and verify those differences by other means.
6. Before each release, state the rollback target (artifact and configuration), the conditions under which it can run, and who decides.

Each item reduces a specific risk and none of them proves the next deployment will succeed. What they give you is the ability to answer, for any environment at any time: which artifact is running, which configuration it was given, what evidence existed when it was promoted, and what you would do to return to the previous state.

References

• Reproducible Builds, Definitions
• Reproducible Builds, Documentation index
• DORA, Deployment automation
• The Twelve-Factor App, III. Config
• Kubernetes Documentation, Images
• OCI Image Specification, Content Descriptors
• Reproducible Builds, SOURCE_DATE_EPOCH
• Kubernetes Documentation, Secrets
• GitHub Docs, Managing environments for deployment
• Kubernetes Documentation, Deployments