You Tested a Different Binary: Artifacts, Verification, and Promotion

A change passes every test in staging and breaks in production. The investigation finds that staging and production were built by separate pipeline jobs, four hours apart, and a transitive dependency published a new patch version in between. Both builds were green. Both came from the same commit. They were not the same software.

That failure is not exotic and it is not a testing problem. It is what happens when a pipeline is understood as a sequence of environments rather than as a machine that produces one thing. The distinction organizes everything else here: a delivery pipeline’s job is to build one artifact and then never build it again. Every later stage examines that same artifact, and every deployment installs it. Rebuild per environment and the tests describe something you did not ship.

What follows is what belongs in that artifact, what each verification stage actually proves, what promoting it between environments asserts, and how far reproducibility is worth chasing. Descriptions follow the Reproducible Builds definitions, the SLSA v1.1 specification, and DORA’s capability documentation, all checked in September 2026.

The artifact is the unit

A build artifact is the deployable output of a build — a container image, a jar, a wheel, a signed package. Published guidance treats it as the boundary between building and deploying: the inputs to an automated deployment are “packages created by the continuous integration (CI) process (these packages should be deployable to any environment, including production),” and the deployment process “should download the packages from an artifact repository.”

Read the parenthesis in that first quote as the whole design constraint. A package that can go to production unchanged is a package that cannot contain anything environment-specific. Which gives a simple test for whether your artifact is the right shape: if producing the production version requires a different build, it is not an artifact — it is a template.

In the artifact	Outside it
Compiled code and assets	Hostnames, endpoints, connection strings
Dependencies at resolved, pinned versions	Credentials and keys, always
The runtime it expects, where the format includes one	Feature flag states and per-environment limits
Its own identity — a version and the commit it came from	Anything that differs between staging and production

Identity in that last row is worth being strict about, because it is what makes every later question answerable. An artifact needs a name that no second build can ever reuse, and a recorded link back to the commit and the build that produced it. Version 2.4.1 alone fails this, because a rebuild can produce another 2.4.1 — which is exactly the situation the opening failure describes. Pair a human-facing version, for which semantic versioning is the usual convention, with an immutable identifier such as a content digest, and let the digest be what deployments refer to.

The cost of getting this wrong is not only the lost testing. It is that no question about production has a reliable answer — what is running, what changed, whether a rollback target still exists — because the thing running was never a durable object anyone kept.

What each stage proves

Once there is one artifact, a pipeline’s stages stop being a chronology and become a series of claims about it. Stating what each claim is prevents the common failure of having four stages that all check roughly the same thing while nothing checks integration.

Stage	The claim it establishes	What it cannot tell you
Build	The source compiles and packages, with dependencies resolved to specific versions	Whether the code is correct
Unit tests	Individual units behave as their authors intended	Whether the units work together, or against real dependencies
Integration tests	The artifact works against real collaborators — a database, a broker, a stub of a third party	Whether it behaves under load, or with production-shaped data
Static analysis and dependency scanning	No violation was detected in the inputs examined, under the rules and advisory data the tools had	Whatever the tools do not look for or cannot analyse, plus anything disclosed later
Deployment to an environment	The artifact starts, reaches what it needs, and answers — the smoke test	Whether the feature is right

The fourth row is worth dwelling on, because a green scan is routinely read as “no known vulnerabilities” and that is not what it says. OWASP’s own summary of static analysis tooling lists among its weaknesses that “current SAST tools are limited — they can automatically identify only a relatively small percentage of application security flaws,” that whole categories are hard to find this way (“authentication problems, access control issues, insecure use of cryptography”), that they are “frequently unable to find configuration issues, since they are not represented in the code,” and that many “have difficulty analyzing code that can’t be compiled.”

Dependency scanning has the analogous limits on its own axis: it finds what it can identify as a dependency, checked against the advisory data it has. A vendored copy, a statically linked library, a package manager it does not parse, or an ecosystem with thin advisory coverage produces a clean result for a reason that has nothing to do with the code being safe. So the claim a passing scan supports is “these rules found nothing in what they examined” — false negatives and unexamined scope are part of the result, not exceptions to it. Recording which tools ran, over what, is what makes the claim legible later.

The third column is the useful one. It is where you find out that nothing in your pipeline establishes the claim you actually care about, and it is how “add more tests” turns into a specific decision about which claim is missing. The distinction between the first two rows is worked through in unit and integration tests; both depend on having controlled inputs, which is what a test fixture provides, and on being able to state what the right answer is, which is the problem of a test oracle.

Order follows from cost and feedback speed rather than from thoroughness: run what is cheap and fast first, so that a broken build is reported in minutes rather than after a forty-minute suite. The constraint on unit tests is stated as a number — they “must also run in a few minutes or less,” with the reasoning spelled out: “if the automated unit tests take longer to run, developers won’t want to run them frequently. If the tests are run infrequently, then a test failure can originate from many different changes, making it hard to debug.” A slow suite does not merely delay feedback; it degrades the diagnostic value of every failure.

Two further rules make the difference between a pipeline people trust and one they route around.

Reliability is not negotiable. The requirement is that “when the tests pass we should be confident the software is releasable, and test failures should indicate a real defect. In particular, don’t tolerate flaky tests.” A test that fails one run in twenty teaches everyone to re-run the job, and from that moment the suite reports nothing — a real failure and a flake look identical. This is also why size is the wrong target: “a test suite of ten tests that is reliable, fast, and trustworthy is much better than a test suite of hundreds of tests that is hard to maintain and that nobody trusts.” Deleting an unreliable test is often an improvement, which is uncomfortable and true.

A red build is the top priority. Published practice includes “an agreement that when the build breaks, fixing it should take priority over any other work” — not out of tidiness, but because the pipeline is a shared instrument. While it is red, nobody else can tell whether their change is safe, so one broken build blocks everyone’s feedback and the cost grows with each commit stacked on top. The related discipline is continuous integration itself, whose rationale is stated plainly: “if something takes a lot of time and energy, you should do it more often, forcing you to make it less painful.”

Ownership decides whether either rule survives. The documented pitfall is “not having developers involved in testing,” with two named consequences — “test suites are frequently in a broken state,” since a code change requiring a test update waits on another team, and “developers write code that is hard to test,” because someone solving a problem without owning its verification has no reason to make it verifiable. And when something does reach production undetected, the prescribed response is to treat it as a gap in the instrument: “the pipeline must be improved, perhaps by adding or updating tests.”

Promotion, and what it asserts

Moving the same artifact through environments is environment promotion, and the published rules for it are short enough to adopt verbatim.

• “Use the same packages for every environment.” The reasoning is the point of this whole article: “keep environment-specific configuration separate from packages. That way, you know that the packages you are deploying to production are the same ones that you tested.”
• “Use the same deployment process for every environment, including production.” Which means the production deployment has been rehearsed every time anything reached any environment, instead of being the one path exercised least and under the most pressure.
• Make it on demand. “Allow anyone with the necessary credentials to deploy any version of the artifact to any environment on demand in a fully automated fashion” — with a test attached: “if you have to create a ticket and wait for someone to prepare an environment, you don’t have a fully automated deployment process.”
• Keep environments recreatable. “Make it possible to recreate the state of any environment from information stored in version control,” which is what makes a rebuilt environment equivalent to the one you tested against rather than merely similar — in its configuration. Version control holds the resources and their settings, not the data that accumulated in them, the state held in external services, or the secrets. So this makes environments reproducible; restoring production after a disaster additionally needs backups, a point-in-time restore, and a way to re-establish secrets and external dependencies.
• Record what is where. The recommended tooling “records which builds are currently in each environment.” Without that, the first question of every incident — what is actually running? — is answered by asking around.

Stated as a claim, a promotion says: this exact artifact passed the checks of every prior stage, and the only thing that changed is configuration. That claim is worth exactly as much as the second clause is true, and it is usually where it leaks. Production has data volumes staging does not, a load balancer staging does not, network policy staging does not, and a feature flag set nobody compared. None of that invalidates promotion — it bounds what promotion proves, which is why the last environment before production deserves an honest inventory of how it differs.

Two techniques address the residue directly. A release gate makes a specific, stated condition block promotion, which is useful when the condition is real and pure friction when it is ceremonial. And shadow testing answers the question staging cannot, by running the new artifact against production traffic without its results being used. Reversing a promotion is deployment of the previous artifact, which is only available if that artifact still exists and is still identifiable — one more reason the retention policy on an artifact repository is a reliability decision rather than a storage one.

Promotion also puts a floor under how small a change can be. If the pipeline runs in twelve minutes and needs no manual coordination, a one-line fix is worth shipping on its own; if it takes a day and three approvals, changes accumulate into batches whether anyone intended that or not — which is the economic argument in small batches, showing up as a concrete pipeline property.

How reproducible does a build need to be

Building once solves the problem of two builds differing. It does not tell you what any particular build was made of — which matters when a dependency turns out to be vulnerable, or when someone asks whether a binary in production came from the source you think it did.

The strongest form of an answer is a reproducible build, and the definition is precise: “a build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” Verification is mechanical — “verified by bit-by-bit comparison,” and this is “usually performed using cryptographically secure hash functions.”

Achieving that means removing the things that differ between two runs of the same build, and the list is longer than most people expect: embedded timestamps, the build path, locale and timezone, the iteration order of files or map keys, any generated random value, and anything read from the network without pinning. The definition treats the environment’s relevant attributes — “dependencies and their versions, build configuration flags and environment variables as far as they are used by the build system (eg. the locale)” — as part of what must be fixed, and adds a design instruction worth adopting even if you never pursue bit-for-bit output: “it is preferable to reduce this set of attributes.” Every input your build depends on is something that can change underneath you.

In practice there are three useful levels, and most organizations should be honest about aiming at the middle one.

1. Recorded provenance. You cannot rebuild the artifact identically, but you have a record of how it was produced. This is the cheapest level with real value, and it is what the SLSA build track’s first level formalizes: the build platform must “automatically generate provenance describing how the artifact was built, including: what entity built the package, what build process they used, and what the top-level input to the build were.” Read that scope precisely — top-level inputs, which is not the full transitive dependency set — and read the specification’s own note alongside it: “provenance may be incomplete and/or unsigned at L1. Higher levels require more complete and trustworthy provenance.” So L1 answers what the build recorded, not “exactly what went into it”; the stated focus is “mistakes, documentation,” and the level is “trivial to bypass or forge.” Identifying the whole dependency set is a separate piece of evidence — a full bill of materials — rather than something this level delivers.
2. Deterministic inputs. Every dependency is pinned and the build environment is itself a versioned artifact, so the inputs you control are the same on a rebuild. That is a claim about inputs, not a proof of equivalent behaviour: a build that embeds a timestamp, a random value, or a response fetched at build time can produce differing behaviour from identical pinned dependencies. Pinning narrows the set of uncontrolled inputs; it does not enumerate them. For most application teams this is the point where the remaining effort stops paying for itself.
3. Bit-for-bit reproducibility. An independent party can rebuild and compare hashes. This is what lets someone verify a binary without trusting the builder, which is why it matters most for widely distributed software and for anything where the build machine is part of the threat model.

Worth knowing: reproducibility and provenance are complementary rather than sequential. SLSA lists “whether reproducible builds are used” as an implementation choice each ecosystem makes, and its higher levels are about trust in the build platform rather than determinism of output — L2 adds signed provenance from a hosted platform against “tampering after the build,” and L3 hardens the platform against “tampering during the build,” requiring controls that “prevent runs from influencing one another, even within the same project” and keep signing material out of reach of “user-defined build steps.” That last requirement is a useful thing to check on your own pipeline today: can a build script read the credentials that sign its own output?

One reassurance about scope. The same specification treats builds “that are built and run on the same machine, such as unit tests” as needing no guarantees at all. The effort belongs on the artifacts you deploy and distribute, not on every build anyone runs — and the reason any of it matters beyond tidiness is the subject of software supply chain security.

Three things a pipeline should not do

• Rebuild for a later stage. Any job that compiles again after the artifact exists has broken the chain of evidence, however convenient it was. This includes the version that hides better: a deployment step that resolves dependencies at install time.
• Branch on the target environment. A build that behaves differently when it is “for production” means the production path is the least-tested one. Environment-specific behaviour belongs in configuration the artifact reads, not in the build that produces it.
• Accumulate stages nobody can explain. A pipeline grows a step per incident, and steps are never removed. If no one can say which claim a stage establishes, it is either duplicating another stage or protecting against something that no longer exists — and the cost is paid on every commit, which is the most expensive place to pay anything.

The check that catches all three is a question you can ask of any pipeline in five minutes: point at the artifact that is in production, and show where it was built and which stages examined that exact file. If the answer involves a rebuild, or a version number rather than a digest, or a stage whose purpose nobody remembers, you have found the work worth doing before adding anything new.

And keep the scope of the instrument in view. A pipeline that builds once, verifies a stated set of claims, and promotes the same bytes is the foundation that makes continuous delivery possible and the frequency and stability measured by the DORA metrics achievable. It still says nothing about whether the change was worth making — and the discipline of integrating work in small pieces so that the pipeline runs often, rather than on long-lived branches, is a habit the tooling supports but cannot supply, as anyone who has merged a three-week branch knows from branch and merge alone.

References: Reproducible Builds, Definitions; SLSA v1.2, Build Track Basics; OWASP, Source Code Analysis Tools; DORA, Deployment automation; DORA, Continuous integration; DORA, Test automation. All checked September 2026.