CI/CD Artifacts, Verification, and Promotion

Consider a fictional release that passes staging tests but breaks in production. Separate pipeline jobs build the two environments four hours apart, and an unpinned transitive dependency gains a patch release between them. Both builds pass their checks and use the same commit, but their dependency sets differ.

The failure comes from promoting a source revision while allowing the deployed software to change. A practical default is to build a release candidate once, retain it, and use that candidate in later checks and deployments. A rebuild needs its own identity and evidence; a shared commit or version label does not establish that its output matches the tested candidate.

The sections below explain the release artifact, the evidence from each verification stage, and the limits of promotion and reproducibility. They use the Reproducible Builds definitions, SLSA v1.2 Build Track, and DORA capability guidance.

The artifact is the unit

Here, a build artifact means a release output such as a container image, JAR, wheel, or signed package. DORA recommends making CI packages the inputs to automated deployment, together with deployment scripts and separate environment configuration. Keep the packages in an artifact repository so later stages can retrieve the candidate they are meant to examine.

Separating environment settings lets a release package move unchanged between staging and production. A package built specifically for production is still an artifact, but it is a separate candidate whose checks must cover that variant. Define the deployment unit explicitly: a container can bundle runtime libraries, whereas a wheel normally represents one Python distribution and requires separately installed dependencies.

In the artifact	Outside it
Compiled code and assets	Hostnames, endpoints, connection strings
Bundled dependencies, or a manifest identifying separately installed dependencies	Credentials and keys, always
The runtime it expects, where the format includes one	Feature flag states and per-environment limits
Its own identity — a version and the commit it came from	Anything that differs between staging and production

Record the release version, source revision, build invocation, and content digest. A version such as 2.4.1 is useful to people; semantic versioning describes compatibility intent for the declared public API. A digest identifies content: another build producing identical bytes has the same digest, while different output needs a different one. An immutable repository mapping from version to digest can also identify a candidate, provided deployment verifies the mapping and records the resolved content.

If a mutable label is the only deployment record, it may no longer identify the software that ran during an incident. Retaining the artifact and recording its digest makes comparison and recovery possible; runtime configuration and external state still need their own records.

What each stage proves

Once there is one artifact, a pipeline’s stages stop being a chronology and become a series of claims about it. Stating what each claim is prevents the common failure of having four stages that all check roughly the same thing while nothing checks integration.

Stage	The claim it establishes	What it cannot tell you
Build	The source compiles and packages, with dependencies resolved to specific versions	Whether the code is correct
Unit tests	The exercised units satisfy the assertions for the supplied inputs	Whether the units work together, or against real dependencies
Integration tests	The exercised component boundaries work under the test conditions; record real services and test doubles separately	Whether it behaves under load, or with production-shaped data
Static analysis and dependency scanning	No violation was detected in the inputs examined, under the rules and advisory data the tools had	Whatever the tools do not look for or cannot analyze, plus anything disclosed later
Deployment smoke test	The artifact starts, reaches what it needs, and answers — the smoke test	Whether the feature is right

A green static-analysis scan does not establish the absence of known vulnerabilities. OWASP describes limits on the proportion and kinds of flaws SAST can detect: authentication, access control, and cryptographic misuse are difficult to automate; configuration outside the code can be missed; and some tools struggle with code that cannot be compiled. These limits explain why the scan result needs its examined inputs and rules alongside it.

Dependency scanning has the analogous limits on its own axis: it finds what it can identify as a dependency, checked against the advisory data it has. A vendored copy, a statically linked library, a package manager it does not parse, or an ecosystem with thin advisory coverage produces a clean result for a reason that has nothing to do with the code being safe. So the claim a passing scan supports is “these rules found nothing in what they examined” — false negatives and unexamined scope are part of the result, not exceptions to it. Recording which tools ran, over what, is what makes the claim legible later.

The third column is the useful one. It is where you find out that nothing in your pipeline establishes the claim you actually care about, and it is how “add more tests” turns into a specific decision about which claim is missing. The distinction between the first two rows is worked through in unit and integration tests; both depend on having controlled inputs, which is what a test fixture provides, and on being able to state what the right answer is, which is the problem of a test oracle.

Run fast, inexpensive checks early so they can reject a candidate before a long suite runs. DORA recommends unit-test feedback within minutes: infrequent testing lets more changes accumulate between failures, making diagnosis harder. Slower integration and acceptance checks remain necessary for the boundaries they exercise.

Two further rules make the difference between a pipeline people trust and one they route around.

Flaky tests weaken the connection between a failure and the change under review, making genuine defects easier to dismiss. Investigate uncontrolled state, timing, isolation, and product concurrency problems before treating a failure as noise. Fix the cause or rewrite the check at a more stable level. If temporary quarantine is necessary, give it an owner and a repair deadline and make the coverage gap visible. Delete a test when its behavior is covered elsewhere or no longer required; deleting the only check does not restore confidence. DORA recommends curating reliable suites, and pytest describes these replacement and removal conditions.

DORA recommends giving a broken shared build priority because later failures become harder to attribute while the baseline is red. Teams can still obtain evidence from isolated checks, but they need a trustworthy integration result. Repairing or reverting the offending change restores that shared feedback loop, which is central to continuous integration.

Developers need responsibility and time to maintain tests alongside the code. DORA describes how separating that work can leave broken suites waiting on another team and allow hard-to-test designs to persist. When a defect escapes, identify the missing or ineffective check and improve the pipeline where that evidence can be obtained.

Promotion, and what it asserts

environment promotion moves a selected candidate through a sequence of environments. DORA recommends the following deployment practices.

• Use the same packages in each environment, with environment-specific configuration supplied separately. This preserves the connection between a package and the checks that examined it.
• Use a shared deployment process so earlier environments exercise the same basic steps. Record production-only branches, permissions, and integrations: sharing a script does not mean every branch has already run.
• Make authorized deployment available on demand through automation. Waiting for a ticket to provision or prepare an environment leaves part of that path manual.
• Version the environment definitions and deployment scripts so the configuration can be recreated. Those records describe resources and settings; disaster recovery also needs recoverable data, secrets, and external service state. Recreating the configuration does not recreate everything that happened inside it.
• Record the build and configuration deployed in each environment and retain deployment output for investigation. This gives incident responders a starting point for identifying what changed.

A promotion should identify the candidate and the checks it passed, together with their configuration and scope. It preserves artifact identity; it does not establish that configuration is the only environmental difference. Production may have different data volumes, infrastructure, network policy, traffic, and flag states. Inventory those differences and decide which need separate validation.

A release gate blocks promotion when a stated requirement is unmet. shadow testing adds evidence from copied operational inputs, with side effects disabled or isolated and its limits recorded. Reversing a promotion requires deployment of the retained previous artifact with suitable configuration and dependencies. That version must still work with the current schema, data, and external interfaces. Redeploying code does not undo data changes or actions already performed.

A fast promotion path with little coordination lowers the repeated cost of shipping a small change. A slow path can encourage batching, although delayed feedback and recovery risk also belong in that trade-off. This is the delivery cost discussed in small batches.

How reproducible does a build need to be

Building once solves the problem of two builds differing. It does not tell you what any particular build was made of — which matters when a dependency turns out to be vulnerable, or when someone asks whether a binary in production came from the source you think it did.

A reproducible build lets an independent party recreate the specified output bytes from the declared source, environment, and instructions. The Reproducible Builds definition includes all three inputs. Comparison, often using secure content hashes, establishes whether the specified outputs match.

Builds can vary with timestamps, absolute paths, locale, timezone, iteration order, randomness, and unpinned network inputs. Record the relevant environment attributes and reduce unnecessary inputs, as the Reproducible Builds guidance recommends. This makes unexplained differences easier to investigate.

Three practices answer different questions. Choose the investment from the artifact’s distribution, threat model, recovery needs, and cost of an undetected difference.

1. Recorded provenance. SLSA Build L1 records the builder, process, and top-level inputs. It helps investigate how an artifact was produced, but the record may be incomplete or unsigned and offers no tamper protection. It does not establish a complete transitive dependency inventory; that needs separate evidence such as a verified software bill of materials. Having provenance neither requires nor rules out reproducibility.
2. Controlled inputs. Pin dependencies and version the build environment to make selected inputs repeatable. A clock, random value, or external response can still change output or behavior. Input control is useful even without full reproducibility, but it is not evidence that all relevant inputs have been fixed.
3. Bit-for-bit reproducibility. An independent rebuild can check that declared inputs and instructions produce the distributed bytes. This reduces reliance on the original builder, while leaving trust in the source, toolchain, and verification environment. It does not authenticate the original builder’s identity, execution time, or approval history.

Reproducibility and provenance provide complementary evidence. SLSA Build L2 adds platform-generated, signed provenance; Build L3 adds stronger protections against interference between runs and prevents user-defined build steps from obtaining provenance signing secrets. Those requirements address trust in the build and its record, rather than requiring deterministic output.

SLSA places development or test builds consumed on the same machine, such as unit tests, in its L0 use case. L0 has no Build Track requirements; it does not mean test execution needs no protections. CI still needs trustworthy inputs and results, appropriate isolation, and protection of credentials and other jobs. Choose those controls for the threat model described in software supply chain security.

Three things a pipeline should not do

• Substitute an unverified rebuild. Prefer reusing the retained candidate. If a rebuild is necessary, compare its output to the tested artifact or validate it as a new candidate; provenance and approval requirements still apply to the new invocation. Installing a previously selected, pinned, hash-verified dependency set is different from choosing floating dependencies during deployment. Wheel installation may generate bytecode without changing the wheel itself: record the runtime and install conditions as part of the deployment evidence.
• Hide an environment-specific variant. Separate runtime settings where possible. When a different build is required, identify and test that candidate explicitly rather than carrying over another variant’s results.
• Keep unexplained stages indefinitely. Find the requirement, owner, and evidence behind a stage before deciding to retain, replace, or remove it. Missing documentation does not establish that the protection is redundant or obsolete.

Start an audit with the deployed artifact’s identity and trace it to build, test, installation, and approval records. Check any rebuild comparison and resolve version labels to immutable content. For each stage, state the claim it checks and the evidence it leaves.

And keep the scope of the instrument in view. A pipeline that builds once, verifies a stated set of claims, and promotes the same bytes is the foundation that makes continuous delivery possible and the frequency and stability measured by the DORA metrics achievable. It still says nothing about whether the change was worth making — and the discipline of integrating work in small pieces so that the pipeline runs often, rather than on long-lived branches, is a habit the tooling supports but cannot supply, as anyone who has merged a three-week branch knows from branch and merge alone.

References

• Reproducible Builds, Definitions
• SLSA v1.2, Build Track Basics
• OWASP, Source Code Analysis Tools
• DORA, Deployment automation
• DORA, Continuous integration
• DORA, Test automation
• PyPA — Binary distribution format
• pip — Secure installs
• pytest — Flaky tests
• SLSA v1.2 — Build: Provenance