Software Supply Chain Controls

A delivery pipeline looks like a function from source code to a running service. In practice it is closer to a function from source code, a few hundred third-party packages, a base image, a set of build-time actions, and several credentials to a running service. Only the first input is written by the team that owns the service. Everything else arrives from someone else, is selected by a version string or a name, and is executed with real authority on a build agent or in production.

Software supply chain security is the discipline of controlling those inputs and the process that combines them. This post builds that discipline from the pipeline outward. For each stage it asks four questions: what enters, who controls it, what could go wrong, and which control answers that failure with evidence that can be checked. The five controls named in the title map onto those questions: provenance for dependencies and images, signing and verification, vulnerability handling, limits on privileged build actions, and exception handling for the cases where a rule cannot be met on time.

What this post assumes. From Progressive Delivery and Compatible Changes it assumes you know that a change reaches production gradually, that a rollout can be stopped or reversed, and that changes must stay compatible while old and new versions coexist. Here that matters because a supply chain control that blocks a deployment is itself a change to the delivery path, and it must be introduced with the same care. From general computer science it assumes familiarity with cryptographic hash functions, public-key signatures, and package managers. Everything specific to supply chain security (provenance, attestations, signing identity, vulnerability scoring, exceptions) is explained below rather than deferred to the prerequisite.

What you should be able to do afterward. Identify the inputs of a pipeline and the trust each one receives; pin dependencies and images so that the bytes you tested are the bytes you ship; write down what a verifier must check before it accepts an artifact; turn a vulnerability feed into decisions rather than a backlog; reduce the authority that build steps hold; and record exceptions so that they end.

Where the examples come from. The pip resolution example and the toy verifier below were written for this post and run locally, and the output shown is the real output of those runs. The other small examples (tag movement, the gate rules) are explained as short reasoning or tables. They use throwaway keys, invented package names, and invented vulnerability records. Snippets for tools that were not installed or run for this post (cosign, a container registry, a Kubernetes admission policy, a GitHub Actions workflow) are marked as not executed, and their described behavior comes from the cited documentation.

Inventory the inputs before choosing controls

A supply chain control only makes sense relative to an input. The table lists the inputs of a typical containerized service pipeline. It is illustrative, not exhaustive, and the right column is the control family this post develops for each row.

Input	Selected by	Typical failure	Control family
Application source	Repository and branch	Unreviewed or attacker-authored change reaches the build	Source protection and review (outside this post), plus provenance recording the exact revision
Language packages	Manifest ranges, resolver rules	Name resolves to the wrong package, a release changes, a maintainer account is abused	Lock file, hashes, controlled indexes
Base image	Tag in a Dockerfile	Tag moves to different content	Digest pinning, provenance of the base
Build tooling and actions	Version tag of an action, a tool download	Mutable reference runs different code than reviewed	Pin to immutable identifiers, restrict permissions
Build environment	The CI platform	Build steps tamper with the output or leak credentials	Hosted, isolated builds; short-lived credentials
Output artifact	Registry tag	Artifact replaced after the build	Digest identity, signature, verification before deploy
Known flaws in any of the above	Public advisories	Deploying a component with a flaw that is being exploited	Inventory, matching, prioritization, gates

Two properties of the table guide the rest of the post. First, the controls are different for different inputs, so a single scanner or a single signature cannot cover the table. Second, a control that records evidence is only useful if some component later checks that evidence and has the authority to refuse. Recording and enforcing are separate work, and most of the failures described below happen in the gap between them. The trust boundary idea applies directly: each row is a place where content crosses from a party you do not control into a process you do, and the control is the decision about what the receiving side accepts.

Dependency and image provenance

What provenance means, and what it does not

In everyday use, provenance means a record of where something came from. In this domain it has a narrower meaning. Build provenance is a record of how an artifact was produced: which builder ran, from which inputs, with which parameters. The artifact is identified in that record by its digest, a cryptographic hash of its bytes, so that the statement binds to exactly one blob rather than to a name that can be reused.

Defined this way, provenance supports a concrete question about what a consumer can conclude. A matching provenance record lets a consumer compare how an artifact was built against what it expects. It cannot show that the source code is free of defects or malicious intent, because the build platform will faithfully record a build of whatever source it was given. Provenance changes the question from “where did this come from?” to “did this come from the source and process I approved?”, which is a question a machine can check.

Provenance formats are not interchangeable guarantees. The SLSA specification, version 1.1, defines a Build track with levels. Build L1 requires that provenance exists, generated automatically by the platform, which helps with debugging and analysis. L2 adds signed provenance generated by a hosted build platform, which the SLSA documentation describes as preventing tampering after the build. L3 adds hardened platform controls that keep runs from influencing each other and keep signing secrets away from user-defined build steps, which the documentation describes as preventing tampering during the build. The levels describe how hard the provenance is to forge, not how safe the software is. SLSA’s own verification guidance adds that L3 does not cover compromise of the build platform itself, for example by a malicious insider, so the choice of which builders to trust is a separate decision.

Pinning dependencies so that a name cannot change meaning

Package manifests usually declare ranges such as “any 1.x release”. A dependency lock file records one resolved graph, including indirect dependencies, so that the build uses what was reviewed instead of whatever resolves today. A lock file by itself fixes versions. It does not by itself prove that the bytes downloaded for a version are the bytes that were reviewed, because the same name and version could in principle be served by a different file. Hash pinning closes that gap by recording the expected digest of each file.

Dependency confusion is the failure that shows why. The pip documentation warns that adding an extra index does not give private packages priority over public candidates with the same name. When pip sees several candidates, it considers them together and normally selects the highest version that satisfies the requirement. If an internal package is called internal-util and an attacker can publish a public package with that name and a higher version, an unconstrained requirement may resolve to the attacker’s file.

The runs below show the selection behavior without touching a network or a real package index. A script written for this post builds three tiny wheels by hand: an “internal” internal-util 1.0.0, a “public” internal-util 9.9.9 standing in for an attacker’s package, and a second 1.0.0 with different bytes standing in for a same-version substitute. pip is then pointed at local directories with --no-index --find-links, a stand-in for configured indexes: pip treats all configured locations as one pool, which is the property that matters. The package contents are harmless strings. The runs used macOS, Python 3.12, and pip 26.2 in a fresh virtual environment with no network access. The helper script that writes a minimal wheel comes first, followed by the commands and the output they produced.

import zipfile, hashlib, base64, sys, os
name, ver, body, out = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
pkg = name.replace('-','_')
files = {
 f"{pkg}/__init__.py": body.encode(),
 f"{pkg}-{ver}.dist-info/METADATA": f"Metadata-Version: 2.1\nName: {name}\nVersion: {ver}\n".encode(),
 f"{pkg}-{ver}.dist-info/WHEEL": b"Wheel-Version: 1.0\nGenerator: handmade\nRoot-Is-Purelib: true\nTag: py3-none-any\n",
}
rec = []
for p,c in files.items():
    h = base64.urlsafe_b64encode(hashlib.sha256(c).digest()).rstrip(b'=').decode()
    rec.append(f"{p},sha256={h},{len(c)}")
rec.append(f"{pkg}-{ver}.dist-info/RECORD,,")
files[f"{pkg}-{ver}.dist-info/RECORD"] = ("\n".join(rec)+"\n").encode()
os.makedirs(os.path.dirname(out) or '.', exist_ok=True)
with zipfile.ZipFile(out,'w') as z:
    for p,c in files.items(): z.writestr(p,c)

python3 mkwheel.py internal-util 1.0.0 'def hello(): return "internal"' idx_internal/internal_util-1.0.0-py3-none-any.whl
python3 mkwheel.py internal-util 9.9.9 'def hello(): return "ATTACKER-CONTROLLED (demo)"' idx_public/internal_util-9.9.9-py3-none-any.whl
python3 mkwheel.py internal-util 1.0.0 'def hello(): return "same version, different bytes"' idx_public/other/internal_util-1.0.0-py3-none-any.whl

# 1. No constraint, two locations
pip install --no-index --find-links idx_internal --find-links idx_public internal-util

Output of the first run (the tail of pip’s output):

Processing ./idx_public/internal_util-9.9.9-py3-none-any.whl
Installing collected packages: internal-util
Successfully installed internal-util-9.9.9

The resolver did what it is documented to do. Nothing was misconfigured in the sense of a software defect, which is why dependency confusion is a design problem and not only an implementation bug. Adding a version range changes the outcome for this particular attack:

pip install --no-index --find-links idx_internal --find-links idx_public 'internal-util>=1,<2'

Installing collected packages: internal-util
Successfully installed internal-util-1.0.0

The range excluded 9.9.9. Two files in that pool were still named 1.0.0, and the requirement cannot tell them apart. In a follow-up run with both files offered, in either order of the two locations, pip happened to install the internal one and hello() returned internal. That outcome came from a tie between equal candidates, and the rule pip applies to such ties was not established here, so it is not something to rely on. That is the residual problem: a version pin identifies a name and number, not content. Hash-checking mode identifies content. The pip documentation describes it as an all-or-nothing mode that requires every requirement, including transitive dependencies, to be pinned with == and hashed. The next runs record the digest of the internal wheel and then offer pip first only the substitute, then both, and finally an unpinned requirement.

H=$(shasum -a 256 idx_internal/internal_util-1.0.0-py3-none-any.whl | cut -d' ' -f1)
echo "internal-util==1.0.0 --hash=sha256:$H" > req.txt

# A: only the same-version substitute is available
pip install --no-index --require-hashes -r req.txt --find-links idx_public/other

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    internal-util==1.0.0 from file:///.../idx_public/other/internal_util-1.0.0-py3-none-any.whl (from -r req.txt (line 1)):
        Expected sha256 3951cf33d3e24840672833d8351cac6ee41e9ab976d992fa5529cede875d81d9
             Got        56b43f6fd70623e6e5a2bee151058769b0003cc76b18de78632a10f1851c2d5e

With both files available (case B), pip installed the file whose hash matched and import internal_util; internal_util.hello() printed internal. With an unpinned requirement such as internal-util>=1 (case C), pip refused with “In –require-hashes mode, all requirements must have their versions pinned with ==”. The three outcomes show what hash pinning adds and what it demands. It makes the accepted set of files explicit, and it fails closed on anything outside that set. It demands that the lock be complete, which is why the mode is strict about unpinned and unhashed entries.

The limits are as instructive. A hash recorded at the moment of first approval trusts whatever was served at that moment. If the first download was already malicious, the hash faithfully pins the malicious file. Hash pinning therefore does not replace the decision about which origin may supply a name. The other measures that address dependency confusion act on origin: reserve or register the internal namespace publicly where the registry supports it, route builds through a proxy that binds protected names to approved upstreams and blocks fallback, and restrict the network destinations a build may reach. Each leaves a residue (new names, policy changes, bypass routes), so they are layered rather than alternatives. A recurring check is to list your internal package names and test, from a build environment, which origin actually wins for each one.

Resolution is also not execution. Selecting a source distribution may run its build backend on the build agent, while installing a wheel normally does not run build hooks, though a wheel’s code can run when imported. Which credentials are exposed depends on where that code executes. That connects dependency control to the later section on privileged build actions.

Identifying images by content

The same distinction between names and content applies to images. A tag and a digest are different kinds of identifiers. A tag is a name that can be moved to different content over time. A digest is derived from the content of a manifest or an image index. The OCI image specification describes a descriptor as a reference that carries a digest, in the form algorithm:encoded, plus the size, and says that implementations should recompute the digest of content from untrusted sources and compare it before using it. The Docker documentation describes the practical consequence: pulling by digest returns the same image each time, while a tag may return different content after someone rebuilds and reassigns it.

A short model makes the failure concrete. Push content A under the tag app:1.4 and record its digest. Later, rebuild and push content B under the same tag. The registry now resolves app:1.4 to B’s digest, while A’s digest still resolves to A’s bytes as long as the registry retains it. The toy registry below, a few lines of Python written for this post, keeps a table of blobs by digest and a table of tags that point at digests, and its output is from a local run.

# toy registry: a tag is a mutable pointer, a digest identifies bytes
import hashlib
blobs, tags = {}, {}
def push(tag, content):
    d = "sha256:" + hashlib.sha256(content).hexdigest(); blobs[d] = content; tags[tag] = d; return d
d1 = push("app:1.4", b"image bytes build #1")
pinned = d1
d2 = push("app:1.4", b"image bytes build #2 (rebuilt, same tag)")
print("tag app:1.4 now ->", tags["app:1.4"][:19], "...")
print("pinned digest   ->", pinned[:19], "...  still resolvable:", pinned in blobs)
print("same?", tags["app:1.4"] == pinned)

Output:

tag app:1.4 now -> sha256:49396f11d95d ...
pinned digest   -> sha256:8eeed1d64b44 ...  still resolvable: True
same? False

In this experiment the tag moved while the earlier digest kept resolving to the earlier bytes. Two conclusions follow, and they should be kept apart. First, if a Dockerfile says FROM base:1.4, two builds of the same commit on different days can start from different filesystems, so the build is not reproducible even if everything else is held constant. Second, a pinned digest also freezes the base against intended patches: a security update published behind the tag is not adopted until someone selects, tests, and promotes a new digest. Pinning converts silent change into explicit change, which is the desired property, but it makes the update process a real obligation. Automated update proposals that change the pinned digest through a reviewed pull request are one way to carry that obligation; the pull request is where the vulnerability and provenance checks of the next sections run.

If the digest refers to a multi-platform index, record which platform manifest was executed as well, because different architectures resolve to different manifests under one index digest. The image format and its layers are the subject of the next post in this series.

A component inventory answers questions that provenance does not

A software bill of materials (SBOM) lists the components inside an artifact and their relationships. CycloneDX, one of the formats, describes components, their direct and transitive dependencies, and can also carry vulnerability disclosure information. The inventory answers different questions than provenance does. Provenance asks how this artifact was built. The inventory asks whether this artifact contains component X at version Y, which running services contain it, and what changed between two releases.

Two cautions keep the inventory honest. First, its completeness depends on where it was generated. An inventory built from the manifest reflects declarations, one built from the finished image reflects what a scanner recognized in the bytes, and neither may see a library vendored into the source or a binary downloaded by a script. Second, the inventory is only useful when joined to deployment records. If you store inventories keyed by artifact digest and record which digest each environment runs, “which services contain this library?” is a join. Without the second table it is a manual search. Both are decisions for the platform, not features of a file format.

Signing and verification

What a signature adds

Digest pinning identifies content but says nothing about who produced it. A digital signature binds a statement to a key. An attacker who can write to a registry can push new bytes with a new digest, but cannot produce a valid signature without access to the signing key. The control therefore moves trust from “the registry holds this name” to “a key I trust signed this digest”.

That sentence hides two decisions. One is which keys or identities are trusted. The other is what the signer was vouching for. A signature over a digest only says the signer approved those bytes. Different signers may mean different things: a build system saying “I built this from this source” is a different claim from a release manager saying “this passed review”. A signature added at the end of a pipeline, by a step that does not know where the inputs came from, tells you little about the inputs. That is the difference the earlier sections tried to establish between provenance (how it was made) and a bare signature (who signed these bytes).

Keys and identities

There are two common ways to manage signing credentials. With a long-lived key, the signer holds a private key and the verifier holds the matching public key. Security then depends on protecting that key, rotating it, and revoking trust after a leak, and the key must be kept out of reach of the steps the build runs. With identity-based (“keyless”) signing, as described in the Sigstore documentation for cosign, an ephemeral key pair is created for each signing, a certificate authority called Fulcio issues a short-lived certificate that binds the public key to an identity proven through an OpenID Connect token, the private key is destroyed, and the signature event is recorded in a transparency log called Rekor. The signature for a container image is stored in the registry alongside the image.

Identity-based signing shifts the verifier’s job. There is no long-lived key to compare against, so the verifier must say which identity it accepts. The cosign documentation for keyless verification states that two options are required: the certificate identity and the certificate OIDC issuer. The commands have this form (not executed for this post, because cosign was not installed):

# Not executed here. Form taken from the Sigstore cosign documentation.
cosign verify registry.example.com/team/app@sha256:<digest> \
  --certificate-identity=<identity of the signer> \
  --certificate-oidc-issuer=<issuer URL>

The important idea is that an unconstrained “signed by someone” check proves almost nothing, because anyone can obtain a valid certificate for their own identity. The identity argument is the policy. For a CI-signed artifact the accepted identity is typically the specific workflow of a specific repository as asserted by the CI provider’s issuer, which makes the repository and workflow part of what is verified. Which identity strings your platform exposes depends on the CI provider, so confirm them from its documentation before writing the policy.

What a verifier must check

A statement about an artifact usually has the following structure in the in-toto attestation framework used by SLSA: a statement type, a subject list whose entries carry digests, a predicateType URI, and a predicate whose content depends on the type. The subject is matched purely by digest, which is how the attestation binds to an artifact. The statement is wrapped in a signed envelope. Verification is therefore several independent checks, and an implementation that does only the first accepts anything the attacker can get signed.

1. Authenticate the statement. Verify the signature, and check that the signer is authorized to speak for the claimed builder. A valid signature from an unrelated key is not acceptable.
2. Bind it to the artifact. The subject digest must equal the digest of the bytes about to be deployed.
3. Check expectations. SLSA’s verification guidance lists the builder identity, the canonical source repository, the build type, and the external parameters, and recommends rejecting unrecognized external parameters, because those are the inputs the requester controls.
4. Optionally recurse. Check the resolved dependencies the provenance lists, with the caveat that SLSA treats the completeness of that list as best effort.

The toy verifier below implements the first three checks. It is not a real attestation format: the statement is a small JSON document, and signatures use Ed25519 through the openssl command line, with demo keys created at runtime. It exists to make the policy explicit and to show which attacks each check catches. The output below is from running it locally with Python 3.12 and OpenSSL 3, one line per case.

"""Toy provenance verifier. Illustrative only; keys are throwaway demo keys."""
import hashlib, json, subprocess, tempfile, os

def sh(*a, **k): return subprocess.run(a, check=True, capture_output=True, **k)

def keygen(name):
    sh("openssl", "genpkey", "-algorithm", "ed25519", "-out", f"{name}.key")
    sh("openssl", "pkey", "-in", f"{name}.key", "-pubout", "-out", f"{name}.pub")

def sign(key, payload: bytes) -> bytes:
    with tempfile.NamedTemporaryFile(delete=False) as f: f.write(payload)
    out = sh("openssl", "pkeyutl", "-sign", "-inkey", key, "-rawin", "-in", f.name).stdout
    os.unlink(f.name); return out

def sig_ok(pub, payload: bytes, sig: bytes) -> bool:
    with tempfile.NamedTemporaryFile(delete=False) as p, tempfile.NamedTemporaryFile(delete=False) as s:
        p.write(payload); s.write(sig)
    r = subprocess.run(["openssl","pkeyutl","-verify","-pubin","-inkey",pub,"-rawin","-in",p.name,"-sigfile",s.name], capture_output=True)
    os.unlink(p.name); os.unlink(s.name); return r.returncode == 0

# Verifier's protected expectations: which key speaks for which builder, and what a good build looks like.
TRUSTED = {"https://builder.example/ci": "builder.pub"}
EXPECT = {"source": "git+https://git.example/team/app", "build_type": "https://builder.example/build/v1",
          "allowed_params": {"workflow", "ref"}}

def verify(artifact: bytes, statement: bytes, sig: bytes):
    st = json.loads(statement)
    b = st["predicate"]["builder"]
    if b not in TRUSTED: return "REJECT: unknown builder"
    if not sig_ok(TRUSTED[b], statement, sig): return "REJECT: bad signature for claimed builder"
    if st["subject"]["sha256"] != hashlib.sha256(artifact).hexdigest(): return "REJECT: subject digest does not match artifact"
    p = st["predicate"]
    if p["source"] != EXPECT["source"]: return "REJECT: unexpected source " + p["source"]
    if p["build_type"] != EXPECT["build_type"]: return "REJECT: unexpected build type"
    extra = set(p["external_parameters"]) - EXPECT["allowed_params"]
    if extra: return "REJECT: unrecognized external parameters " + str(sorted(extra))
    return "ACCEPT"

def statement(artifact, source="git+https://git.example/team/app", params=None, builder="https://builder.example/ci"):
    return json.dumps({"subject": {"sha256": hashlib.sha256(artifact).hexdigest()},
      "predicate": {"builder": builder, "source": source, "build_type": "https://builder.example/build/v1",
                    "external_parameters": params or {"workflow": "release.yml", "ref": "refs/tags/v1.2.0"}}},
      sort_keys=True).encode()

keygen("builder"); keygen("rogue")
art = b"app-binary-v1.2.0"
cases = []
s = statement(art);                                   cases.append(("good build, signed by builder key", art, s, sign("builder.key", s)))
cases.append(("artifact swapped after the build", b"app-binary-EVIL", s, sign("builder.key", s)))
s2 = statement(art);                                  cases.append(("signed by a different key than the builder's", art, s2, sign("rogue.key", s2)))
s3 = statement(art, source="git+https://git.example/fork/app"); cases.append(("built from a fork (valid signature)", art, s3, sign("builder.key", s3)))
s4 = statement(art, params={"workflow":"release.yml","ref":"refs/tags/v1.2.0","extra_script":"curl ..."}); cases.append(("unexpected build parameter (valid signature)", art, s4, sign("builder.key", s4)))
s5 = statement(art, builder="https://other.example/ci");      cases.append(("unknown builder identity", art, s5, sign("builder.key", s5)))
for name, a, st, sg in cases: print(f"{name:52s} -> {verify(a, st, sg)}")

Output:

good build, signed by builder key                    -> ACCEPT
artifact swapped after the build                     -> REJECT: subject digest does not match artifact
signed by a different key than the builder's         -> REJECT: bad signature for claimed builder
built from a fork (valid signature)                  -> REJECT: unexpected source git+https://git.example/fork/app
unexpected build parameter (valid signature)         -> REJECT: unrecognized external parameters ['extra_script']
unknown builder identity                             -> REJECT: unknown builder

Read the fourth and fifth rows carefully. The fork build and the extra-parameter build both carry valid signatures from the trusted builder. A verifier that stopped after the signature check would have accepted them. They are rejected only because the verifier holds expectations about source and parameters that the build cannot change. This is the practical meaning of the advice that expectations must be protected: if the same pipeline that produces the artifact could also edit the EXPECT table, the check would verify whatever the attacker wanted. The expectations, the trusted keys or identities, and the verifier code should live where build steps cannot write.

Where to verify

Verification can run at three places, and they catch different failures. At build time it gives early feedback to developers, but it checks only what the build saw. At registry ingestion it can keep unacceptable artifacts out, but it cannot see tampering that happens after ingestion or in transit. At deployment it checks the artifact that is about to run. In Kubernetes, the API server runs admission control after authentication and authorization and before an object is persisted. Admission can be extended by validating webhooks and by built-in declarative ValidatingAdmissionPolicy, so a cluster can refuse to create a workload whose image reference does not satisfy a policy. Which of these hooks supports your signature or attestation format depends on the tools you install, and that was not tested for this post.

Defense in depth here means verifying at more than one point, with the deployment check as the one closest to execution, and protecting the policy and verifier from the build. Be precise about what admission does not do: it checks requests at creation or update time. It does not check images already running, and it applies only to requests that pass through it, so bypass routes such as an emergency credential that can disable the webhook need their own control. Failing closed on verifier outage is safer than failing open but can block recovery, so decide the outage behavior deliberately and rehearse it, a topic that returns in the exception section.

Keep the claims modest. A verified signature and matching provenance show that the expected builder built the expected source in the expected way. They do not show that the source is correct, that its dependencies are free of flaws, or that the build platform itself was uncompromised.

Vulnerabilities

From advisories to findings

Vulnerability handling begins with matching: an advisory names a component and affected versions, and the inventory says which artifacts contain that component. One advisory therefore becomes many findings, one for each artifact or deployment that contains the component, and each finding has its own exposure. Decisions are made about findings, not about advisories.

The scoring information attached to an advisory answers different questions, and mixing them is the most common reason a backlog cannot be ordered. CVSS expresses the technical severity of a flaw under a reasonable worst case; the FIRST user guide for version 4.0 has a section titled “CVSS Base Score (CVSS-B) Measures Severity, not Risk” and recommends adding threat and environmental metrics. EPSS estimates the probability that exploitation will be observed in the next thirty days, and its FAQ states that it does not measure impact or account for environment-specific context, so it is not a complete risk score. CISA’s Known Exploited Vulnerabilities catalog lists vulnerabilities for which there is evidence of exploitation in the wild; CISA describes it as the authoritative source of such vulnerabilities and invites organizations to use it in prioritization. Severity, likelihood, and observed exploitation are three inputs. Priority is something your organization computes after adding exposure (is the vulnerable code reachable and is the service reachable) and what the system holds.

Reachability is the hard input. An inventory match shows that a component is present. It does not show that the vulnerable function is called or that the affected feature is enabled. The VEX (Vulnerability Exploitability eXchange) idea, supported by formats such as CycloneDX, is a way to record that a product is or is not affected by a given vulnerability and why. A VEX statement is an analysis result with assumptions, and it should be revisited when the code or configuration changes. Treat “not affected” as a claim that needs evidence, not as a way to clear a dashboard.

A gate that uses more than one input

A gate decides whether a build or promotion may proceed. A threshold on severity alone is easy to audit and detached from exposure. The example policy below combines the inputs. The findings, identifiers, and dates are invented, the rules are an example and not a recommendation of specific numbers, and the outcomes follow directly from the rules.

Finding	Inputs	Rule that applies	Decision
ADV-0001	CVSS 9.8, not known exploited, reachability unknown, fix exists	Block at severity 9.0 or higher unless known unreachable	Block
ADV-0002	CVSS 5.3, known exploited, reachable	Block anything known exploited	Block
ADV-0003	CVSS 7.5, unreachable, no fix, exception valid to a future date	An unexpired exception allows the finding	Allow until the expiry
ADV-0004	CVSS 8.1, reachable, no fix, exception expired	An expired exception no longer applies; no other rule allows it	Block

The same rules, written as a small script with invented findings and dates, produce the decisions in the table. The script is an illustration of the rule structure and not a recommended implementation. Its output is from a local run with the date fixed in the script.

"""Illustrative vulnerability gate. All findings, IDs and dates are invented for this demo."""
import datetime as dt
TODAY = dt.date(2026, 3, 1)
findings = [  # one row per (advisory, artifact digest, component) instance
  {"id": "ADV-0001", "component": "libfoo 2.3.1", "cvss": 9.8, "kev": False, "reachable": "unknown", "fix": "2.3.2"},
  {"id": "ADV-0002", "component": "libbar 1.0.4", "cvss": 5.3, "kev": True,  "reachable": "yes",     "fix": "1.0.5"},
  {"id": "ADV-0003", "component": "libbaz 0.9.0", "cvss": 7.5, "kev": False, "reachable": "no",      "fix": None},
  {"id": "ADV-0004", "component": "libqux 3.1.0", "cvss": 8.1, "kev": False, "reachable": "yes",     "fix": None},
]
exceptions = {  # finding id -> record
  "ADV-0003": {"expires": dt.date(2026, 6, 1), "approver": "risk-owner", "reason": "not reachable; no upstream fix", "ends_if_kev": True},
  "ADV-0004": {"expires": dt.date(2026, 2, 15), "approver": "risk-owner", "reason": "no upstream fix", "ends_if_kev": True},
}
def decide(f):
    ex = exceptions.get(f["id"])
    if ex:
        if ex["expires"] < TODAY: return "BLOCK (exception expired %s)" % ex["expires"]
        if f["kev"] and ex["ends_if_kev"]: return "BLOCK (exception ended: exploited in the wild)"
        return "ALLOW (exception until %s)" % ex["expires"]
    if f["kev"]: return "BLOCK (known exploited)"
    if f["cvss"] >= 9.0 and f["reachable"] != "no": return "BLOCK (severity %.1f, reachability %s, fix %s)" % (f["cvss"], f["reachable"], f["fix"])
    return "ALLOW (below gate; tracked)"
for f in findings: print(f["id"], f["component"], "->", decide(f))

Output:

ADV-0001 libfoo 2.3.1 -> BLOCK (severity 9.8, reachability unknown, fix 2.3.2)
ADV-0002 libbar 1.0.4 -> BLOCK (known exploited)
ADV-0003 libbaz 0.9.0 -> ALLOW (exception until 2026-06-01)
ADV-0004 libqux 3.1.0 -> BLOCK (exception expired 2026-02-15)

The rows illustrate four situations. ADV-0001 is severe, a fix exists, and reachability is unknown, so the example policy treats unknown as not safe. ADV-0002 has a modest severity score but is known to be exploited, so it blocks regardless of the score. ADV-0003 has no upstream fix and is judged unreachable, so a recorded exception allows it for a limited time. ADV-0004 has an exception that has lapsed, so it blocks again. The gate is a release gate in the sense that it produces a decision from evidence at a fixed point in the pipeline.

A gate also has failure modes of its own. A scanner outage can be read as “no findings”. Newly published advisories can make an already-released artifact fail the gate tomorrow, so scanning only at build time leaves deployed artifacts unwatched; a periodic re-evaluation of stored inventories against new advisories closes that gap. And a gate that blocks everything for weeks will be bypassed, which is why an exception path is a requirement of a workable policy rather than a loophole in it.

Privileged build actions

What is privileged in a build

A build step is privileged when it can do something an unreviewed change should not be able to do: read secrets, write to the registry, sign artifacts, change infrastructure, or deploy. A pipeline run holds these capabilities through credentials injected into the environment, the default token of the CI platform, a signing identity, and the network position of the build agent. The attack surface is everything that can influence what a privileged step executes: workflow definitions, third-party actions, dependencies that run build hooks, pull request content, caches, and artifacts from other jobs.

Least privilege applied to a pipeline means three separate things. Give each job only the permissions its task needs. Make credentials short-lived and obtainable only by the jobs that need them (see secrets management). And keep untrusted code out of jobs that hold privileges.

A public case: a mutable action reference

In March 2025, CISA published an alert about the compromise of a widely used third-party GitHub Action, tj-actions/changed-files, tracked as CVE-2025-30066. According to the alert, the malicious change could disclose secrets from workflow runs, including access keys, personal access tokens, npm tokens, and private keys, and the alert recommended auditing affected repositories, checking public logs for exposed credentials, rotating all identified secrets, and updating to a fixed version. It also states that the issue was likely related to the compromise of another action. CISA’s catalog of known exploited vulnerabilities later listed the issues. These are facts from the alert; the reading that follows is interpretation.

Interpretation: workflows that referred to the action by a version tag ran whatever code the tag pointed to when the workflow executed, and the maintainers’ account decides where a tag points. A reference to a full-length commit SHA does not move. GitHub’s hardening guidance states that pinning to a full-length commit SHA is currently the only way to use an action as an immutable release and explains that an attacker would need a SHA-1 collision on a valid Git object to replace it. Pinning does not make an action safe, because the pinned code may itself be malicious, and it freezes the action against fixes, so the same update process described for image digests applies. But it turns a silent change into a reviewable one. The incident also shows why the response is a credential problem and not only a code problem: once secrets have been exposed, the repair is rotation, which is faster when credentials are short-lived and narrowly scoped.

Reducing what a step can do

The following workflow fragment shows the controls as configuration. It was written for this post and not executed, so treat it as a sketch of the documented mechanisms rather than a tested pipeline. The action reference and the identifiers are placeholders.

# Sketch only. Not executed. Names, SHAs, and permissions are placeholders.
name: build
on:
  pull_request:          # untrusted code runs here WITHOUT secrets
  push:
    branches: [main]

permissions:
  contents: read         # default for every job: read-only

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-length-commit-sha>   # pinned, reviewed
      - name: Use an untrusted value safely
        env:
          TITLE: ${{ github.event.pull_request.title }}   # via env, not inline in the script
        run: ./scripts/check_title.sh "$TITLE"

  publish:
    if: github.ref == 'refs/heads/main'
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write    # only this job may request an identity token for signing/deploy
    steps:
      - uses: actions/checkout@<full-length-commit-sha>
      - run: ./scripts/build_and_push.sh   # pushes by digest, then signs the digest

The mechanisms in the sketch correspond to the documented guidance. The default token permission is read-only for repository contents and is raised per job. Untrusted values such as a pull request title are passed to scripts through an intermediate environment variable instead of being interpolated into the script text, which is the guidance’s defense against script injection. Jobs that handle pull requests from forks do not receive publish rights. GitHub’s guidance also warns that the pull_request_target and workflow_run triggers expose the repository when combined with checking out untrusted pull request code, because those workflows run with elevated privileges; it recommends using them only when the privileged context is needed.

Structure matters as much as syntax. The build job that runs repository code and resolves dependencies should not hold the credential that deploys to production. A separate release step with a different identity can verify the artifact, as the verifier above does, before using that credential. Separate identities are not sufficient if one attacker-controlled change can modify both the build and the release steps, so protect the release definition with its own review rule. Where an organization requires that the person who writes a change cannot approve its release, separation of duties names that control, but it is outside the scope of what the pipeline configuration alone can guarantee.

Finally, check what the build can reach. A build that needs only an approved package proxy and the registry should not have open outbound internet access, and a build agent should not retain credentials or caches between runs for unrelated repositories. These restrictions are verifiable by test: run a deliberately forbidden fetch from a build in a controlled environment and confirm it fails.

Exception handling

Every control above will eventually produce a result that cannot be acted on in time: the fix does not exist, the upgrade breaks a compatible interface, the signing service is down during an incident, or a legacy service cannot meet the provenance rule. Without a defined path, teams create their own: they disable the gate, add an allowlist entry with no owner, or move the work to a pipeline that is not checked. A vulnerability exception is the controlled version. It records an authorized decision to defer remediation past its standard window so the scope, reasoning, residual risk, and duration are reviewable.

A usable exception record has the following fields, each answering a question a later reader will ask.

• The finding and its scope. Which artifacts and deployments, not only which advisory.
• Why remediation is not happening now. A reason such as “no upstream fix” or “scheduled retirement”. “Low priority” restates the decision.
• The exposure assessment. What an attacker would need to reach and what the system holds.
• Compensating controls and their verification. A control that has been configured but never tested has an unknown effect.
• The approver and their authority. The person who requests an exception is not automatically the person who may accept the risk; check delegated authority and any required independent review.
• An expiry. After the expiry date the exception is void and the finding is due; it is not a reminder to look again.
• An early-termination trigger. Evidence of exploitation in the wild is the usual one, so the exploited-vulnerability catalog belongs in the exception’s own conditions.

The gate example above encodes the last two fields. ADV-0003’s exception is honored until its date and would end earlier if the finding appeared in a catalog of exploited vulnerabilities, and ADV-0004’s lapsed exception makes the gate block again. Those behaviors are what separate an exception from a permanent allowlist. The same rule applies to the other controls: a bypass of the signature check during an outage should be an explicit, time-limited, logged state that someone approved, not a configuration flag that nobody remembers to restore.

Exceptions fail in predictable ways. Renewals become automatic and the original reasoning is never re-examined; renewal should require a fresh exposure assessment, not a signature on the old one. Compensating measures are assumed to work. Acceptance is granted by someone without authority for the consequence. And repeated exceptions in one area can hide a capacity problem such as missing maintenance windows. Useful review measures are the age distribution of open exceptions, the count of expired exceptions still in force, and the share of controlled exceptions with current verification evidence. An age distribution is more informative than a count, because the long tail is where forgotten decisions sit. When you look at concentration by system or owner, compare it with system size and scan coverage before concluding anything about remediation capacity.

Putting the controls in order

The controls interact, so the order in which you introduce them matters. A sequence that works with the dependencies between them is shown below. It is a suggested order of work, not a maturity model, and each step produces evidence that the next step uses.

1. Inventory and identify. Record the digest of every artifact the pipeline produces and the resolved dependencies it used. Nothing later can be verified without a stable identity.
2. Pin. Commit lock files with hashes, pin base images and build actions by immutable identifiers, and route dependency resolution through controlled origins. Introduce update proposals at the same time so that pinning does not freeze patches.
3. Generate provenance on a hosted builder. Record builder, source, and parameters. At this point you can read the provenance but nothing enforces it.
4. Verify in report-only mode, then enforce. Run the verifier at deployment time and log the rejections it would have made. Resolve the legitimate failures, then switch to enforcing. This mirrors the progressive rollout of any change that can block deployments.
5. Add vulnerability gating with an exception path. Start by blocking only what is clearly indefensible, such as known-exploited findings, and expand as the data quality improves.
6. Reduce build privileges. Narrow default permissions, split build and release identities, shorten credential lifetimes, and restrict build network access.
7. Review the exceptions and the verifier itself on a schedule. Check who can change the expectations, the trusted identities, and the exception register.

Three limits deserve to be stated plainly. First, none of these controls reviews the code. A malicious change committed through the normal review process produces a validly signed artifact with genuine provenance. Second, the build platform is trusted: a compromised platform can produce misleading provenance, and higher SLSA levels reduce but do not remove that trust. Third, inventories and scans are incomplete in ways that depend on tooling, so a clean report is a statement about what the tool could see. Evidence from these controls supports decisions; it does not turn them into guarantees.

The arrangement is best judged by drills rather than configuration reviews. A refused deployment of an unsigned image in a non-production environment shows that the admission check is wired in. Resolving an internal package name from a build with the public index reachable shows which origin wins. A simulated disclosure of a build credential, timed from detection to rotation of everything that identity held, shows whether credential scoping works in practice. Each drill tests an assumption that a configuration review would only read. The next post in this series, Containers: Images, Isolation, and Resource Boundaries, explains what an image is made of and what a running container is and is not isolated from, which is the content that digests and signatures in this post point at.

References

• SLSA v1.1, Security levels (Build track)
• SLSA v1.1, Verifying artifacts
• Sigstore documentation, Cosign signing overview
• Sigstore documentation, Verifying signatures with cosign
• in-toto Attestation Framework, Statement specification
• pip documentation, Secure installs
• Open Container Initiative, Image specification: Content descriptors
• Docker documentation, Image tags and digests
• CycloneDX specification overview
• FIRST, CVSS v4.0 User Guide
• FIRST, EPSS FAQ
• CISA, Known Exploited Vulnerabilities Catalog
• CISA alert, Supply chain compromise of a third-party GitHub Action (CVE-2025-30066)
• GitHub Docs, Security hardening for GitHub Actions
• GitHub Docs, Using GitHub’s security features to secure your use of GitHub Actions
• Kubernetes documentation, Admission Controllers