Content-Addressed Identifier

A content-addressed identifier is a name computed from the thing it names. The name is a cryptographic hash, such as SHA-256, of the data’s bytes, usually written with the algorithm as a prefix (sha256:3b14…). Storage built on such names is called content-addressed: the store maps a hash to the bytes that produced it, and a lookup by hash can be checked by hashing what comes back.

This differs from a conventional name, which is a label that someone assigned and can later reassign. A path that gets overwritten or a tag such as latest answers “which one?” differently over time. A content-addressed identifier cannot be reassigned, because changing what it points to would change the hash, which makes it a different identifier.

What the construction gives you

• Immutability by construction. An identifier can only ever refer to one sequence of bytes. “Updating” an object means writing a new object under a new identifier.
• Integrity checking. Whoever receives the bytes can recompute the hash and compare it with the identifier, with no need to trust the server, the disk, or the transfer that delivered them.
• Deduplication. Identical content has an identical name, so a store keeps one copy however many references exist.
• Safe caching. A cache keyed by the hash never needs invalidation for that key, since the content cannot change.

A mutable name is still needed for the question “which one is current?” The usual design keeps both: immutable content-addressed objects underneath, and a small table of movable names (tags, aliases, branches) that point at identifiers. Records of what happened, such as “this model was evaluated”, should store the identifier, not the movable name. Container registries follow this split, as described in Image Tag and Digest, and Git names its objects the same way.

A hash identifies bytes, so the bytes must be well defined

Hashing is exact. Two objects that mean the same thing but are serialized differently get unrelated identifiers. For files that were produced once and stored, this is not a problem: the identifier of the stored bytes is simply what it is. The trouble starts when the identifier is computed from a logical value, such as a configuration, a manifest, or a set of weights, that is serialized each time it is hashed. Then the serializer becomes part of the identity, and any variation in it silently changes the name. The script below shows common sources of variation, each with the same logical content.

import hashlib, json, unicodedata
def h(b): return hashlib.sha256(b).hexdigest()[:12]

a = {"lr": 0.1, "epochs": 3}
b = {"epochs": 3, "lr": 0.1}
print("default dumps :", h(json.dumps(a).encode()), h(json.dumps(b).encode()))
canon = lambda o: json.dumps(o, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
print("canonical     :", h(canon(a)), h(canon(b)))

print("1 vs 1.0 json :", json.dumps({"x": 1}), json.dumps({"x": 1.0}))
print("-0.0 vs 0.0   :", json.dumps(-0.0), json.dumps(0.0), "equal:", -0.0 == 0.0)

s1, s2 = "caf\u00e9", "cafe\u0301"
print("NFC vs NFD    :", s1 == s2, h(s1.encode()), h(s2.encode()), "| after NFC:", unicodedata.normalize("NFC", s1) == unicodedata.normalize("NFC", s2))
print("LF vs CRLF    :", h(b"a\nb\n"), h(b"a\r\nb\r\n"))

The script ran under Python 3.12.14 on macOS 26.7.1. Its output:

default dumps : 2d4e26e41162 5c6f40768368
canonical     : 4b73727c43cb 4b73727c43cb
1 vs 1.0 json : {"x": 1} {"x": 1.0}
-0.0 vs 0.0   : -0.0 0.0 equal: True
NFC vs NFD    : False 850f7dc43910 81ef060bcd98 | after NFC: True
LF vs CRLF    : 911169ddaaf1 58055bdcc737

The same dictionary with its keys in a different order gets two different hashes under default JSON output and one hash under a canonical form (sorted keys, no optional whitespace, UTF-8). The integer 1 and the float 1.0 serialize differently although they compare equal in many languages, and so do -0.0 and 0.0. Two visually identical strings can differ in Unicode normalization form, and the same text file differs between Unix and Windows line endings. A canonical form has to fix each of these choices, and every system that computes the identifier has to apply the same rules. Other common sources are timestamps or host names embedded in archives and metadata, floating-point formatting that differs by library, and non-deterministic ordering of files in a directory or of entries in a set.

Canonicalization has a limit in the other direction: it decides what counts as “the same”. Folding a case or a normalization form into one identifier is correct only if the consumer never cares about the difference. When in doubt, hash less processed bytes and accept that equivalent objects get different names, which is an extra copy and not a wrong answer.

Identifying a composite object

A directory, a model with its tokenizer, or a dataset made of many files needs one identifier. The standard approach hashes a manifest that lists the members and their own identifiers, so the object’s identifier depends on every member, and changing one member changes it. Two details prevent mistakes. First, join the members unambiguously: hashing the concatenation of ab and c gives the same result as a and bc (both are SHA-256 of abc, ba7816bf8f01 in the shortened form used above), whereas prefixing each part with its length, or hashing a structured canonical manifest, keeps them apart (601d5476e2cc and 3fafa1cf2f19 in the same run). Second, include in the hashed input everything that should change the identity, such as the object’s kind, so that a model and a dataset with identical bytes are not given the same name by accident. Hash trees such as Merkle trees extend the idea so that large collections can be compared and verified in parts.

What an identifier does not tell you

• It does not say where the bytes came from or who vouches for them. A digest matches whatever was hashed, including something malicious. Trust comes from how the identifier was obtained, for example from a signed record, not from the hash.
• It does not capture inputs that were not hashed. The digest of a trained model does not name the data or code that produced it. Systems that need this fold the identifiers of the inputs into the identity, or record them beside it.
• It does not say whether two different identifiers behave differently. Weights re-saved with a new timestamp get a new identifier and the same behavior.
• It depends on the hash function. The guarantee that different content gets a different name rests on the collision resistance of the algorithm, which is why the algorithm is recorded in the identifier. MD5 and SHA-1 have known collision weaknesses and are poor choices for new identifiers.

References

• Python documentation, hashlib: secure hashes and message digests
• Python documentation, json: JSON encoder and decoder
• Unicode Standard Annex #15, Unicode Normalization Forms