Delivery Manifest
A delivery manifest describes the files belonging to one declared delivery. It can name the source, dataset, business period, revision, exact file or object versions, byte sizes, row counts, and checksums. The producer publishes it through an agreed completion protocol so consumers know which input set to validate.
If a manifest lists three immutable files, accept that revision only after all three are available and validated. A partial directory listing or an old manifest next to an overwritten file is not the same evidence. Keep the checked bytes or exact versions for publication rather than reading a potentially changed path again.
A matching checksum verifies a relationship between bytes and the declared digest. It does not show that all intended business records were exported. A count calculated from the same incomplete export can agree too. Use independent source controls when available, and distinguish the producer’s manifest from the consumer’s load ledger.
Recognizing a delivery you already processed
Producers resend. A partner reruns an export, an operator retriggers a transfer, a retry lands after a timeout that had already succeeded. The manifest is what lets the consumer tell this is new from I have this, provided it carries an identifier that is stable across resends.
The identifier that works is the tuple that names the delivery rather than the transfer: source, dataset, period, and revision. Filenames and arrival times do not qualify — both change on a resend of identical content. With that tuple recorded on the consumer side, an arriving delivery falls into one of four cases.
| Situation | What it means | Action |
|---|---|---|
| Tuple not seen before, revision newer than the one applied | A new delivery | Validate and apply |
| Tuple not seen before, revision older than the one applied | A delayed earlier correction | Reject, or apply only where the period has no newer revision. Applying it overwrites the newer figures |
| Tuple seen, same checksum | A resend of what you already have | Acknowledge and do nothing |
| Tuple seen, different checksum | The producer changed the contents without changing the revision — a conflict, not an update | Reject and raise it. Applying either version silently makes the two sides disagree about what revision 2 contains |
The last row — same revision, different bytes — is the one worth enforcing rather than assuming, because it is a producer-side mistake that a consumer can detect and nobody else will.
The out-of-order row is the one idempotence does not cover, and it is worth being precise about why, because our replacement is idempotent is a common reason to skip the check. Applying the same revision twice is harmless. Applying revisions out of order is not: replace the period with revision 1 (100), then with revision 2 (80), then receive revision 1 again — each replacement is individually idempotent, and the period ends at 100. Idempotence makes a repeat safe; it says nothing about which version wins.
So the applied revision per period has to be stored durably and compared before applying, which makes the receipt record part of correctness rather than an optimization. And it has to be stored with the data: the familiar two-writes problem applies here too — apply the data, record the revision, and a crash between them leaves the next delivery of the same tuple looking new. Commit both in one transaction where they share a store, or keep the applied revision inside the data itself (a column on the period, or a pointer swapped atomically with the replacement) so that the comparison and the write cannot disagree.
The same applies to the conflict check: it needs the checksum of the applied revision retained, so it too is correctness rather than bookkeeping.
Further reading
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.