Dataset Publication Manifest

A dataset publication manifest identifies the exact files that a reader should treat as one published dataset version. It can include the table, run or snapshot ID, object versions, sizes, schemas, source references, and code version. A delivery manifest describes an incoming delivery; this publication manifest defines a consumer-visible output snapshot.

Write and validate a fresh immutable run before switching the publication reference. Readers load that reference once and follow only the selected manifest, instead of listing a prefix containing both complete and incomplete runs. An intentionally empty dataset can have an empty file list; it must still be a deliberate validated publication.

Reject duplicate file references that would count rows twice and references outside the declared dataset scope. In the main lab every part belongs to one table and run. Full table formats may legitimately reuse older immutable files, so their scope rules differ. Matching sizes and versions do not prove source completeness or validate row contents.

Retain files for current snapshots, retained history, and active readers. A new current pointer does not mean every reader abandoned the prior snapshot. A custom JSON manifest works only with readers implementing its protocol; ordinary query engines do not infer transaction semantics from the filename. Use an appropriate table protocol when multiple writers, partitions, and recovery need stronger coordination.

See Designing an Object-Storage Data Lake for worked examples.