Dataset Datasheet

A dataset datasheet is a structured document that travels with a dataset and answers the questions a user needs answered before trusting it: why it was made, what it contains, how it was collected and labeled, what it may and may not be used for, and under what rights. Gebru et al. propose it by analogy with the datasheets of electronic components, which state operating ranges and known limits so that an engineer does not have to infer them from the part. Datasets lack an equivalent, so a user often finds out about a license restriction, a biased sampling frame, or a labeling shortcut only after a model has been trained on it.

Why it is needed

Fine-tuning and evaluation results can be no better than the data's provenance. Three kinds of failure trace back to undocumented data:

• Evaluation validity. If a user does not know which sources the data came from, they cannot check whether the test material overlaps with training or pretraining data, or whether labels came from a model that later serves as the system under test.
• Mismatch with deployment. A dataset collected from one population or period answers questions about that population. Without a record of what was sampled and what was excluded, the mismatch is invisible until deployment.
• Rights and privacy. Permission to use data depends on terms attached to each source, consent, and the presence of personal data. A license string copied from a hosting platform is a lead and not proof: Longpre et al., auditing more than 1,800 text datasets, reported license omission rates above 70% and error rates above 50% in license information on common hosting platforms. Documentation that traces each source to its original terms is the remedy.

What a datasheet contains

Gebru et al. organize questions along the dataset's life cycle, and a datasheet answers them in prose and lists. A working outline:

1. Motivation. The purpose of the dataset, the task it was built for, who created it, and who funded it.
2. Composition. What an instance is, how many there are, what each field means, whether the data is a sample of a larger set and how it was sampled, whether there are labels, errors, noise, or missing values, whether it includes sensitive or personal data, and whether there are known splits.
3. Collection process. How the data was acquired (direct observation, scraping, purchase, generation by a model), the time period, any consent obtained, and any preprocessing or filtering.
4. Labeling. Who labeled, with what instructions, how disagreements were resolved, and what the measured agreement was.
5. Uses. The tasks the data has been used for, the tasks it is suited to, and the tasks it should not be used for.
6. Distribution and maintenance. The license and terms for each source and for the whole, the access path, how errors are reported and corrected, and the versioning policy.

The value comes from specific answers. "Collected from public web pages" is less useful than "collected between two dates from pages permitted by their crawl rules, with duplicates removed by normalized hash and a three-word-shingle similarity threshold of 0.8."

How it is used

For a dataset a team builds, writing the datasheet during construction forces decisions to be made on purpose: the sampling frame, the labeling rule, and the rights check each become a written statement that can be reviewed. For a dataset taken from elsewhere, reading the datasheet (or writing one from what can be verified) is the due-diligence step before training. A minimal operational version is a manifest recording, per source, its origin, its license and where that license was verified, the collection date, the transformations applied, and the splits it entered, kept with the data and updated when the data changes.

What it is not

A datasheet documents; it does not clean the data, license it, or guarantee quality. It is also not a model card: a model card describes a trained model's performance and limits, whereas a datasheet describes the data that went into it. A datasheet is only as accurate as its author's knowledge, so statements that were not verified should be marked as unknown and not filled in.

References

• Gebru et al., "Datasheets for Datasets"
• Longpre et al., "The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI"