Data Catalog

A data catalog organizes metadata about data resources so people can discover and assess them. Metadata is information describing a resource, such as its name, schema, owner, update time, or permitted use. The catalog usually points to resources rather than storing every underlying record itself.

Finding a table is only the beginning

Suppose a search returns two order datasets. One is a current operational snapshot; the other is a frozen monthly report. Descriptions, time coverage, definitions, and contacts help the user choose. A table name alone cannot establish which is suitable.

A catalog can connect business terms, quality observations, and lineage. W3C's DCAT provides a vocabulary for describing cataloged resources and exchanging their metadata; it does not require every organization to use one particular catalog product.

How an engine finds the data

A technical catalog, sometimes called a metastore, also supplies metadata to query engines. For an external orders table, an engine can resolve the table name to its schema, file format, storage location, and partition information. AWS Glue’s Data Catalog stores these descriptions. An engine uses them to interpret and locate the data, subject to its connector support and permissions. Some table formats instead require a reference to table metadata that identifies a committed version and its files.

Registering that entry does not copy the source rows or prove that a writer has finished all its files. A stale location or incompatible schema can make a valid catalog entry unusable; a directory may expose incomplete output unless publication is coordinated separately. Keep catalog changes consistent with data changes and account for metadata caches. Some catalog implementations participate in a table format’s atomic commit protocol, but a discovery entry alone provides no such guarantee.

Business discovery and engine metadata can live in the same product or in connected systems. Descriptions and ownership help a person choose a table; schema and location metadata help an engine read it. Keeping those records connected avoids presenting a well-described dataset whose executable definition points somewhere else.

Discovery is not authorization or certification

Seeing an entry does not mean the user may read its data. Access controls must still be enforced at the relevant systems. Metadata itself may also need restricted visibility.

A quality label can become stale. Check what was measured, when, and for which version. A catalog is useful when its records remain connected to responsible people and current evidence.

References

• W3C: Data Catalog Vocabulary
• AWS Glue: Data discovery and cataloging
• Apache Iceberg: Table specification