Partition Pruning

Partition pruning excludes storage partitions that cannot satisfy a query’s partition predicates. A date-and-country directory layout can let a reader discard unrelated locations before opening their data files.

A query for one ingestion date can prune other ingestion dates. A query for event time needs a valid relationship between event time and ingestion time before using that shortcut. Late arrivals can make a convenient ingestion-date filter omit required events.

Pruning chooses candidate locations, not necessarily matching rows. Remaining files still need the applicable filters. Column projection and row-group statistics can reduce work further; their support and ordering depend on the reader.

Compare the query result against a trusted reference and inspect the plan and runtime metrics. A plan listing three candidate files does not prove three network requests or a particular billed byte count. Keep null values and interval boundaries explicit.

See External Tables and the Lake/Warehouse Boundary for worked examples.

Reference: Official documentation.


Discover more from Insightful Data Lab

Subscribe to get the latest posts sent to your email.

Similar Posts

Questions, corrections, or additional insights?

This site uses Akismet to reduce spam. Learn how your comment data is processed.