Partition Pruning
Partition pruning excludes storage partitions that cannot satisfy a query’s partition predicates. A date-and-country directory layout can let a reader discard unrelated locations before opening their data files.
A query for one ingestion date can prune other ingestion dates. A query for event time needs a valid relationship between event time and ingestion time before using that shortcut. Late arrivals can make a convenient ingestion-date filter omit required events.
Pruning chooses candidate locations, not necessarily matching rows. Remaining files still need the applicable filters. Column projection and row-group statistics can reduce work further; their support and ordering depend on the reader.
Compare the query result against a trusted reference and inspect the plan and runtime metrics. A plan listing three candidate files does not prove three network requests or a particular billed byte count. Keep null values and interval boundaries explicit.
See External Tables and the Lake/Warehouse Boundary for worked examples.
Reference: Official documentation.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.
