Column Pruning

Column pruning removes columns that a query does not need from parts of its plan. With a suitable columnar reader, reducing the required schema can avoid reading unused column data.

Where the reader and optimizer support it, it is the cheapest of the read-reduction mechanisms, and cheap in a specific sense: it needs no statistics, no partition layout, no clustering, and no tuning. The engine works out which columns the query requires, and a columnar reader can skip the rest because each column’s bytes are stored separately. What defeats it is a query whose result really does require everything: an outermost SELECT * on a wide table returns every column, so the mechanism has nothing to remove, which is why it is usually lost by habit rather than by configuration. An asterisk inside a view or subquery is a different case. When the outer query uses only some of the view’s columns, optimizers can usually prune the rest, although whether a particular engine does so through a particular view is something its plan shows. Judge by the projection the scan performed, not by whether the SQL contains *.

It also acts on a different axis from the mechanisms it is often listed beside. Partition pruning and file skipping reduce how many rows are considered, by excluding whole locations or chunks; column pruning reduces how many columns are read from whatever remains. The two multiply rather than overlap: a query for four columns of one day reads roughly four columns of one day, which is why a narrow query on a wide, well-partitioned table can touch a tiny fraction of the stored bytes.

The final SELECT list is not the whole requirement. A query that returns amount but filters country still needs country to evaluate the predicate unless a supported source mechanism fully resolves that condition. Join keys, grouping keys, and expressions also contribute required inputs.

For a Parquet query that groups Korean orders by country and sums amount, an unused note can disappear from ReadSchema while country and amount remain. Check the scan’s required schema and any residual filter together; deleting the filter column blindly can change or break the computation.

A smaller projection does not always mean proportionally fewer bytes read. Row-oriented formats, metadata reads, compression units, reader support, and caching affect actual work. Use the plan to identify the intended reduction and storage metrics to assess realized savings.

See Spark DataFrames and Spark SQL for worked examples.

Further reading

• Apache Spark documentation