Data Formats: Parquet, ORC, Avro, and Arrow

Rows, columns, and the work a reader avoids

A storage object contains bytes that a query engine must interpret. To total order amounts by country, a reader needs country and amount for many rows. In a CSV, fields from the same row are adjacent; locating selected fields usually requires scanning through intervening text. In a columnar file, values from the same column are grouped, so unrelated columns can be omitted from the read. CSV can still be compressed, parsed with an explicit schema, or stopped early for a limited query. It lacks Parquet’s built-in column-chunk directory and block statistics.

Parquet and ORC are columnar storage formats. Avro serializes records and is useful for events and sequential record processing. Arrow defines a columnar memory layout and also IPC stream and file formats. They belong in the same discussion because one pipeline can receive Avro records, write Parquet files, and process decoded columns as Arrow arrays. Choosing among them starts with the reader’s work: whole records, selected columns across many rows, or exchanging existing arrays.

Inside a Parquet file

A row group is a horizontal slice of the table. Within it, each primitive leaf column has a column chunk divided into pages. The footer describes the schema and chunk locations; chunk metadata can contain statistics. Minimum, maximum, and null counts are not guaranteed to be present for every column. A reader first obtains metadata, then locates relevant chunks. Fetching the footer still requires I/O, even when all data pages can be skipped. Parquet file layout

Encoding changes the representation of values before a compression codec reduces bytes. A dictionary stores distinct values and encodes references to them; repeated countries can use small indices. Run-length and bit-packed encodings represent repetitions and small integers, while delta encodings exploit differences between adjacent values. A writer may fall back from dictionary encoding when the dictionary grows. Seeing PLAIN alongside RLE_DICTIONARY does not by itself prove fallback: dictionary values also use PLAIN. Snappy and Zstandard then operate on encoded page data. Their benefits and CPU costs depend on values, ordering, settings, and implementation; no fixed compression ranking follows from the names. PyArrow writer options

Nested structures are represented through leaf columns plus definition and repetition levels. Definition levels distinguish absent optional ancestors or values; repetition levels preserve repeated-field boundaries. The levels themselves are encoded, rather than stored as two ordinary integers beside every value. A null list, an empty list, and a list containing a null element must survive a round trip as different values. Typed nested fields let a reader select supported subfields without parsing a JSON string. Unknown or original payloads can still be retained as text when replay or schema variability requires it. Nested encoding

PAR1
Row group 0: [order_id pages] [country pages] [amount pages] ...
Row group 1: [order_id pages] [country pages] [amount pages] ...
File metadata: schema, chunk offsets, optional statistics
Footer length | PAR1

What pruning proves

Partition pruning excludes files using partition values. Projection limits required columns, including any columns needed to evaluate a filter. Predicate pushdown passes filtering work to the data reader; statistics can rule out row groups, and supported indexes or bloom filters can narrow work further. These are complementary mechanisms, not a universal three-step execution order. Metadata overlap only identifies candidates. The reader still evaluates rows in candidate groups to return exact matches. Missing statistics require a conservative decision to retain a group. Parquet reading and filtering

For the half-open interval [20 January, 21 January), a group is impossible when its maximum is before the 20th or its minimum is on or after the 21st. A group spanning both dates remains a candidate even if no row actually matches. Sorting timestamps makes group ranges narrower. Take a hypothetical file of one million evenly spaced rows covering 81 days, starting at 00:00 on 1 January and sorted by time, with ten row groups of 100,000 rows. Each group then spans about 8.1 days: group 2 (rows 200,000 to 299,999) covers roughly 17 January 04:48 to 25 January 07:12, while group 1 ends before the 20th and group 3 starts after the 21st. Only group 2 is a candidate for the query on 20 January; with a different start time, a one-day window that crosses a group boundary can overlap two groups. If the same rows were assigned to groups in random order, each group would hold about 1,200 rows from that day and its minimum and maximum would lie near the ends of the 81-day range, so all ten groups would be candidates. That is a property of this arrangement, not a general rule, and it does not imply reading every column or every byte of the file.

Compressed chunk sizes in file metadata are logical sizes, not measured storage traffic. Footer reads, range coalescing, readahead, cache hits, and page-level decisions affect actual requests and bytes. Use query plans and storage or filesystem instrumentation to measure those. Sorting also costs compute and delays writes; choose it from recurring filter patterns rather than assuming any sort always pays.

Row groups, files, and schema changes

Larger groups can improve compression and reduce metadata overhead, but require more buffering and offer coarser skipping. File size separately affects task scheduling, listing, opens, and compaction. A 128 MB row-group target and files of a few hundred MB can be starting hypotheses for a workload, not universal defaults. PyArrow’s row_group_size is a row count, whereas ORC stripe_size is expressed in bytes. Measure compressed sizes, memory, scan selectivity, and latency with the actual engine before setting a production target.

Each Parquet file has its own schema. An explicit schema-unification policy by name can allow a new nullable channel column to be returned as null for older files. Reading both files under only the older schema instead returns all rows but hides channel. A rename becomes two separate names under this policy. A catalog or table format using stable field IDs can provide different rename semantics.

Type evolution also depends on the reader policy. PyArrow’s default unification rejects int64 versus string; permissive promotion can combine int32 and int64 into int64. Accepting a type is not proof that its business meaning is compatible: cents and dollars still need an explicit conversion. Define nullability, units, timestamp timezone and precision, field identity, and allowed migrations in the table contract. New nullable fields, deliberate backfills, and tested type widening are decisions rather than automatic permission to change any file. Schema unification rules

ORC: stripes and indexed column streams

ORC groups rows into stripes containing column streams, indexes, and a stripe footer. File-level metadata describes the file; row indexes and optional bloom filters help compatible readers skip smaller ranges. A bloom filter can rule out an absent value, but a positive result can be a false positive and still requires checking data. These features depend on writer settings and reader support. Hive’s transactional behavior requires its table and execution machinery; an ORC file alone is not a transaction system. ORC specification

Choose between ORC and Parquet using the engines that must consume the data, supported types, filter execution, and representative benchmarks. Matching the codec does not match all encodings, group sizes, or tuning. Projection and stripe pruning are separate behaviors to verify. ORC writer options

Avro: records and reader/writer schemas

Avro encodes a record’s fields in schema order. An object container file carries its writer schema and stores blocks of records with synchronization markers and an optional codec. Individual event payloads can also use Avro encoding, but their framing and access to the writer schema must be agreed with the consumer; a registry ID is an integration convention, not a mandatory prefix of every Avro datum.

During schema resolution the reader and writer schemas are matched. If the new reader expects a field absent from the writer schema, the reader’s default supplies its value; without that default, resolution fails. A nullable type alone does not supply a default. A reader default fills an absent field; it does not replace a null already written for that field. Reader-ignored fields, allowed type promotions, and aliases also have specified rules.

Record-oriented events suit a consumer that acts on each complete event. Avro container files can be split at block boundaries, but do not provide Parquet-style column chunks and min/max pruning. Selecting two fields still requires traversing records or encoded fields. Avro can be scanned analytically; repeated wide-table analytics are a reason to materialize a columnar copy. Avro specification

Arrow: arrays, IPC, and zero-copy conditions

Arrow arrays describe typed column buffers. Fixed-width values have a regular layout; strings use offsets and a data buffer; validity bitmaps represent nulls; nested arrays reference child arrays. Compatible libraries can share these buffers instead of constructing Python objects row by row. A Parquet read still decompresses and decodes storage encodings to produce Arrow arrays. That read is not automatically zero-copy. Buffer sharing depends on type, chunking, ownership, and the receiving API. Arrow columnar format

Arrow also defines IPC streams and seekable files. Feather V2 uses the IPC file format and supports optional LZ4 or Zstandard compression. An uncompressed memory-mapped file can expose compatible buffers without copying them into a second user-space allocation, but compression requires decompression and conversion can allocate new buffers. Flight transports Arrow batches between services; networking still has transport and serialization costs. Feather V2

Arrow files are not universally larger or faster, and their suitability for an archive depends on reader and storage requirements. For a shared analytical lake, reader interoperability and filtering commonly favor Parquet or ORC. For a typed cache or array exchange, IPC can avoid repeated conversion work.

Format	Layout	Typical fit
Parquet	Column chunks in row groups	Analytical scans; supported projection and statistics
ORC	Column streams in stripes	Analytical scans; supported stripe/row indexes
Avro	Records; container blocks or framed events	Event processing; writer/reader schema resolution
Arrow	Column buffers; IPC streams/files	Array processing, compatible exchanges and caches

Turn the comparison into a file contract

For the subscription-company example, retain received payloads when replay requires exact originals; write typed analytical tables as Parquet where the consuming engines support it. Test Zstandard against Snappy on representative data. Record row-group rows, observed byte sizes, partition keys, sort columns, compaction thresholds, and supported readers.

Use Avro for event contracts when producers and consumers agree on schema resolution and framing. Use Arrow IPC for compatible caches or hand-offs. Preserve stable nested fields as structs and lists, and retain unknown payloads separately when needed. Before dropping a costly reference column, check debugging, joins, replay, and audit requirements. Projection already lets unrelated queries skip that column.

File formats do not define which files comprise an atomic table snapshot. Concurrent publication, schema governance, updates, and garbage collection still require a catalog, manifest, or table layer. A codec choice cannot repair inconsistent snapshots.

References

• Parquet file layout
• PyArrow writer options
• Nested encoding
• Parquet reading and filtering
• Schema unification rules
• ORC specification
• ORC writer options
• Avro specification
• Arrow columnar format
• Flight
• Feather V2