Apache Arrow

Apache Arrow defines typed columnar memory layouts for analytical processing. Arrays describe value buffers and null information; variable-length strings also use offsets. Nested values are represented with child arrays.

Compatible APIs can share existing buffers rather than build row objects. This requires compatible types, ownership, and buffer lifetime. A Parquet read generally decodes and decompresses data into arrays, so an Arrow result does not establish a zero-copy read.

Arrow IPC supports streams and seekable files for exchanging arrays. Feather V2 uses the IPC file format. Memory mapping an uncompressed compatible file can avoid an additional buffer copy; decompression or type conversion may allocate memory.

Choose the representation for the consumer. Typed array exchange and a multi-engine analytical archive have different needs. Validate values and inspect allocations when making a zero-copy claim; a successful round trip only checks the tested data.

See Data Formats: Parquet, ORC, Avro, and Arrow for worked examples.

Further reading

• Official format documentation