Apache Parquet Explained: Columnar Storage and File Format

Apache Parquet is a columnar file format used by data engineers, analysts, and platform teams to store large datasets for analytical queries. Its layout lets readers fetch selected columns and skip data that cannot match a filter, while compression and encoding reduce the amount of storage and I/O. This explainer follows a Parquet file from its footer to its pages, shows a small Python example, and distinguishes the file format from related systems such as Apache Arrow and Apache Iceberg.

Why Apache Parquet Is Being Talked About

Data lakes and analytical platforms often keep data in object storage, where files are inexpensive and many different engines can read them. That flexibility creates a practical question: how should a file be arranged so a query that needs a few fields does not have to read and decode every row? Apache Parquet addresses that problem with a column-oriented on-disk layout and a self-describing footer.

Parquet appears throughout data platforms because it is supported by tools such as Spark, Trino, DuckDB, and PyArrow. The Apache Parquet project describes it as an open-source column-oriented format designed for efficient storage and retrieval. It is not a new database or a complete data lake system; it is one interoperable layer used by those systems.

What Is Apache Parquet?

Apache Parquet is a binary file format for structured data. It stores records with a schema, organizes values by column within row groups, and records the layout and schema in metadata at the end of the file. A reader can inspect that metadata, identify the column chunks it needs, and avoid reading unrelated columns.

The official Parquet file-format documentation describes a file that starts and ends with the PAR1 magic bytes. Between them are column chunks followed by a footer containing file metadata and its length. Writing the footer last allows a writer to stream data into a file before it has finished collecting all of the locations and statistics needed to describe it.

Parquet supports primitive physical types, logical annotations such as strings and dates, and nested structures. The format is specified separately from any one query engine: the Apache Parquet format specification and its Thrift definitions describe how compatible readers and writers interpret the file. This makes Parquet portable, but it does not guarantee that every engine supports every optional feature or codec.

The Problem Parquet Solves

In a row-oriented text file such as CSV, fields for one record sit next to each other. That is convenient for writing and inspecting small datasets, but an analytical query that calculates an average over one numeric field may still need to read and parse every field in every row. Text also repeats delimiters and often repeats values that could be encoded more compactly.

Parquet groups values from the same column together. A query engine can read only the chunks for the columns it needs, decompress those chunks, and use statistics to skip some row groups. This can reduce storage and data transfer for scan-heavy workloads. The improvement depends on the query and the data: Parquet still has footer and decoding costs, and it is not automatically the best choice for tiny files, frequent single-row updates, or workloads that retrieve complete records one at a time.

How Apache Parquet Works

A Parquet file has a hierarchy. A file contains one or more row groups; each row group contains a column chunk for each column; each column chunk is divided into pages. The official specification repository summarizes these units and their roles in its format overview.

A reader normally seeks to the end of the file to read the footer, obtains the schema and chunk offsets, then reads the selected column chunks. On object storage, this often translates into range reads rather than downloading a whole file. If statistics show that every value in a row group falls outside a filter range, a compatible reader may skip that group. Statistics are an optimization, not an index with guaranteed coverage: they can be absent, limited, or insufficient for a particular predicate.

Values in a page can use different encodings. Dictionary encoding can replace repeated values with small integer references; run-length or bit-packing techniques can compact repeated or small integers; delta encodings can represent sequences of related values efficiently. A compression codec such as Snappy, Gzip, or Zstandard then reduces page data further. Encoding and compression are separate choices: encoding represents values efficiently, while compression finds additional redundancy in the encoded bytes.

Nested records need more than a flat sequence of values. Parquet uses definition and repetition levels to preserve optional and repeated fields without converting every nested record into a separate row. This is useful for event payloads and semi-structured records, but readers must agree on logical type and nested-field conventions to interpret values consistently.

Key Concepts and Design Choices

Row-group size. Larger row groups can provide more sequential I/O and better compression, but they make pruning coarser and may need more memory during writing or reading. Smaller groups can improve parallelism and filter selectivity, while increasing metadata overhead and creating more work for query planners. Choose based on measured query patterns and engine guidance rather than a universal row count.

Column projection and statistics. A query that selects a few columns can avoid unrelated chunks. Min/max statistics may support row-group pruning when values are ordered or reasonably selective. Sorting data by commonly filtered columns can make those summaries more useful, but it does not turn Parquet into a general-purpose indexed database.

Logical types and compatibility. The stored physical type and the intended meaning are not always identical. For example, a byte array annotated as a string has a different interpretation from arbitrary binary data. Timestamp units, decimal representations, nested schemas, and newer format features can expose differences between implementations. Test representative files with every reader and writer that will exchange them.

Files are not tables. A Parquet file describes data inside that file. It does not define a transaction across many files, choose which files currently belong to a dataset, or provide a catalog. Apache Iceberg’s table format and snapshot architecture explains the additional metadata layer that manages a table made from files such as Parquet. For a broader platform view, see building a data lake.

These related technologies solve different problems:

Apache Arrow is not Parquet in memory: it defines an in-memory layout and transport ecosystem, while Parquet defines persistent files. The Apache Arrow columnar memory and Flight RPC explainer covers that interchange layer.

Real-World Uses

Parquet is a good fit for event analytics, data warehouse exports, machine-learning feature datasets, and long-lived data lake files that are scanned by multiple tools. A pipeline might ingest events as JSON, validate and transform them, then write partitioned Parquet files for later SQL queries. A query engine can project only the requested fields and use partition or file statistics to limit its reads.

It is less suited to workloads that continuously update individual records in place. Applications commonly write new files and use a table format or database to coordinate changes. Parquet also does not replace a schema registry, access-control system, catalog, backup policy, or data-quality checks.

Getting Started: Write and Inspect a Parquet File

PyArrow provides Python APIs for constructing Arrow tables and reading or writing Parquet. The PyArrow write_table reference documents options such as row-group size, dictionary encoding, compression, and statistics. Create an environment and install PyArrow:

python -m venv .venv
source .venv/bin/activate
python -m pip install pyarrow

In Windows PowerShell, activate with .\.venv\Scripts\Activate.ps1. Save the following as parquet_demo.py and run python parquet_demo.py:

import pyarrow as pa
import pyarrow.parquet as pq

events = pa.table(
    {
        "event_id": [101, 102, 103],
        "region": ["west", "east", "west"],
        "latency_ms": [180, 95, 220],
    }
)

pq.write_table(
    events,
    "events.parquet",
    compression="zstd",
    row_group_size=2,
)

parquet_file = pq.ParquetFile("events.parquet")
print(f"Rows: {parquet_file.metadata.num_rows}")
print(f"Row groups: {parquet_file.metadata.num_row_groups}")
print(parquet_file.schema)

slow_events = pq.read_table(
    "events.parquet",
    columns=["region", "latency_ms"],
    filters=[("latency_ms", ">", 100)],
)
print(slow_events)

The example writes two row groups, prints metadata from the file footer, then asks PyArrow to read only two columns and filter by latency. A query engine may use row-group statistics to avoid reading groups that cannot match; it still has to evaluate matching rows, so this is not equivalent to a database index.

For a quick verification outside the script, python -m pip show pyarrow checks that the package is installed. You can also inspect the footer with python -c "import pyarrow.parquet as pq; print(pq.read_metadata('events.parquet'))". If the filter returns no rows, inspect the input values and predicate; if the file is unreadable in another engine, compare its supported logical types and compression codecs before changing the data.

For production, benchmark representative data and queries. Compare compression ratio, CPU use, bytes scanned, and task parallelism. Keep an eye on file and row-group sizes: a large number of tiny files adds listing and planning overhead, while oversized groups can limit pruning and parallel reads. When data is partitioned into directories, avoid high-cardinality partition keys that create many nearly empty files.

Common Misconceptions

“Columnar means every query is faster.” Columnar projection and compression help many scans, but query shape, file size, sort order, metadata, codecs, and engine implementation all matter. Point lookups and small records may be better served by another storage model.

“Parquet is a database or a data lake table format.” It is a file format. It does not coordinate concurrent writers or define a transaction spanning files. A catalog, query engine, and possibly a table format provide those other responsibilities.

“Every Parquet reader interprets every file identically.” Implementations can differ in support for newer logical types, nested schemas, codecs, and optional metadata. Interoperability comes from following the specification, but operational compatibility still needs to be tested across the tools in use.

Related Articles

• Learn how Apache Spark processes large datasets and writes Parquet output.
• See how Apache Iceberg organizes data files into tables.
• Compare Parquet with Apache Arrow’s in-memory columnar format.
• Review the wider storage and compute layers in building a data lake.

Changelog and Last Updated

• Changelog: Initial publication.
• Last updated: Reviewed against the Apache Parquet format documentation and PyArrow API reference.