Apache Iceberg Table Format and Snapshots Explained

Apache Iceberg is an open table format that lets analytics engines manage large datasets on object storage as reliable tables instead of collections of loosely coordinated files. Data engineers and platform teams use it to support consistent reads, concurrent writes, schema changes, and recovery without tying table data to one processing engine. This explainer follows a table from its catalog entry through its metadata and data files, then shows how to create a small local table with PyIceberg.

Why Apache Iceberg Is Being Talked About

Cloud object storage made it inexpensive to keep analytical data in open file formats such as Parquet. But a directory of Parquet files does not, by itself, define which files belong to a table, how a write becomes visible, or how a query should interpret a renamed column. As data lakes serve more teams and engines, those coordination details become important.

Apache Iceberg has gained attention as one of the open table formats used to add table-level metadata and transactions above files in a data lake. Its appeal is the separation of storage from compute: multiple compatible engines can work with the same table while the underlying data remains in formats such as Parquet. The Apache Iceberg project describes the format and its broad integration ecosystem.

What Is Apache Iceberg?

Apache Iceberg is a table format specification and project. It describes how table metadata, snapshots, manifests, and data files fit together so that an engine can read and update a table consistently. It is not a database server, query engine, object store, or file format that replaces Parquet.

Think of Parquet as a way to encode the rows in individual files. Iceberg adds the table’s inventory and history: which files make up a particular version, what schema applies, how those files are partitioned, and where the next update should be committed. Spark, Flink, Trino, and other compatible engines provide computation; a catalog helps them find a table’s current metadata.

The Iceberg documentation covers the format’s integrations and operational concepts. The table format specification defines the metadata and file structures that implementations use to interoperate.

The Problem Iceberg Solves

Without a table format, a data lake often relies on directory conventions and a metastore entry. Writers may create new files while readers list directories, and an overwrite may expose a mixture of old and new output if the process fails partway through. Concurrent writers can also interfere with one another, while a renamed column can be mistaken for a different column if readers identify fields only by name or position.

Iceberg replaces directory listing as the source of truth with explicit metadata. A query starts from one committed table state, so files written by an unfinished operation are not accidentally treated as table data. Writers prepare a new state and publish it through a catalog’s commit mechanism. Metadata tracks stable field IDs, schema history, partition specifications, and snapshots, rather than relying on physical folder names to express all of these facts.

This does not make every failure disappear. Engines and catalogs must implement compatible commit behavior, and administrators still need access controls, backups, retention policies, and maintenance jobs. Iceberg provides a shared table model; it does not operate the entire data platform.

How Iceberg Works: Metadata, Snapshots, and Commits

An Iceberg table is a metadata tree. The catalog identifies the current table metadata file. That metadata records schemas, partition specifications, table properties, and snapshots. A snapshot points to a manifest list, which indexes manifest files describing data and delete files.

A reader loads the table metadata and chooses a snapshot. It follows that snapshot’s manifest list and manifests to find the files it needs. Because the snapshot is a stable reference to a particular table state, a later commit does not silently change the set of files already selected by that reader.

For a write, an engine creates data or delete files and prepares new manifests and metadata. It then asks the catalog to commit the new metadata as the current version. That atomic metadata update is the visibility point: readers see the old committed state or the new one, not a directory partially filled by the writer. If another writer commits first, the second writer may need to refresh its view, retry, or report a conflict, depending on the operation and catalog.

Manifests also carry statistics such as partition values and column bounds. Engines can use these to skip files that cannot match a query filter. The physical directory layout is therefore an implementation detail, not the table’s definition.

Key Components and Table Operations

Schemas and field IDs. Iceberg assigns stable IDs to columns. A column can be renamed or reordered without confusing it with another field in older files, because readers resolve it by ID rather than position alone. Schema evolution still needs compatible readers and sensible data contracts; stable IDs do not make every arbitrary type change safe.

Partition specs. A partition spec describes how values are transformed into partition values, such as deriving a day from a timestamp. Iceberg hides these physical partition details from query authors and allows partition evolution. A table can change its partitioning strategy without requiring every existing file to be rewritten immediately.

Snapshots and references. Each successful table change creates a snapshot. Snapshots enable time travel and rollback to a prior state while the required files remain available. Some implementations also support named branches and tags, which can make isolated experiments or retention policies easier to express. These capabilities depend on the Iceberg version, catalog, and engine in use.

Delete files and row-level changes. Depending on the table format version and engine, updates and deletes can be represented with position-delete or equality-delete files rather than rewriting every affected data file. Readers must combine those delete records with data files correctly. Verify that all engines touching a table support the features and format version it uses.

Catalogs. A catalog stores or resolves table identifiers and coordinates the commit that changes the current metadata pointer. Implementations include REST, Hive-compatible, cloud-provider, and SQL-backed catalogs. The catalog is not the table’s data itself, and different catalogs have different authentication, concurrency, and operational requirements.

Maintenance. Repeated writes can create many small files and manifests. Compaction rewrites data into more efficient files; manifest rewrites can improve planning. Snapshot expiration and orphan-file cleanup reclaim storage, but removing files too aggressively can break time travel or readers still using older states. Set retention deliberately and follow the engine’s documented procedures.

Real-World Uses

Iceberg is useful when an organization stores analytical data in object storage but needs more reliable table behavior than raw files provide. A batch warehouse can use snapshots to publish a completed daily load without exposing intermediate output. A streaming pipeline can commit incremental changes while analysts query a consistent earlier snapshot. Data science teams can preserve prior table states for reproducibility, provided retention policies keep the necessary files.

Its open metadata model can also help multiple engines share tables rather than maintaining separate copies for every compute system. That benefit is conditional: engines must agree on the format features and catalog protocol they support, and teams still need to test schema changes, writes, deletes, and maintenance across the actual toolchain.

Getting Started with a Local PyIceberg Table

PyIceberg provides Python access to Iceberg tables. The PyIceberg documentation covers its client configuration and supported catalogs. For a local learning exercise, use a virtual environment and install PyArrow plus the SQLite catalog support. This example writes a small table to a local warehouse; it is for development, not a production catalog or multi-writer deployment.

Create and activate the environment, then install the client:

python -m venv .venv
source .venv/bin/activate
python -m pip install "pyiceberg[pyarrow,sql-sqlite]"

On Windows PowerShell, activate the environment with .\.venv\Scripts\Activate.ps1 instead. Then save the following as create_table.py and run it with python create_table.py:

from pathlib import Path

import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType

catalog = load_catalog(
    "local",
    type="sql",
    uri="sqlite:///iceberg_catalog.db",
    warehouse=Path("warehouse").resolve().as_uri(),
)
catalog.create_namespace("analytics")

schema = Schema(
    NestedField(field_id=1, name="event_id", type=StringType(), required=True),
    NestedField(field_id=2, name="event_type", type=StringType(), required=True),
)
table = catalog.create_table("analytics.events", schema=schema)
table.append(
    pa.Table.from_pylist(
        [
            {"event_id": "evt-001", "event_type": "page_view"},
            {"event_id": "evt-002", "event_type": "purchase"},
        ]
    )
)

print(table.scan().to_arrow())
print(f"Snapshots: {len(table.snapshots())}")

The catalog database records table metadata locations, while the warehouse directory stores Iceberg metadata and data files. Verify that the script prints the two rows and at least one snapshot. If the namespace or table already exists, use a fresh catalog database or remove only this disposable local exercise’s catalog and warehouse before rerunning; do not apply that cleanup pattern to production storage.

For a deployed system, configure a supported catalog and object-store credentials, then run the same create, append, and scan workflow through an engine’s documented Iceberg integration. Confirm the engine and catalog support the table’s format version and any features you plan to use before enabling concurrent writers.

Common Misconceptions

“Iceberg stores the data instead of Parquet.” Iceberg stores table metadata and references files. Parquet remains a common format for the row data, and the storage system remains separate.

“A snapshot is a backup.” A snapshot is a logical table state. It can support time travel while its files are retained, but it does not by itself protect against loss of the bucket, credentials, or catalog. Maintain and test independent recovery procedures.

“Any engine can read any Iceberg table.” Interoperability depends on implementation support for the table’s format version, catalog, and features. Check compatibility across every reader and writer, especially before using row-level deletes or newer metadata features.

Related Articles

• See the broader storage and processing layers in building a data lake.
• Compare platform trade-offs in data lake vs. data warehouse architecture.
• Learn about a common processing engine in Apache Spark and big data processing.
• For event ingestion, read Apache Kafka event streaming.

Changelog and Last Updated

• Changelog: Initial publication.
• Last updated: At initial publication; reviewed against the Apache Iceberg table format specification and PyIceberg documentation.