Skip to main content

Paimon Row Format

Description​

Apache Paimon Row (.row) is a row-oriented file format optimized for O(1) random access by row number. Data is stored in ZSTD-compressed blocks with a compact null-bitmap row encoding, a delta/ZigZag/varint block index, and a fixed 32-byte footer ending in the ROWS magic.

Maturity: experimental.

File Extensions​

  • .row — Paimon Row data files

Important: The format does not embed schema. Reads require an explicit schema.

Dependencies​

pip install iterabledata[paimon-row]
# or
pip install iterabledata[paimon]

Requires zstandard. Nested/VARIANT types are not supported in the first experimental cut.

Usage​

from iterable import open_iterable

schema = [("id", "bigint"), ("name", "string"), ("score", "double")]

with open_iterable("people.row", mode="w", iterableargs={"schema": schema}) as dest:
    dest.write_bulk([
        {"id": 1, "name": "Alice", "score": 1.5},
        {"id": 2, "name": "Bob", "score": 2.0},
    ])

with open_iterable("people.row", iterableargs={"schema": schema}) as source:
    for row in source:
        print(row)

Parameters​

ParameterTypeDefaultDescription
schemalist/dict/pyarrow.Schemarequired on readPhysical field order and types
block_sizeint65536Uncompressed block flush threshold (bytes)

Supported primitive types: boolean, tinyint, smallint, int/date/time, bigint, float, double, string, binary.

Detection​

  • Extension: .row
  • Footer magic: trailing ROWS (0x524F5753 as ASCII bytes) on seekable sources

Limitations​

  • Schema must be provided for reads
  • Nested ARRAY/MAP/ROW and VARIANT are not supported yet
  • Stream mode without a filename is not the preferred path; use filenames for footer-based I/O

Error Handling​

  • Missing dependency: optional libraries raise ImportError with an install hint (pip install 'iterabledata[<extra>]' when an extra exists).
  • Write mode: read-only formats raise WriteNotSupportedError or ValueError when opened with mode="w".
  • Bad or unsupported input: may raise ValueError, OSError, or library-specific errors.
  • See Troubleshooting for decoding, detection, and engine issues.

See Also​