Skip to main content

HDF5 Format

Description​

HDF5 (Hierarchical Data Format version 5) is a data model, library, and file format for storing and managing data. It's designed for storing large amounts of numerical data and is commonly used in scientific computing, particularly for arrays and tables.

File Extensions​

  • .hdf5 - HDF5 files
  • .h5 - HDF5 files (alias)

Implementation Details​

Reading​

The HDF5 implementation:

  • Uses h5py library for reading
  • Requires dataset_path parameter to specify which dataset to read
  • Supports 1D and 2D arrays
  • Handles structured arrays (tables)
  • Requires file path (not stream)

Writing​

Writing support:

  • Creates HDF5 files and structured datasets
  • Writes rows to dataset_path
  • Requires a file path (not a stream)
  • Non-structured datasets raise WriteNotSupportedError

Key Features​

  • Hierarchical structure: Supports groups and datasets
  • Large data: Designed for large numerical datasets
  • Scientific computing: Common in scientific applications
  • Totals support: Can count total rows in dataset

Usage​

from iterable import open_iterable

# List available datasets
from iterable.datatypes.hdf5 import HDF5Iterable

# Discover datasets before opening
datasets = HDF5Iterable('data.h5').list_tables('data.h5')
print(f"Available datasets: {datasets}")  # e.g., ['/data', '/group/dataset1']

# Reading with dataset path
with open_iterable('data.h5', iterableargs={
    'dataset_path': '/data'  # Path to dataset in HDF5 file
}) as source:
    for row in source:
        print(row)

# List datasets after opening (reuses file handle)
with open_iterable('data.h5', iterableargs={'dataset_path': '/data'}) as source:
    all_datasets = source.list_tables()  # Reuses open file handle
    print(f"All datasets: {all_datasets}")

# Writing structured datasets
with open_iterable('output.h5', mode='w', iterableargs={
    'dataset_path': '/data'
}) as dest:
    dest.write({'col1': 1, 'col2': 2})
# Non-structured datasets raise WriteNotSupportedError

Parameters​

  • dataset_path (str): Required - Path to dataset in HDF5 file (e.g., /data, /group/dataset)

Limitations​

  1. h5py dependency: Requires h5py package
  2. Dataset path required: Must specify which dataset to read/write
  3. File path required: Requires filename, not stream
  4. Flat data only: Only supports tabular/array data
  5. Structure complexity: Complex HDF5 structures may require manual handling

Compression Support​

HDF5 has built-in compression support (separate from file-level compression):

  • HDF5 supports various compression filters (gzip, szip, etc.)

HDF5 files can also be compressed with file-level codecs:

  • GZip (.h5.gz)
  • BZip2 (.h5.bz2)
  • LZMA (.h5.xz)
  • LZ4 (.h5.lz4)
  • ZIP (.h5.zip)
  • Brotli (.h5.br)
  • ZStandard (.h5.zst)

Use Cases​

  • Scientific computing: Storing large numerical datasets
  • Machine learning: Storing training data
  • Data analysis: Large-scale data analysis
  • Simulations: Storing simulation results

Error Handling​

  • Missing dependency: optional libraries raise ImportError with an install hint (pip install 'iterabledata[<extra>]' when an extra exists).
  • Read-only: opening with mode="w" raises WriteNotSupportedError or ValueError.
  • Bad or unsupported input: may raise ValueError, OSError, or library-specific errors.
  • See Troubleshooting for decoding, detection, and engine issues.
  • Parquet - Columnar format for analytics
  • Arrow - Columnar in-memory format