Capability Matrix

IterableData supports many formats, but capabilities vary (some are read-only; some support totals(), bulk ops, or streaming).

This page is an overview to help pick formats and avoid surprises. For format-specific details, see the corresponding page under Data File Formats.

Legend

Read: supports read() / iteration
Write: supports write() / write_bulk()
Bulk: supports read_bulk() and/or write_bulk() efficiently
Totals: supports has_totals() + totals()
Streaming: can process large inputs without loading whole file into memory

Common formats (high usage)

Format	Read	Write	Bulk	Totals	Streaming	Notes
CSV/TSV	✅	✅	✅	✅	✅	Delimiter/encoding detection supported
JSONL/NDJSON	✅	✅	✅	✅	✅	Best for streaming JSON records
JSON	✅	✅	✅	✅	❌	Loads whole document in memory (array JSON)
Parquet	✅	✅	✅	✅	⚠️	Read is streaming in batches; write is batched
XML	✅	❌	✅	❌	✅	Requires `tagname`; iterative parse
DuckDB (engine)	✅	❌	✅	✅	⚠️	Fast scans via DuckDB; limited codec support
DuckDB (datatype)	✅	✅	✅	✅	❌	Database file `.duckdb`/`.ddb`
WARC	✅	✅	✅	❌	✅	Uses `warcio`

How to interpret "Streaming"

Some formats (e.g. Parquet) read in record batches but still require internal buffering; others (JSON) load everything. In general:

Prefer JSONL/CSV for truly streamy pipelines.
Prefer Parquet for analytics and columnar storage.

Memory Considerations

⚠️ Non-Streaming Formats

Some formats load entire files into memory before processing. These formats are marked with ❌ in the Streaming column. When working with large files (>100MB), consider:

Use streaming alternatives when available:
- Instead of JSON → use JSONL (line-by-line JSON)
- Instead of HTML → extract data first, then process
- Instead of TOML/ARFF → convert to CSV/JSONL first

Check file size before processing:

import os
file_size = os.path.getsize('data.html')
if file_size > 100 * 1024 * 1024:  # 100MB
    print("Warning: Large file - consider using streaming format")

Query capabilities programmatically:

from iterable.helpers.capabilities import get_capability

is_streaming = get_capability("html", "streaming")
if not is_streaming:
    print("Warning: HTML format loads entire file into memory")

Formats That Load Entire Files

The following formats load entire files into memory and may cause memory issues with large files:

Feed formats (RSS/Atom): feed, rss, atom
ARFF: arff (Attribute-Relation File Format)
HTML: html, htm (requires full document parsing)
TOML: toml (configuration format)
HOCON: hocon (configuration format)
EDN: edn (Extensible Data Notation)
Bencode: bencode (BitTorrent format)
ASN.1: asn1 (Abstract Syntax Notation)
iCal: ical, ics (calendar format)
Turtle RDF: turtle, ttl (RDF format)
VCF: vcf (Variant Call Format)
MHTML: mhtml (archived web pages)
FlexBuffers: flexbuffers (serialization format)
PC-Axis: px (statistical data format)
MVT: mvt (Mapbox Vector Tile - single tile per file)

Note: JSON format has conditional streaming - it uses streaming for files >10MB, but loads smaller files entirely.

Best Practices

For large files (>100MB): Use streaming formats (CSV, JSONL, Parquet, XML with tagname)
For small files (under 10MB): Non-streaming formats are acceptable
For very large files (>1GB): Always use streaming formats or process in chunks
Monitor memory usage: Use system monitoring tools when processing large files

Read-Only Formats

Some formats in IterableData are read-only and do not support write operations. Attempting to write to these formats will raise a WriteNotSupportedError.

Why Some Formats Are Read-Only

Read-only formats typically fall into one of these categories:

Legacy/Proprietary Formats: Formats like SAS, Stata, SPSS, and PC-Axis (PX) are primarily used for data analysis and are rarely written programmatically. These formats often have complex internal structures that make writing difficult.
Specialized Formats: Formats like PCAP (network packet capture), HTML, and feed formats (RSS/Atom) are typically generated by specialized tools rather than written programmatically.
Not Yet Implemented: Some formats like Avro, DBF, XML, and XLS have read support but write support hasn't been implemented yet. These may be added in future releases.
Columnar Storage Formats: Some modern columnar formats like Delta Lake, Iceberg, and Hudi are primarily managed through their respective engines rather than direct file writing.

Checking Write Support Programmatically

You can check if a format supports writing using the capability system:

from iterable.helpers.capabilities import supports_write, get_capability

# Using the convenience function
if supports_write("csv"):
    print("CSV supports writing")
else:
    print("CSV is read-only")

# Using the capability system
is_writable = get_capability("pcap", "writable")
if is_writable:
    print("PCAP supports writing")
else:
    print("PCAP is read-only")

Handling WriteNotSupportedError

When attempting to write to a read-only format, you'll receive a WriteNotSupportedError:

from iterable import open_iterable
from iterable.exceptions import WriteNotSupportedError

try:
    with open_iterable("data.pcap", mode="w") as dest:
        dest.write({"timestamp": 1234567890, "data": b"..."})
except WriteNotSupportedError as e:
    print(f"Format '{e.format_id}' does not support writing: {e.reason}")
    # Consider converting to a writable format first

Read-Only Format List

The following formats are currently read-only (27 formats total):

Statistical Data Formats:

ARFF (Attribute-Relation File Format)
SAS (.sas, .sas7bdat)
SPSS (.sav, .spss)
Stata (.dta, .stata)
RData (.rdata, .rda)
RDS (.rds)
PC-Axis (.px)

Network and Web Formats:

PCAP (.pcap, .pcapng) - Network packet capture
HTML (.html, .htm)
Feed formats (.atom, .rss) - RSS/Atom feeds

Columnar Storage Formats:

Delta Lake
Apache Iceberg
Apache Hudi

Other Formats:

Avro (.avro) - Write support not yet implemented
DBF (.dbf) - Write support not yet implemented
DXF (.dxf) - CAD format
MVT (.mvt, .pbf) - Mapbox Vector Tiles
NetCDF (.nc, .netcdf) - Scientific data format
ODS (.ods) - OpenDocument Spreadsheet
PSV (.psv) - Pipe-separated values
XLS (.xls) - Legacy Excel format
XLSX (.xlsx) - Excel format (note: XLSX actually has write support, but XLS does not)
XML (.xml) - Write support not yet implemented
Zipped (.zipped) - Generic zip container
ZipXML (.zipxml) - XML in zip container
FlatBuffers (.fbs) - Serialization format

Note: The XLSX format actually supports writing, but XLS (legacy Excel) does not. Some formats in this list may gain write support in future releases.

Finding All Read-Only Formats

You can programmatically find all read-only formats:

from iterable.helpers.capabilities import list_all_capabilities

all_caps = list_all_capabilities()
read_only_formats = [
    fmt_id for fmt_id, caps in all_caps.items()
    if caps.get("writable") is False
]

print(f"Read-only formats ({len(read_only_formats)}):")
for fmt_id in sorted(read_only_formats):
    print(f"  - {fmt_id}")

Workarounds for Read-Only Formats

If you need to write data in a format that's currently read-only, consider:

Convert to a writable format: Read from the read-only format and write to a writable one:

from iterable import open_iterable

# Read from read-only format
with open_iterable("data.sas") as source:
    # Write to writable format
    with open_iterable("output.parquet", mode="w") as dest:
        for row in source:
            dest.write(row)

Use format-specific libraries: For formats like SAS, Stata, or SPSS, consider using their native Python libraries (e.g., pyreadstat for SAS/Stata/SPSS) if you need write support.
Check for updates: Write support may be added in future releases. Check the changelog or issue tracker for planned features.

Programmatic Capability Queries

You can programmatically query format capabilities using the iterable.helpers.capabilities module. This allows you to:

Discover which formats support specific features before attempting operations
Build format-agnostic code that adapts to available capabilities
Make informed decisions about format selection
Generate documentation or tooling based on capability data

Basic Usage

from iterable.helpers.capabilities import (
    get_format_capabilities,
    get_capability,
    list_all_capabilities
)

# Get all capabilities for a format
caps = get_format_capabilities("csv")
print(caps)
# {
#   'readable': True,
#   'writable': True,
#   'bulk_read': True,
#   'bulk_write': True,
#   'totals': True,
#   'streaming': True,
#   'flat_only': True,
#   'tables': False,
#   'compression': True,
#   'nested': False
# }

# Query a specific capability
is_writable = get_capability("json", "writable")  # True
has_totals = get_capability("parquet", "totals")  # True
supports_tables = get_capability("xlsx", "tables")  # True

# List capabilities for all formats
all_caps = list_all_capabilities()
for format_id, capabilities in all_caps.items():
    if capabilities.get("tables"):
        print(f"{format_id} supports multiple tables")

API Reference

`get_format_capabilities(format_id: str) -> dict[str, bool | None]`

Get all capabilities for a specific format.

Parameters:

format_id (str): Format identifier (file extension or format name, e.g., "csv", "json", "parquet")

Returns:

Dictionary with capability flags (True/False/None)

Raises:

ValueError: If format_id is not recognized
ImportError: If format class cannot be loaded (optional dependency missing)

Example:

caps = get_format_capabilities("csv")
if caps["writable"]:
    print("CSV format supports writing")
if caps.get("totals"):
    print("CSV format supports row count totals")

`get_capability(format_id: str, capability: str) -> bool | None`

Get a specific capability for a format.

Parameters:

format_id (str): Format identifier
capability (str): Capability name (see Available Capabilities below)

Returns:

Capability value (True/False/None), or None if capability name is invalid

Raises:

ValueError: If format_id is not recognized

Example:

# Check if format supports writing
is_writable = get_capability("json", "writable")
if is_writable:
    print("JSON format can be written to")

# Check if format supports multiple tables
has_tables = get_capability("xlsx", "tables")
if has_tables:
    print("Excel format supports multiple sheets")

`supports_write(format_id: str) -> bool`

Check if a format supports write operations. This is a convenience function that returns True only if the format explicitly implements write methods (not just the base class default that raises WriteNotSupportedError).

Parameters:

format_id (str): Format identifier (file extension or format name)

Returns:

True if format supports writing, False otherwise

Raises:

ValueError: If format_id is not recognized
ImportError: If format class cannot be loaded (optional dependency missing)

Example:

from iterable.helpers.capabilities import supports_write

if supports_write("csv"):
    print("CSV format supports writing")
else:
    print("CSV format is read-only")

if supports_write("pcap"):
    print("PCAP format supports writing")
else:
    print("PCAP format is read-only")

`list_all_capabilities() -> dict[str, dict[str, bool | None]]`

List capabilities for all registered formats.

Returns:

Dictionary mapping format IDs to their capability dictionaries

Example:

all_caps = list_all_capabilities()

# Find all formats that support writing
writable_formats = [
    fmt_id for fmt_id, caps in all_caps.items()
    if caps.get("writable")
]
print(f"Writable formats: {writable_formats}")

# Find all formats that support multiple tables
multi_table_formats = [
    fmt_id for fmt_id, caps in all_caps.items()
    if caps.get("tables")
]
print(f"Multi-table formats: {multi_table_formats}")

Practical Examples

Example 1: Format Selection Based on Capabilities

from iterable.helpers.capabilities import get_format_capabilities

def choose_output_format(needs_totals=True, needs_nested=True):
    """Choose an output format based on required capabilities."""
    candidates = ["csv", "json", "jsonl", "parquet"]
    
    for fmt_id in candidates:
        caps = get_format_capabilities(fmt_id)
        if needs_totals and not caps.get("totals"):
            continue
        if needs_nested and caps.get("flat_only"):
            continue
        if not caps.get("writable"):
            continue
        return fmt_id
    
    return None

# Find format that supports totals and nested data
format_id = choose_output_format(needs_totals=True, needs_nested=True)
print(f"Recommended format: {format_id}")

Example 2: Adaptive Code Based on Capabilities

from iterable import open_iterable
from iterable.helpers.capabilities import get_capability

def process_file(filename, format_id):
    """Process a file, adapting behavior based on format capabilities."""
    with open_iterable(filename) as source:
        # Use totals if available
        if get_capability(format_id, "totals"):
            total = source.totals()
            print(f"Total rows: {total}")
        
        # Use bulk operations if available
        if get_capability(format_id, "bulk_read"):
            batch_size = 10000
            while True:
                batch = source.read_bulk(batch_size)
                if not batch:
                    break
                process_batch(batch)
        else:
            # Fall back to row-by-row processing
            for row in source:
                process_row(row)

Example 3: Discover Formats with Specific Features

from iterable.helpers.capabilities import list_all_capabilities

def find_formats_with_feature(capability_name):
    """Find all formats that support a specific capability."""
    all_caps = list_all_capabilities()
    matching_formats = []
    
    for format_id, capabilities in all_caps.items():
        if capabilities.get(capability_name):
            matching_formats.append(format_id)
    
    return sorted(matching_formats)

# Find all formats that support totals
formats_with_totals = find_formats_with_feature("totals")
print(f"Formats with totals support: {formats_with_totals}")

# Find all formats that support multiple tables
formats_with_tables = find_formats_with_feature("tables")
print(f"Formats with table support: {formats_with_tables}")

# Find all writable formats
writable_formats = find_formats_with_feature("writable")
print(f"Writable formats: {len(writable_formats)} formats")

Example 4: Error Handling for Missing Dependencies

from iterable.helpers.capabilities import get_format_capabilities

def safe_get_capabilities(format_id):
    """Safely get capabilities, handling missing optional dependencies."""
    try:
        caps = get_format_capabilities(format_id)
        # Check if capabilities are unknown (None values indicate missing deps)
        if all(v is None for v in caps.values()):
            print(f"Warning: Format '{format_id}' requires optional dependencies")
            print("Capabilities cannot be determined without installing dependencies")
        return caps
    except ImportError as e:
        print(f"Format '{format_id}' requires optional dependencies: {e}")
        return None
    except ValueError as e:
        print(f"Unknown format: {e}")
        return None

# Try to get capabilities for a format that might need optional deps
caps = safe_get_capabilities("parquet")
if caps:
    print(f"Parquet capabilities: {caps}")

Available Capabilities

The following capabilities are tracked for each format:

readable: Format supports reading data (read() method)
writable: Format supports writing data (write() method)
bulk_read: Format supports bulk reading (read_bulk() method)
bulk_write: Format supports bulk writing (write_bulk() method)
totals: Format supports row count totals (has_totals() returns True)
streaming: Format supports streaming (doesn't load entire file into memory)
flat_only: Format only supports flat (non-nested) data structures
tables: Format supports multiple tables/sheets/datasets (has_tables() returns True)
compression: Format supports compression codecs (GZIP, BZIP2, etc.)
nested: Format can preserve nested data structures (opposite of flat_only)

Value Meanings:

True: Capability is supported
False: Capability is not supported
None: Capability cannot be determined (e.g., when optional dependencies are missing)

Performance Notes

Capability detection results are cached per format to avoid repeated introspection
The first call to get_format_capabilities() for a format performs introspection
Subsequent calls return cached results
Cache is valid for the lifetime of the Python process (format classes don't change at runtime)

Extended catalog fields

iterable.catalog.export_catalog() emits _schema_version (1.1) and each format descriptor includes the following reviewable declarations:

native_bulk_read / native_bulk_write: whether the backend has a native batch path; this is distinct from the legacy API-method flags.
read_memory / write_memory: bounded, whole_input, whole_output, backend_defined, or unknown.
selection: supported projection, filter, row-group, or slice selectors.
codec_support: codec names declared for the format.
source_constraints: requirements such as filename or directory.
maturity: stable, experimental, or partial.

Unknown values remain explicit rather than being inferred optimistically from method existence. Use dev/scripts/audit_format_descriptors.py to review the complete canonical descriptor table before changing generated catalog data.

Legend​

Common formats (high usage)​

How to interpret "Streaming"​

Memory Considerations​

⚠️ Non-Streaming Formats​

Formats That Load Entire Files​

Best Practices​

Read-Only Formats​

Why Some Formats Are Read-Only​

Checking Write Support Programmatically​

Handling WriteNotSupportedError​

Read-Only Format List​

Finding All Read-Only Formats​

Workarounds for Read-Only Formats​

Programmatic Capability Queries​

Basic Usage​

API Reference​

get_format_capabilities(format_id: str) -> dict[str, bool | None]​

get_capability(format_id: str, capability: str) -> bool | None​

supports_write(format_id: str) -> bool​

list_all_capabilities() -> dict[str, dict[str, bool | None]]​

Practical Examples​

Example 1: Format Selection Based on Capabilities​

Example 2: Adaptive Code Based on Capabilities​

Example 3: Discover Formats with Specific Features​

Example 4: Error Handling for Missing Dependencies​

Available Capabilities​

Performance Notes​

Extended catalog fields​

Legend

Common formats (high usage)

How to interpret "Streaming"

Memory Considerations

⚠️ Non-Streaming Formats

Formats That Load Entire Files

Best Practices

Read-Only Formats

Why Some Formats Are Read-Only

Checking Write Support Programmatically

Handling WriteNotSupportedError

Read-Only Format List

Finding All Read-Only Formats

Workarounds for Read-Only Formats

Programmatic Capability Queries

Basic Usage

API Reference

`get_format_capabilities(format_id: str) -> dict[str, bool | None]`

`get_capability(format_id: str, capability: str) -> bool | None`

`supports_write(format_id: str) -> bool`

`list_all_capabilities() -> dict[str, dict[str, bool | None]]`

Practical Examples

Example 1: Format Selection Based on Capabilities

Example 2: Adaptive Code Based on Capabilities

Example 3: Discover Formats with Specific Features

Example 4: Error Handling for Missing Dependencies

Available Capabilities

Performance Notes

Extended catalog fields