Capability Matrix

IterableData supports many formats, but capabilities vary (some are read-only; some support totals(), bulk ops, or streaming).

This page is an overview to help pick formats and avoid surprises. For format-specific details, see the corresponding page under Data File Formats.

Legend

• Read: supports read() / iteration
• Write: supports write() / write_bulk()
• Bulk: supports read_bulk() and/or write_bulk() efficiently
• Totals: supports has_totals() + totals()
• Streaming: can process large inputs without loading whole file into memory

Common formats (high usage)

Format	Read	Write	Bulk	Totals	Streaming	Notes
CSV/TSV	✅	✅	✅	✅	✅	Delimiter/encoding detection supported
JSONL/NDJSON	✅	✅	✅	✅	✅	Best for streaming JSON records
JSON	✅	✅	✅	✅	❌	Loads whole document in memory (array JSON)
Parquet	✅	✅	✅	✅	⚠️	Read is streaming in batches; write is batched
XML	✅	❌	✅	❌	✅	Requires tagname; iterative parse
DuckDB (engine)	✅	❌	✅	✅	⚠️	Fast scans via DuckDB; limited codec support
DuckDB (datatype)	✅	✅	✅	✅	❌	Database file .duckdb/.ddb
WARC	✅	✅	✅	❌	✅	Uses warcio

How to interpret "Streaming"

Some formats (e.g. Parquet) read in record batches but still require internal buffering; others (JSON) load everything. In general:

• Prefer JSONL/CSV for truly streamy pipelines.
• Prefer Parquet for analytics and columnar storage.

Memory Considerations

⚠️ Non-Streaming Formats

Some formats load entire files into memory before processing. These formats are marked with ❌ in the Streaming column. When working with large files (>100MB), consider:

1. Use streaming alternatives when available:

   • Instead of JSON → use JSONL (line-by-line JSON)
   • Instead of HTML → extract data first, then process
   • Instead of TOML/ARFF → convert to CSV/JSONL first

2. Check file size before processing:

   import os
   file_size = os.path.getsize('data.html')
   if file_size > 100 * 1024 * 1024:  # 100MB
       print("Warning: Large file - consider using streaming format")

3. Query capabilities programmatically:

   from iterable.helpers.capabilities import get_capability
   
   is_streaming = get_capability("html", "streaming")
   if not is_streaming:
       print("Warning: HTML format loads entire file into memory")

Formats That Load Entire Files

The following formats load entire files into memory and may cause memory issues with large files:

• Feed formats (RSS/Atom): feed, rss, atom
• ARFF: arff (Attribute-Relation File Format)
• HTML: html, htm (requires full document parsing)
• TOML: toml (configuration format)
• HOCON: hocon (configuration format)
• EDN: edn (Extensible Data Notation)
• Bencode: bencode (BitTorrent format)
• ASN.1: asn1 (Abstract Syntax Notation)
• iCal: ical, ics (calendar format)
• Turtle RDF: turtle, ttl (RDF format)
• VCF: vcf, vcard (vCard contact files; not genomic VCF)
• MHTML: mhtml (archived web pages)
• FlexBuffers: flexbuffers (serialization format)
• PC-Axis: px (statistical data format)
• MVT: mvt (Mapbox Vector Tile - single tile per file)

Note: JSON format has conditional streaming - it uses streaming for files >10MB, but loads smaller files entirely.

Best Practices

• For large files (>100MB): Use streaming formats (CSV, JSONL, Parquet, XML with tagname)
• For small files (under 10MB): Non-streaming formats are acceptable
• For very large files (>1GB): Always use streaming formats or process in chunks
• Monitor memory usage: Use system monitoring tools when processing large files

Read-Only Formats

Some formats in IterableData are read-only and do not support write operations. Attempting to write to these formats will raise a WriteNotSupportedError.

Why Some Formats Are Read-Only

Read-only formats typically fall into one of these categories:

1. Legacy/Proprietary Formats: Formats like SAS, Stata, SPSS, and PC-Axis (PX) are primarily used for data analysis and are rarely written programmatically. These formats often have complex internal structures that make writing difficult.

2. Specialized Formats: Formats like PCAP (network packet capture), HTML, and feed formats (RSS/Atom) are typically generated by specialized tools rather than written programmatically.

3. Not Yet Implemented: Some formats like DBF, XML, and XLS have read support but write support hasn't been implemented yet.

4. Lakehouse tables: Hudi is read-only here. Delta Lake and Iceberg support writes through their Python libraries.

Checking Write Support Programmatically

You can check if a format supports writing using the capability system:

from iterable.helpers.capabilities import supports_write, get_capability

# Using the convenience function
if supports_write("csv"):
    print("CSV supports writing")
else:
    print("CSV is read-only")

# Using the capability system
is_writable = get_capability("pcap", "writable")
if is_writable:
    print("PCAP supports writing")
else:
    print("PCAP is read-only")

Handling WriteNotSupportedError

When attempting to write to a read-only format, you'll receive a WriteNotSupportedError:

from iterable import open_iterable
from iterable.exceptions import WriteNotSupportedError

try:
    with open_iterable("data.pcap", mode="w") as dest:
        dest.write({"timestamp": 1234567890, "data": b"..."})
except WriteNotSupportedError as e:
    print(f"Format '{e.format_id}' does not support writing: {e.reason}")
    # Consider converting to a writable format first

Read-Only Format List

Query the live set with list_all_capabilities() (the count changes as writers are added). Common read-only families:

Statistical data: ARFF, SAS, SPSS, Stata, RData, RDS, PC-Axis (PX)

Spreadsheets: XLS, XLSX, XLSB, ODS (convert to CSV or Parquet to write)

Network, web, and markup: PCAP, HTML, RSS/Atom feeds, XML, ZipXML, TAR

Lakehouse: Apache Hudi (Delta Lake and Iceberg do write)

Other: DBF, DXF, MVT, NetCDF, FlatBuffers, genomic VCF/BCF, SAM/BAM/CRAM, and several scientific/geospatial readers

PSV writes (it is a CSV pipe-delimiter subclass). Avro, WARC, MBOX, and vCard (.vcf) also write. HDF5 writes structured datasets; other dataset layouts raise WriteNotSupportedError.

Do not treat vCard (.vcf) as genomic Variant Call Format — that is genomic_vcf.

Finding All Read-Only Formats

You can programmatically find all read-only formats:

from iterable.helpers.capabilities import list_all_capabilities

all_caps = list_all_capabilities()
read_only_formats = [
    fmt_id for fmt_id, caps in all_caps.items()
    if caps.get("writable") is False
]

print(f"Read-only formats ({len(read_only_formats)}):")
for fmt_id in sorted(read_only_formats):
    print(f"  - {fmt_id}")

Workarounds for Read-Only Formats

If you need to write data in a format that's currently read-only, consider:

1. Convert to a writable format: Read from the read-only format and write to a writable one:

   from iterable import open_iterable
   
   # Read from read-only format
   with open_iterable("data.sas") as source:
       # Write to writable format
       with open_iterable("output.parquet", mode="w") as dest:
           for row in source:
               dest.write(row)

2. Use format-specific libraries: For formats like SAS, Stata, or SPSS, consider using their native Python libraries (e.g., pyreadstat for SAS/Stata/SPSS) if you need write support.

3. Check for updates: Write support may be added in future releases. Check the changelog or issue tracker for planned features.

Programmatic Capability Queries

You can programmatically query format capabilities using the iterable.helpers.capabilities module. This allows you to:

• Discover which formats support specific features before attempting operations
• Build format-agnostic code that adapts to available capabilities
• Make informed decisions about format selection
• Generate documentation or tooling based on capability data

Basic Usage

from iterable.helpers.capabilities import (
    get_format_capabilities,
    get_capability,
    list_all_capabilities
)

# Get all capabilities for a format
caps = get_format_capabilities("csv")
print(caps)
# {
#   'readable': True,
#   'writable': True,
#   'bulk_read': True,
#   'bulk_write': True,
#   'totals': True,
#   'streaming': True,
#   'flat_only': True,
#   'tables': False,
#   'compression': True,
#   'nested': False
# }

# Query a specific capability
is_writable = get_capability("json", "writable")  # True
has_totals = get_capability("parquet", "totals")  # True
supports_tables = get_capability("xlsx", "tables")  # True

# List capabilities for all formats
all_caps = list_all_capabilities()
for format_id, capabilities in all_caps.items():
    if capabilities.get("tables"):
        print(f"{format_id} supports multiple tables")

API Reference

get_format_capabilities(format_id: str) -> dict[str, bool | None]

Get all capabilities for a specific format.

Parameters:

• format_id (str): Format identifier (file extension or format name, e.g., "csv", "json", "parquet")

Returns:

• Dictionary with capability flags (True/False/None)

Raises:

• ValueError: If format_id is not recognized
• ImportError: If format class cannot be loaded (optional dependency missing)

Example:

caps = get_format_capabilities("csv")
if caps["writable"]:
    print("CSV format supports writing")
if caps.get("totals"):
    print("CSV format supports row count totals")

get_capability(format_id: str, capability: str) -> bool | None

Get a specific capability for a format.

Parameters:

• format_id (str): Format identifier
• capability (str): Capability name (see Available Capabilities below)

Returns:

• Capability value (True/False/None), or None if capability name is invalid

Raises:

• ValueError: If format_id is not recognized

Example:

# Check if format supports writing
is_writable = get_capability("json", "writable")
if is_writable:
    print("JSON format can be written to")

# Check if format supports multiple tables
has_tables = get_capability("xlsx", "tables")
if has_tables:
    print("Excel format supports multiple sheets")

supports_write(format_id: str) -> bool

Check if a format supports write operations. This is a convenience function that returns True only if the format explicitly implements write methods (not just the base class default that raises WriteNotSupportedError).

Parameters:

• format_id (str): Format identifier (file extension or format name)

Returns:

• True if format supports writing, False otherwise

Raises:

• ValueError: If format_id is not recognized
• ImportError: If format class cannot be loaded (optional dependency missing)

Example:

from iterable.helpers.capabilities import supports_write

if supports_write("csv"):
    print("CSV format supports writing")
else:
    print("CSV format is read-only")

if supports_write("pcap"):
    print("PCAP format supports writing")
else:
    print("PCAP format is read-only")

list_all_capabilities() -> dict[str, dict[str, bool | None]]

List capabilities for all registered formats.

Returns:

• Dictionary mapping format IDs to their capability dictionaries

Example:

all_caps = list_all_capabilities()

# Find all formats that support writing
writable_formats = [
    fmt_id for fmt_id, caps in all_caps.items()
    if caps.get("writable")
]
print(f"Writable formats: {writable_formats}")

# Find all formats that support multiple tables
multi_table_formats = [
    fmt_id for fmt_id, caps in all_caps.items()
    if caps.get("tables")
]
print(f"Multi-table formats: {multi_table_formats}")

Practical Examples

Example 1: Format Selection Based on Capabilities

from iterable.helpers.capabilities import get_format_capabilities

def choose_output_format(needs_totals=True, needs_nested=True):
    """Choose an output format based on required capabilities."""
    candidates = ["csv", "json", "jsonl", "parquet"]
    
    for fmt_id in candidates:
        caps = get_format_capabilities(fmt_id)
        if needs_totals and not caps.get("totals"):
            continue
        if needs_nested and caps.get("flat_only"):
            continue
        if not caps.get("writable"):
            continue
        return fmt_id
    
    return None

# Find format that supports totals and nested data
format_id = choose_output_format(needs_totals=True, needs_nested=True)
print(f"Recommended format: {format_id}")

Example 2: Adaptive Code Based on Capabilities

from iterable import open_iterable
from iterable.helpers.capabilities import get_capability

def process_file(filename, format_id):
    """Process a file, adapting behavior based on format capabilities."""
    with open_iterable(filename) as source:
        # Use totals if available
        if get_capability(format_id, "totals"):
            total = source.totals()
            print(f"Total rows: {total}")
        
        # Use bulk operations if available
        if get_capability(format_id, "bulk_read"):
            batch_size = 10000
            while True:
                batch = source.read_bulk(batch_size)
                if not batch:
                    break
                process_batch(batch)
        else:
            # Fall back to row-by-row processing
            for row in source:
                process_row(row)

Example 3: Discover Formats with Specific Features

from iterable.helpers.capabilities import list_all_capabilities

def find_formats_with_feature(capability_name):
    """Find all formats that support a specific capability."""
    all_caps = list_all_capabilities()
    matching_formats = []
    
    for format_id, capabilities in all_caps.items():
        if capabilities.get(capability_name):
            matching_formats.append(format_id)
    
    return sorted(matching_formats)

# Find all formats that support totals
formats_with_totals = find_formats_with_feature("totals")
print(f"Formats with totals support: {formats_with_totals}")

# Find all formats that support multiple tables
formats_with_tables = find_formats_with_feature("tables")
print(f"Formats with table support: {formats_with_tables}")

# Find all writable formats
writable_formats = find_formats_with_feature("writable")
print(f"Writable formats: {len(writable_formats)} formats")

Example 4: Error Handling for Missing Dependencies

from iterable.helpers.capabilities import get_format_capabilities

def safe_get_capabilities(format_id):
    """Safely get capabilities, handling missing optional dependencies."""
    try:
        caps = get_format_capabilities(format_id)
        # Check if capabilities are unknown (None values indicate missing deps)
        if all(v is None for v in caps.values()):
            print(f"Warning: Format '{format_id}' requires optional dependencies")
            print("Capabilities cannot be determined without installing dependencies")
        return caps
    except ImportError as e:
        print(f"Format '{format_id}' requires optional dependencies: {e}")
        return None
    except ValueError as e:
        print(f"Unknown format: {e}")
        return None

# Try to get capabilities for a format that might need optional deps
caps = safe_get_capabilities("parquet")
if caps:
    print(f"Parquet capabilities: {caps}")

Available Capabilities

The following capabilities are tracked for each format:

• readable: Format supports reading data (read() method)
• writable: Format supports writing data (write() method)
• bulk_read: Format supports bulk reading (read_bulk() method)
• bulk_write: Format supports bulk writing (write_bulk() method)
• totals: Format supports row count totals (has_totals() returns True)
• streaming: Format supports streaming (doesn't load entire file into memory)
• flat_only: Format only supports flat (non-nested) data structures
• tables: Format supports multiple tables/sheets/datasets (has_tables() returns True)
• compression: Format supports compression codecs (GZIP, BZIP2, etc.)
• nested: Format can preserve nested data structures (opposite of flat_only)

Value Meanings:

• True: Capability is supported
• False: Capability is not supported
• None: Capability cannot be determined (e.g., when optional dependencies are missing)

Performance Notes

• Capability detection results are cached per format to avoid repeated introspection
• The first call to get_format_capabilities() for a format performs introspection
• Subsequent calls return cached results
• Cache is valid for the lifetime of the Python process (format classes don't change at runtime)

Extended catalog fields

iterable.catalog.export_catalog() emits _schema_version (1.1) and each format descriptor includes the following reviewable declarations:

• native_bulk_read / native_bulk_write: whether the backend has a native batch path; this is distinct from the legacy API-method flags.
• read_memory / write_memory: bounded, whole_input, whole_output, backend_defined, or unknown.
• selection: supported projection, filter, row-group, or slice selectors.
• codec_support: codec names declared for the format.
• source_constraints: requirements such as filename or directory.
• maturity: stable, experimental, or partial.

Unknown values remain explicit rather than being inferred optimistically from method existence. Use dev/scripts/audit_format_descriptors.py to review the complete canonical descriptor table before changing generated catalog data.