Migration Guide

This guide helps you migrate your code when upgrading between versions of Iterable Data.

Version 1.0.7 and Later

Exception Hierarchy Improvements

What Changed: IterableData now uses a comprehensive exception hierarchy instead of generic ValueError, NotImplementedError, and Exception. This allows for more specific error handling and better error messages.

Before:

from iterable import open_iterable

try:
    with open_iterable('data.parquet', mode='w') as dest:
        dest.write({'key': 'value'})
except NotImplementedError:
    print("Writing not supported")
except ValueError as e:
    print(f"Error: {e}")  # Generic error, hard to handle specifically
except Exception as e:
    print(f"Unexpected error: {e}")

After (Recommended):

from iterable import open_iterable
from iterable.exceptions import (
    WriteNotSupportedError,
    FormatParseError,
    ReadError,
    IterableDataError
)

try:
    with open_iterable('data.parquet', mode='w') as dest:
        dest.write({'key': 'value'})
except WriteNotSupportedError as e:
    print(f"Writing to {e.format_id} not supported: {e.reason}")
    # Handle unsupported write operation
except FormatParseError as e:
    print(f"Parse error in {e.format_id} at row {e.row_number}: {e.message}")
    # Handle parsing errors with context
except ReadError as e:
    print(f"Read error: {e.message}")
    # Handle read errors
except IterableDataError as e:
    # Catch all IterableData errors
    print(f"IterableData error: {e.message}")
    if e.error_code:
        print(f"Error code: {e.error_code}")

Migration Steps:

Import specific exceptions: Import the exceptions you need from iterable.exceptions
Update exception handlers: Replace generic ValueError/Exception catches with specific exception types
Use error codes: Check error_code attribute for programmatic error handling
Access exception attributes: Use format-specific attributes like format_id, row_number, byte_offset, etc.

Exception Mapping:

NotImplementedError → WriteNotSupportedError (for write operations)
NotImplementedError → FormatNotSupportedError (for unsupported formats)
ValueError (format parsing) → FormatParseError
ValueError (resource requirements) → ReadError or WriteError
ValueError (format detection) → FormatDetectionError
Generic Exception → Specific exception types based on context

Benefits:

Better error handling: Catch specific error types programmatically
Richer error context: Access row numbers, byte offsets, format IDs, etc.
Error codes: Use error codes for programmatic handling
Backward compatible: Old code still works, but new code can be more specific

Example: Handling Format-Specific Errors:

from iterable import open_iterable
from iterable.exceptions import FormatParseError

try:
    with open_iterable('data.csv') as source:
        for row in source:
            process(row)
except FormatParseError as e:
    # Access detailed error information
    print(f"Failed to parse {e.format_id} format")
    if e.filename:
        print(f"File: {e.filename}")
    if e.row_number:
        print(f"Row: {e.row_number}")
    if e.byte_offset:
        print(f"Byte offset: {e.byte_offset}")
    if e.original_line:
        print(f"Problematic line: {e.original_line}")
    # Handle or log the error appropriately

Example: Using Error Codes:

from iterable import open_iterable
from iterable.exceptions import IterableDataError

try:
    with open_iterable('data.unknown') as source:
        pass
except IterableDataError as e:
    if e.error_code == "FORMAT_DETECTION_FAILED":
        # Try with explicit format
        with open_iterable('data.unknown', format='csv') as source:
            pass
    elif e.error_code == "FORMAT_NOT_SUPPORTED":
        # Install missing dependencies or use different format
        print(f"Format not supported: {e.message}")
    else:
        # Handle other errors
        print(f"Error: {e.message}")

Context Manager Support

What Changed: Iterable Data now supports Python's context manager protocol (with statements).

Before (Still Supported):

from iterable import open_iterable

source = open_iterable('data.csv')
try:
    for row in source:
        process(row)
finally:
    source.close()

After (Recommended):

from iterable import open_iterable

# Recommended: Using context manager
with open_iterable('data.csv') as source:
    for row in source:
        process(row)
# File automatically closed

Migration Steps:

Replace try/finally blocks with with statements
Remove manual close() calls when using context managers
Old code still works - migration is optional but recommended

Benefits:

Cleaner, more Pythonic code
Automatic resource cleanup
Better error handling

Version 1.0.6

Enhanced Documentation

What Changed: Comprehensive documentation improvements with better examples and API reference.

Action Required: Review updated documentation for best practices and new patterns.

Improved Examples

What Changed: All examples updated to show best practices.

Action Required: Update your code to follow new patterns shown in documentation.

Version 1.0.5

DuckDB Engine Support

What Changed: Added optional DuckDB engine for high-performance querying.

Before:

from iterable import open_iterable

source = open_iterable('data.csv.gz')
for row in source:
    process(row)
source.close()

After (Optional Enhancement):

from iterable import open_iterable

# Use DuckDB engine for better performance on large files
with open_iterable('data.csv.gz', engine='duckdb') as source:
    total = source.totals()  # Fast row counting
    for row in source:
        process(row)

Migration Steps:

Install DuckDB: pip install duckdb
Add engine='duckdb' parameter when opening supported formats
Old code continues to work with internal engine

Benefits:

Faster queries on large CSV/JSONL files
Fast row counting
SQL-like operations

Pipeline Processing Framework

What Changed: Added pipeline() function for data transformation workflows.

Before:

from iterable import open_iterable

source = open_iterable('input.csv')
destination = open_iterable('output.jsonl', mode='w')

for row in source:
    transformed = transform(row)
    destination.write(transformed)

source.close()
destination.close()

After (Optional Enhancement):

from iterable import open_iterable
from iterable.pipeline import pipeline

with open_iterable('input.csv') as source:
    with open_iterable('output.jsonl', mode='w') as destination:
        def transform_record(record, state):
            return transform(record)
        
        pipeline(
            source=source,
            destination=destination,
            process_func=transform_record
        )

Migration Steps:

Import pipeline from iterable.pipeline.core
Refactor transformation logic into process_func
Use pipeline for progress tracking and error handling

Benefits:

Built-in progress tracking
Error handling framework
State management
Cleaner code structure

Bulk Operations Support

What Changed: Enhanced support for bulk read/write operations.

Before:

from iterable import open_iterable

dest = open_iterable('output.jsonl', mode='w')
for record in records:
    dest.write(record)
dest.close()

After (Recommended):

from iterable import open_iterable

with open_iterable('output.jsonl', mode='w') as dest:
    dest.write_bulk(records)  # Much faster

Migration Steps:

Collect records into batches
Use write_bulk() instead of individual write() calls
Use read_bulk() for reading multiple records

Benefits:

Significantly better performance
Reduced I/O operations
Better memory efficiency

General Migration Tips

Testing Your Migration

Test with small files first: Verify migration works with small test files
Compare outputs: Ensure migrated code produces same results
Check performance: Verify performance improvements (if applicable)
Test error handling: Ensure error handling still works correctly

Backward Compatibility

Old code still works: Most changes are additive, not breaking
Gradual migration: You can migrate incrementally
No forced changes: Old patterns remain supported

Common Migration Patterns

Pattern 1: Adding Context Managers

# Old
source = open_iterable('data.csv')
try:
    for row in source:
        process(row)
finally:
    source.close()

# New
with open_iterable('data.csv') as source:
    for row in source:
        process(row)

Pattern 2: Using Bulk Operations

# Old
dest = open_iterable('output.jsonl', mode='w')
for record in records:
    dest.write(record)
dest.close()

# New
with open_iterable('output.jsonl', mode='w') as dest:
    dest.write_bulk(records)

Pattern 3: Adding DuckDB Engine

# Old
source = open_iterable('large_data.csv.gz')
for row in source:
    process(row)
source.close()

# New (optional, for better performance)
with open_iterable('large_data.csv.gz', engine='duckdb') as source:
    total = source.totals()  # Fast counting
    for row in source:
        process(row)

Pattern 4: Using Factory Methods

# Old: Traditional initialization
from iterable.datatypes.csv import CSVIterable

source = CSVIterable(filename='data.csv', mode='r', encoding='utf-8')
try:
    for row in source:
        process(row)
finally:
    source.close()

# New: Factory method (optional, clearer intent)
from iterable.datatypes.csv import CSVIterable

with CSVIterable.from_file('data.csv', encoding='utf-8') as source:
    for row in source:
        process(row)

Pattern 5: Adding Type Hints

# Old: No type hints
def process_csv(filename):
    with open_iterable(filename) as source:
        return [row for row in source]

# New: With type hints (optional, improves IDE support)
from typing import Any
from iterable import open_iterable

def process_csv(filename: str) -> list[dict[str, Any]]:
    with open_iterable(filename) as source:
        return [row for row in source]

Version 1.1.0 and Later (Architecture Improvements)

Factory Methods for Initialization

What Changed: Added factory methods (from_file(), from_stream(), from_codec()) for clearer initialization. The traditional __init__() method remains fully supported for backward compatibility.

Before:

from iterable.datatypes.csv import CSVIterable

# Traditional initialization
source = CSVIterable(filename='data.csv', mode='r', encoding='utf-8')

After (Optional Enhancement):

from iterable.datatypes.csv import CSVIterable

# Factory method - clearer intent
source = CSVIterable.from_file('data.csv', encoding='utf-8')

# Or with stream
import io
stream = io.StringIO("id,name\n1,test\n")
source = CSVIterable.from_stream(stream)

Migration Steps:

Factory methods are optional - old code continues to work
Use factory methods for clearer code intent
Factory methods provide better validation and error messages
All factory methods support the same options parameter

Benefits:

Clearer API - intent is explicit (from_file vs from_stream)
Better validation - errors caught earlier
Protected attributes - prevents accidental overrides
Backward compatible - old code still works

Type Hint Improvements

What Changed: Comprehensive type hints added throughout the library. This improves IDE support and static type checking but doesn't affect runtime behavior.

Before:

# No type hints - unclear what types are expected
def process_file(filename):
    source = open_iterable(filename)
    return list(source)

After (Optional Enhancement):

from typing import Any
from iterable import open_iterable

# Type hints improve IDE support and static checking
def process_file(filename: str) -> list[dict[str, Any]]:
    with open_iterable(filename) as source:
        return list(source)

Migration Steps:

No code changes required - type hints are additive
Use type hints in your code for better IDE support
Run mypy for static type checking (optional)
Type hints help catch errors before runtime

Benefits:

Better IDE autocomplete and error detection
Static type checking with mypy
Self-documenting code
No runtime impact

Improved Initialization Validation

What Changed: Better validation of initialization parameters, including protection against overriding internal attributes.

Before:

# Could accidentally override internal attributes
source = CSVIterable(filename='data.csv', options={'stype': 'invalid'})
# Might cause unexpected behavior

After:

# Protected attributes prevent accidental overrides
try:
    source = CSVIterable.from_file('data.csv', options={'stype': 'invalid'})
except ValueError as e:
    print(f"Error: {e}")  # Clear error message

Migration Steps:

No changes required - validation is automatic
If you see ValueError: Cannot override protected attribute, remove that parameter from options
Use public API methods instead of trying to override internal state

Benefits:

Prevents accidental bugs
Clearer error messages
More robust initialization

Breaking Changes

None in Recent Versions

All recent versions maintain backward compatibility. The changes in Phase 3 (factory methods, type hints, improved validation) are additive and don't break existing code:

✅ Old initialization patterns still work
✅ No API changes that break existing code
✅ Type hints are optional and don't affect runtime
✅ Factory methods are optional enhancements

If you encounter any issues:

Check the CHANGELOG for detailed changes
Review Troubleshooting Guide for solutions
Report issues on GitHub

Getting Help

If you need help with migration:

Check documentation: Review updated guides and examples
Review examples: Check use case examples for patterns
Test incrementally: Migrate one feature at a time
Ask for help: Open an issue on GitHub if you encounter problems

Troubleshooting Guide - Common issues and solutions
Best Practices - Recommended patterns
API Reference - Full API documentation
CHANGELOG - Detailed version history

Version 1.0.7 and Later​

Exception Hierarchy Improvements​

Context Manager Support​

Version 1.0.6​

Enhanced Documentation​

Improved Examples​

Version 1.0.5​

DuckDB Engine Support​

Pipeline Processing Framework​

Bulk Operations Support​

General Migration Tips​

Testing Your Migration​

Backward Compatibility​

Common Migration Patterns​

Pattern 1: Adding Context Managers​

Pattern 2: Using Bulk Operations​

Pattern 3: Adding DuckDB Engine​

Pattern 4: Using Factory Methods​

Pattern 5: Adding Type Hints​

Version 1.1.0 and Later (Architecture Improvements)​

Factory Methods for Initialization​

Type Hint Improvements​

Improved Initialization Validation​

Breaking Changes​

None in Recent Versions​

Getting Help​

Related Topics​

Version 1.0.7 and Later

Exception Hierarchy Improvements

Context Manager Support

Version 1.0.6

Enhanced Documentation

Improved Examples

Version 1.0.5

DuckDB Engine Support

Pipeline Processing Framework

Bulk Operations Support

General Migration Tips

Testing Your Migration

Backward Compatibility

Common Migration Patterns

Pattern 1: Adding Context Managers

Pattern 2: Using Bulk Operations

Pattern 3: Adding DuckDB Engine

Pattern 4: Using Factory Methods

Pattern 5: Adding Type Hints

Version 1.1.0 and Later (Architecture Improvements)

Factory Methods for Initialization

Type Hint Improvements

Improved Initialization Validation

Breaking Changes

None in Recent Versions

Getting Help

Related Topics