Skip to main content

Error handling patterns

This document describes error handling patterns used in undatum for contributors and maintainers.

Overview​

All commands should use the centralized error handling infrastructure in undatum/common/errors.py to provide consistent, user-friendly error messages.

Error Handling Infrastructure​

Custom Exception Classes​

All custom exceptions inherit from UndatumError:

from undatum.common.errors import (
    FileNotFoundError,
    PermissionError,
    ValidationError,
    FormatError,
    DependencyError,
    DatabaseError,
    ConfigurationError,
)

Exception Hierarchy​

UndatumError (base)
├── FileNotFoundError
├── PermissionError
├── ValidationError
├── FormatError
├── ConfigurationError
├── DependencyError
└── DatabaseError

Common Patterns​

1. File Path Validation​

Always validate file paths before processing:

from undatum.common.path_utils import validate_file_path
from undatum.common.errors import FileNotFoundError, PermissionError, find_similar_files

def my_command(fromfile, options=None):
    # Validate input file exists and is readable
    try:
        validate_file_path(fromfile, check_read=True)
    except FileNotFoundError as e:
        suggestions = find_similar_files(fromfile)
        raise FileNotFoundError(fromfile, suggestions) from e
    except PermissionError as e:
        raise PermissionError(fromfile, operation="read") from e

2. Output File Validation​

For output files, check write permissions:

from undatum.common.path_utils import validate_file_path
from undatum.common.errors import PermissionError

def my_command(fromfile, tofile, options=None):
    # Validate output file can be written
    try:
        validate_file_path(tofile, check_write=True)
    except PermissionError as e:
        raise PermissionError(tofile, operation="write") from e

3. Field Validation​

Validate field names with suggestions:

from undatum.common.errors import ValidationError, find_similar_field_names

def my_command(fromfile, options=None):
    fields = options.get('fields')
    if not fields:
        raise ValidationError(
            "Missing required parameter 'fields'",
            field="fields"
        )
    
    available_fields = get_available_fields(fromfile)
    invalid_fields = [f for f in fields if f not in available_fields]
    
    if invalid_fields:
        for field in invalid_fields:
            suggestions = find_similar_field_names(field, available_fields)
            raise ValidationError(
                f"Field '{field}' does not exist",
                field=field,
                suggestions=suggestions
            )

4. Format Validation​

Validate file formats with supported formats list:

from undatum.common.errors import FormatError

SUPPORTED_FORMATS = ['csv', 'jsonl', 'parquet', 'avro']

def my_command(fromfile, options=None):
    file_type = detect_file_type(fromfile)
    if file_type not in SUPPORTED_FORMATS:
        raise FormatError(
            fromfile,
            file_type,
            supported_formats=SUPPORTED_FORMATS
        )

5. Dependency Checking​

Check for optional dependencies:

from undatum.common.errors import DependencyError

def my_command(fromfile, options=None):
    try:
        import yaml
    except ImportError:
        raise DependencyError(
            'pyyaml',
            feature='YAML template support'
        )

6. Database Error Handling​

Handle database errors with masked connection URIs:

from undatum.common.errors import DatabaseError

def my_command(connection_uri, options=None):
    try:
        conn = connect_to_database(connection_uri)
    except Exception as e:
        raise DatabaseError(
            f"Connection failed: {str(e)}",
            db_type="postgresql",
            connection_uri=connection_uri  # Will be masked automatically
        ) from e

7. Validation Error with Context​

Provide context for validation errors:

from undatum.common.errors import ValidationError

def my_command(fromfile, options=None):
    method = options.get('method')
    valid_methods = ['redact', 'hash', 'mask']
    
    if method not in valid_methods:
        raise ValidationError(
            f"Invalid method '{method}'",
            field="method",
            suggestions=valid_methods,
            context={
                "valid_options": valid_methods,
                "provided": method
            }
        )

Error Message Guidelines​

1. Be Clear and Actionable​

Bad:

raise ValueError("Error occurred")

Good:

raise ValidationError(
    "Missing required parameter 'fields'",
    field="fields"
)

2. Provide Suggestions​

Bad:

raise FileNotFoundError("/path/to/file.csv")

Good:

suggestions = find_similar_files("/path/to/file.csv")
raise FileNotFoundError("/path/to/file.csv", suggestions)

3. Include Context​

Bad:

raise ValidationError("Invalid value")

Good:

raise ValidationError(
    f"Invalid method '{method}'",
    field="method",
    suggestions=['redact', 'hash', 'mask']
)

4. Mask Sensitive Information​

Bad:

raise DatabaseError(f"Connection failed: {connection_uri}")

Good:

raise DatabaseError(
    "Connection failed",
    connection_uri=connection_uri  # Automatically masked
)

Exit Codes​

Use appropriate exit codes:

  • 1: User errors (invalid input, file not found)
  • 2: Configuration errors (missing dependencies, invalid config)
  • 3: System errors (permission denied, database errors)
  • 4: Internal errors (unexpected exceptions)

Exit codes are automatically set by exception classes. Don't manually set sys.exit().

Global Error Handling​

The global error handler in undatum/__main__.py automatically:

  • Catches all exceptions
  • Formats error messages
  • Sets appropriate exit codes
  • Shows full tracebacks in verbose mode

You don't need to handle errors at the command level unless you want to add specific context.

Testing Error Handling​

Test error scenarios in your command tests:

import pytest
from undatum.common.errors import FileNotFoundError, ValidationError

def test_command_file_not_found():
    """Test command with nonexistent file."""
    from undatum.cmds.my_command import MyCommand
    
    command = MyCommand()
    with pytest.raises(FileNotFoundError):
        command.process("/nonexistent/file.csv", {})

def test_command_validation_error():
    """Test command with invalid parameters."""
    from undatum.cmds.my_command import MyCommand
    
    command = MyCommand()
    with pytest.raises(ValidationError) as exc_info:
        command.process("data.csv", {})  # Missing required option
    assert "fields" in str(exc_info.value).lower()

Best Practices​

  1. Always validate inputs early: Check file existence, permissions, and parameters at the start of command methods
  2. Use custom exceptions: Never raise generic ValueError or FileNotFoundError (Python built-in)
  3. Provide suggestions: Use find_similar_files() and find_similar_field_names() for typos
  4. Include context: Add relevant information to error messages
  5. Mask sensitive data: Never expose passwords or API keys in error messages
  6. Test error scenarios: Write tests for all error paths
  7. Follow patterns: Use existing commands as examples

Example: Complete Command with Error Handling​

"""Example command with proper error handling."""
from undatum.common.path_utils import validate_file_path
from undatum.common.errors import (
    FileNotFoundError,
    PermissionError,
    ValidationError,
    find_similar_files,
    find_similar_field_names,
)

class ExampleCommand:
    """Example command handler."""
    
    def process(self, fromfile, tofile, options=None):
        """Process file with comprehensive error handling."""
        if options is None:
            options = {}
        
        # Validate input file
        try:
            validate_file_path(fromfile, check_read=True)
        except FileNotFoundError as e:
            suggestions = find_similar_files(fromfile)
            raise FileNotFoundError(fromfile, suggestions) from e
        except PermissionError as e:
            raise PermissionError(fromfile, operation="read") from e
        
        # Validate output file
        try:
            validate_file_path(tofile, check_write=True)
        except PermissionError as e:
            raise PermissionError(tofile, operation="write") from e
        
        # Validate required parameters
        fields = options.get('fields')
        if not fields:
            raise ValidationError(
                "Missing required parameter 'fields'",
                field="fields"
            )
        
        # Validate field names
        available_fields = self._get_fields(fromfile)
        fields_list = fields.split(',') if isinstance(fields, str) else fields
        
        for field in fields_list:
            if field not in available_fields:
                suggestions = find_similar_field_names(field, available_fields)
                raise ValidationError(
                    f"Field '{field}' does not exist",
                    field=field,
                    suggestions=suggestions
                )
        
        # Process file
        # ... actual processing logic ...
    
    def _get_fields(self, fromfile):
        """Get available fields from file."""
        # ... implementation ...
        return ['field1', 'field2', 'field3']
  • undatum/common/errors.py - Error handling infrastructure
  • undatum/common/path_utils.py - File path validation utilities
  • undatum/__main__.py - Global error handler
  • tests/test_error_handling.py - Error handling tests