Skip to main content

SequenceFile Format (Hadoop)

Description​

SequenceFile is a flat file format used by Apache Hadoop for storing key-value pairs. It's a binary format designed for efficient storage and retrieval in Hadoop. SequenceFile supports compression and is commonly used in Hadoop MapReduce jobs.

File Extensions​

  • .seq - SequenceFile files
  • .sequencefile - SequenceFile files (alias)

Implementation Details​

Reading​

The SequenceFile implementation:

  • Parses SequenceFile format
  • Reads file header with metadata
  • Extracts key-value pairs
  • Handles sync markers
  • Converts records to dictionaries

Writing​

Writing is not currently supported for SequenceFile format.

Key Features​

  • Hadoop format: Designed for Hadoop
  • Key-value pairs: Stores key-value data
  • Sync markers: Supports random access
  • Compression: Supports compression
  • Nested data: Supports complex data structures

Usage​

from iterable import open_iterable

# Basic reading
source = open_iterable('data.seq', iterableargs={
    'key_name': 'key',
    'value_name': 'value'
})
for record in source:
    print(record)  # Contains key and value
source.close()

Parameters​

  • key_name (str): Key name for record key (default: key)
  • value_name (str): Key name for record value (default: value)

Limitations​

  1. Read-only: SequenceFile format does not support writing
  2. Binary format: Not human-readable
  3. Hadoop-specific: Designed for Hadoop
  4. Format complexity: SequenceFile format can be complex
  5. Simplified implementation: May not support all SequenceFile features

Compression Support​

SequenceFile has built-in compression support (separate from file-level compression):

  • None, Record, Block compression

SequenceFile files can also be compressed with file-level codecs:

  • GZip (.seq.gz)
  • BZip2 (.seq.bz2)
  • LZMA (.seq.xz)
  • LZ4 (.seq.lz4)
  • ZIP (.seq.zip)
  • Brotli (.seq.br)
  • ZStandard (.seq.zst)

Use Cases​

  • Hadoop processing: Processing Hadoop data
  • MapReduce: MapReduce job data
  • Data storage: Storing data in Hadoop
  • Big data: Large-scale data processing
  • Avro - Another Hadoop format
  • Parquet - Columnar format for Hadoop