Skip to main content

GELF Format (Graylog Extended Log Format)

Description​

GELF (Graylog Extended Log Format) is a JSON-based log format used by Graylog and other log management systems. It's similar to JSONL but with specific fields required by the GELF specification. GELF messages are JSON objects sent one per line.

File Extensions​

  • .gelf - GELF format files

Implementation Details​

Reading​

The GELF implementation:

  • Reads JSONL-style format (one JSON object per line)
  • Parses each line as a JSON object
  • Validates GELF-specific fields (version, timestamp, short_message)
  • Converts each log entry to a dictionary

Writing​

Writing support:

  • Writes GELF-formatted JSON objects
  • Validates required GELF fields
  • Writes one JSON object per line

Key Features​

  • JSON-based: Uses JSON format
  • Structured logging: Designed for structured log data
  • GELF validation: Validates GELF-specific fields
  • Totals support: Can count total log lines
  • Nested data: Supports complex nested structures

Usage​

from iterable import open_iterable

# Basic reading
with open_iterable('logs.gelf') as source:
    for row in source:
        print(row)

# Writing
with open_iterable('output.gelf', mode='w') as dest:
    dest.write({
        'version': '1.1',
        'host': 'server1',
        'short_message': 'Test message',
        'timestamp': 1234567890.123,
        'level': 6
    })

Parameters​

  • encoding (str): File encoding (default: utf8)

GELF Required Fields​

  • version (string): GELF version (e.g., "1.1")
  • timestamp (number): Unix timestamp
  • short_message or message (string): Log message

Limitations​

  1. GELF validation: Requires GELF-specific fields
  2. Line-based: Each record must fit on a single line
  3. JSON parsing: Invalid JSON will cause errors
  4. Field requirements: Must include required GELF fields

Compression Support​

GELF files can be compressed with all supported codecs:

  • GZip (.gelf.gz)
  • BZip2 (.gelf.bz2)
  • LZMA (.gelf.xz)
  • LZ4 (.gelf.lz4)
  • ZIP (.gelf.zip)
  • Brotli (.gelf.br)
  • ZStandard (.gelf.zst)

Use Cases​

  • Graylog: Integrating with Graylog log management
  • Structured logging: Structured log data collection
  • Log aggregation: Aggregating logs from multiple sources
  • Monitoring: Application and system monitoring

Error Handling​

  • Missing dependency: optional libraries raise ImportError with an install hint (pip install 'iterabledata[<extra>]' when an extra exists).
  • Write mode: read-only formats raise WriteNotSupportedError or ValueError when opened with mode="w".
  • Bad or unsupported input: may raise ValueError, OSError, or library-specific errors.
  • See Troubleshooting for decoding, detection, and engine issues.
  • JSONL - Similar line-delimited JSON format
  • Apache Log - Web server log format
  • CEF - Security event format