dedup
Removes duplicate rows. Can deduplicate by all fields or specified key fields.
Write with --output. A trailing path is not a positional argument.
# Deduplicate by all fields
undatum dedup data.csv --output output.csv
# Deduplicate by key fields
undatum dedup data.jsonl --key-fields email --output output.jsonl
# Keep last duplicate
undatum dedup data.csv --key-fields id --keep last --output output.csv
--key-fields— comma-separated keys (all fields if omitted)--keep first|last--output— output path (stdout if omitted)--low-memory,--engine--format-in— override type detection (this command does not take--format-in)
Also accepts --table, --flatten-nested, --on-error, --error-log, and --quotechar (shared options).
Reference
Reads: any readable format · Writes: any writable format; CSV/TSV/JSON/JSON Lines on stdout · Memory: streaming; up to 100,000 keys in memory, then a disk index (--low-memory uses the disk index from the start) · Engines: auto, duckdb, python
undatum dedup [OPTIONS] INPUT_FILE
| Argument | Description |
|---|---|
INPUT_FILE | Path to input file. (required) |
| Option | Description | Default |
|---|---|---|
-o, --output TEXT | Optional output file path. If not specified, prints to stdout. | |
--key-fields TEXT | Comma-separated list of field names to use for deduplication (default: all fields). | |
--keep TEXT | Which duplicate to keep: 'first' (default) or 'last'. | first |
-d, --delimiter TEXT | CSV delimiter character (auto-detected when omitted). | |
--quotechar TEXT | CSV quote character (iterabledata default '"' when omitted). | |
--encoding TEXT | File encoding (e.g., 'utf8', 'latin1'). | |
--verbose / --no-verbose | Enable verbose logging output. | --no-verbose |
-F, --format-in TEXT | Override file type detection (e.g., 'csv', 'jsonl'). | |
-e, --engine [auto|duckdb|python] | Processing engine: auto (default), duckdb, or python. | |
--duckdb-threads INTEGER | Number of threads for DuckDB engine. | |
--duckdb-memory TEXT | Memory limit for DuckDB (e.g., '4GB', '512MB'). | |
--duckdb-temp-dir TEXT | Temporary directory for DuckDB. | |
--low-memory / --no-low-memory | Force disk-backed exact deduplication (spill keys to SQLite). | --no-low-memory |
--table, --sheet TEXT | Table or sheet name for multi-table sources (Excel, SQLite, lakehouse). | |
--start-page INTEGER | Sheet index (0-based) for Excel files. | 0 |
--trust | Acknowledge pickle deserialization risk when reading pickle sources. | |
--on-error TEXT | Parse-error policy: raise (default), skip, or warn. | |
--error-log TEXT | Append parse errors as JSONL (use with --on-error skip or warn). | |
--flatten-nested | Unfold nested dict / array-of-dict fields into dotted paths (e.g. city.lat). | |
--max-nested-depth INTEGER | With --flatten-nested, maximum nest depth to unfold (engine default 5). | |
--keep-nested-parents / --no-keep-nested-parents | With --flatten-nested, keep parent dict/array fields alongside dotted children. | --keep-nested-parents |
-O, --format-out TEXT | Output format (e.g. csv, jsonl, parquet). Defaults to the --output extension, or to the input's text format on stdout. |
Deprecated spellings (removed in 2.0): --filetype → --format-in.
See also shared options.