⚡️ ComboSorter

A high-performance Python toolkit for processing and sorting large structured text datasets. Optimised for speed and memory efficiency on high-volume files.

---

🚀 Highlights

• Streaming Architecture: Processes files line-by-line. Handle multi-gigabyte files without crashing your RAM.
• Module Chaining: Apply multiple transformations in a single pass (e.g., Sanitize → Filter → Dedupe).
• GNU Sort Integration: Uses external gsort for near-instant deduplication and alphabetization on high-end hardware.
• Case Preservation: Intelligently handles email normalization while preserving password case sensitivity.
• Safe Sanitization: Conservative regex scripts to clean malformed emails without losing valid data.

---

🛠 Installation & Setup

No complex installation is required. This is a standalone Python utility.

Requirements:

• Python 3.8+
• (Optional) GNU Coreutils: Recommended for the fastest deduplication on macOS/Linux.

  # macOS
  brew install coreutils

Running the Tool:

Interactive Mode:

python3 main.py

Batch / CLI Mode:

# Process one file with Extract E:P (H) and Remove Duplicates (G)
python3 main.py --input messy_logs.txt --modules H,G --out-dir ./output

# Process multiple files or every immediate file in a directory
python3 main.py --input first.txt second.txt --modules H,G
python3 main.py --input ./input-directory --modules H,G

# Also produce one combined result in input order
python3 main.py --input first.txt second.txt --modules H,G --combine-output combined.txt

# Suppress status output, or show full paths and detailed module progress
python3 main.py --input first.txt --modules H,G --quiet
python3 main.py --input first.txt --modules H,G --verbose

--input accepts one or more files and non-recursively expands regular, non-symlink files from supplied directories. Duplicate path and hardlink aliases are rejected. Without --combine-output, each input retains its own per-file output; the batch is rejected before processing if a planned output would overwrite another input or output. With --combine-output PATH, inputs are processed in a private temporary workspace and only the combined result is retained. The combined file is private (0600) and atomically replaced rather than appended, so reruns neither duplicate earlier results nor pollute an input directory with generated files.

Normal CLI output uses compact per-file status lines and basenames. Live speed is shown only for work that runs long enough to measure meaningfully on an interactive terminal. --quiet suppresses progress and success messages; errors and requested data such as Domain Stats remain visible. --verbose restores detailed module progress and full paths. ANSI color is used only on an interactive terminal and is disabled when NO_COLOR is set.

Domain splitting defaults to the directory containing main.py; override it with --out-dir PATH. With multiple inputs, all preceding modules run in isolated workspaces and the processed streams are merged before one split operation. Domain files use private temporary outputs, serialized publication, and rollback if any file cannot be committed. A private recovery journal restores the previous complete generation after an interrupted or crashed publication on the next Split Domain run; backup data and deletion authority are never stored in the output directory. Stale-output tracking is likewise kept in private application state, so files placed in the output directory cannot authorize deletion. Managed domain outputs are not re-ingested when the same directory is used as both batch input and split output.

---

🧩 Available Modules

Key	Module	Description
0	Normal Edit	Trims whitespace and removes empty lines.
1	Strong Edit	Basic email validation + conservative sanitization.
2	Extreme Edit	Strict validation + Sanitization + Lowercase Email + Email/Pass Dedupe.
3	Capitalize	Converts the email/username part to UPPERCASE.
4	Decapitalize	Converts the email/username part to lowercase.
5	Randomize	Shuffles the list (⚠️ requires RAM for the shuffle).
6	Alphabetize	Sorts the list (uses gsort if available).
7	Domain Filter	Keep only lines matching a specific domain (e.g., gmail.com).
8	Country Filter	Keep only lines matching a TLD/Country code (e.g., .fr).
9	U/P to E/P	Converts user:pass to user@domain:pass.
A	E/P to U/P	Drops the domain, converting user@domain:pass to user:pass.
B	Custom Append	Appends a custom string to either the Left or Right part.
C	Password Length	Filters combos based on a Min/Max password length range.
D	Email Length	Filters combos based on a Min/Max email/username length.
E	Remove Custom	Removes specific characters or Regex patterns from either part.
F	Split Domain	Breaks a single large list into individual {domain}.txt files.
G	Remove Duplicate	High-speed deduplication (supports gsort parallel processing).
H	Extract E:P	Extracts valid email:password pairs from messy/unstructured logs.
I	Domain Stats	Analyzes and displays statistics for the top 20 domains in the list.

---

💡 Advanced Usage: Chaining

ComboSorter supports sequential chaining. Instead of running one operation at a time, you can input a comma-separated list of modules. F (Split Domain) is a terminal fan-out operation: it must be the final module and returns the output directory rather than a single file.

Example: 2,G,6

1. Extreme Edit: Clean and normalize the list.
2. Remove Duplicate: Ensure every line is unique.
3. Alphabetize: Produce a perfectly ordered final result.

---

⚙️ Configuration (Environment Variables)

You can customize the engine behavior by setting environment variables before running the script:

Variable	Default	Description
COMBOSORTER_SANITIZE	1	Set to 0 to disable auto-cleaning of emails.
COMBOSORTER_STRICT	0	Set to 1 to enable stricter email-format validation.
COMBOSORTER_LOG_LEVEL	INFO	Set to DEBUG for detailed processing logs.
COMBOSORTER_STATE_DIR	$XDG_STATE_HOME/combosorter or ~/.local/state/combosorter	Overrides the private Split Domain state directory.

---

🛡 License & Safety

• Private outputs: Generated files are restricted to the current user (0600). Split Domain keeps at most 64 output files open at once.
• Keep Backups: Always keep a copy of your source .txt files.
• Encoding: The tool defaults to UTF-8 with ignore for invalid characters to ensure non-stop processing of messy lists.

---

Developed with focus on performance and reliability.