loading
Loading content
loading
Utilities
Remove duplicates from a wordlist without sorting it to maintain order of probability.
overview
duplicut removes duplicate lines from a wordlist without sorting it. A strong wordlist is ordered by probability so the likeliest guesses come first, and a plain sort-then-uniq would scramble that order; duplicut strips the repeats and leaves the order untouched.
It is built for large lists. Multi-threading, a configurable memory limit, and a maximum line size let it churn through multi-gigabyte wordlists without exhausting RAM. Optional passes lowercase or uppercase every entry and drop non-printable lines in the same run.
Reach for it right before a fuzzing or brute-forcing stage so the list feeding ffuf, gobuster, or a password cracker is clean and free of wasted duplicate guesses.
source github.com/nil0x42/duplicut
use cases
Concatenate several lists into one, then run duplicut to strip the overlap while keeping probability order, so ffuf or gobuster never spends a request on a repeated guess.
Set --threads and --memlimit so a huge wordlist dedupes fast on a worker without swapping or getting killed for exceeding RAM.
Fold entries with --lowercase or --uppercase and drop non-printable lines with --printable during the dedup, producing a tidy list without reaching for a second tool.
Dedupe a rockyou-plus-custom merge before feeding hashcat or john, cutting cracking time by removing candidates the tool would otherwise try twice.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | · | Wordlist file to deduplicate. |
| threads | STRING | --threads | Max threads to use (default max). |
| memory-limit | STRING | --memlimit | Cap peak memory so large lists do not exhaust RAM (default max). |
| maximum-line-size | STRING | --line-max-size | Max line size (default 64). Lines longer than this are dropped. |
| lowercase | BOOLEAN | --lowercase | Convert every entry to lowercase during the dedup. |
| uppercase | BOOLEAN | --uppercase | Convert every entry to uppercase during the dedup. |
| printable | BOOLEAN | --printable | Keep only ASCII-printable lines, dropping the rest. |
Showing key inputs. duplicut exposes 7 inputs in total.
example
# concatenate lists, then dedupe without reorderingcat rockyou.txt custom-paths.txt > merged.txtduplicut merged.txt -o clean.txt --threads 8 --memlimit 2G123456password123456789qwertyabc123password1iloveyouadminletmein… (3,221,225 duplicate lines removed; probability order preserved)guidance
Use duplicut to dedupe a wordlist while keeping its probability order, right before a fuzzing or brute-forcing stage. To filter or extract domains by level instead of deduplicating, use dsieve. To append only new lines to a growing file, use anew.
Appends only lines not already present in a file. duplicut deduplicates a whole list at once without reordering it.
Filters and extracts domains by subdomain level. A sibling utility for a different list-shaping job.
faq
related
Import, export, and link workflow data with Airtable.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Extract all hosted zones from AWS Route53.
Output file lines by batch size, given START_LINE and END_LINE.
Extract a batch range from a file's lines or a folder's files, with parallel processing.
A raw wordlist feeds duplicut, which strips duplicates while keeping probability order and passes the clean list to ffuf so discovered paths land as output.
Facts on this page come from the live Trickest tool library.