loading
Loading content
loading
Utilities
Deduplicate a wordlist without sorting so probability order stays intact.
overview
Before ffuf or a cracker, run duplicut on the merged wordlist so downstream nodes do not burn requests or hashes on repeats.
Reach for it when you concatenated SecLists or custom lists and need duplicates gone without a sort that wrecks guess order.
Input is a FILE wordlist; outputs are FILE and FOLDER. Set --threads and --memlimit when the list is multi-gigabyte.
source github.com/nil0x42/duplicut
use cases
Concatenate several lists into one, then run duplicut to strip the overlap while keeping probability order, so ffuf or gobuster never spends a request on a repeated guess.
Set --threads and --memlimit so a huge wordlist dedupes fast on a worker without swapping or getting killed for exceeding RAM.
Fold entries with --lowercase or --uppercase and drop non-printable lines with --printable during the dedup, producing a tidy list without reaching for a second tool.
Dedupe a rockyou-plus-custom merge before feeding hashcat or john, cutting cracking time by removing candidates the tool would otherwise try twice.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | · | Wordlist file to deduplicate. |
| threads | STRING | --threads | Max threads to use (default max). |
| memory-limit | STRING | --memlimit | Cap peak memory so large lists do not exhaust RAM (default max). |
| maximum-line-size | STRING | --line-max-size | Max line size (default 64). Lines longer than this are dropped. |
| lowercase | BOOLEAN | --lowercase | Convert every entry to lowercase during the dedup. |
| uppercase | BOOLEAN | --uppercase | Convert every entry to uppercase during the dedup. |
| printable | BOOLEAN | --printable | Keep only ASCII-printable lines, dropping the rest. |
Showing key inputs. duplicut exposes 7 inputs in total.
example
# concatenate lists, then dedupe without reorderingcat rockyou.txt custom-paths.txt > merged.txtduplicut merged.txt -o clean.txt --threads 8 --memlimit 2G123456password123456789qwertyabc123password1iloveyouadminletmein… (3,221,225 duplicate lines removed; probability order preserved)guidance
Use duplicut to dedupe a wordlist while keeping probability order, right before fuzzing or brute forcing. To filter domains by level instead, use dsieve. To append only new lines to a growing file, use anew.
Appends only lines not already present in a file. duplicut deduplicates a whole list at once without reordering it.
Filters and extracts domains by subdomain level. Different list-shaping job than bulk wordlist dedupe.
faq
related
Spider a URL and return a wordlist for password crackers.
Wordlist and mined-word subdomain permutation.
Import, export, and upsert workflow data against an Airtable base.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Authenticated Route53 hosted-zone inventory.
A raw wordlist feeds duplicut, which strips duplicates while keeping probability order and passes the clean list to ffuf so discovered paths land as output.
Facts on this page come from the live Trickest tool library.