How do I control memory and thread use?

Pass --memlimit to cap peak RAM (for example 2G) and --threads to set the worker count. Both default to the machine maximum, so tune them down when duplicut shares a box with other nodes.

Does duplicut change list contents beyond duplicates?

Only when you ask. By default it drops duplicates and nothing else. --lowercase, --uppercase, and --printable rewrite or filter entries, and --line-max-size (default 64) drops longer lines.

How does duplicut compare to anew?

anew appends only new lines to a growing file as a stream passes through. duplicut dedupes any flat wordlist in bulk without reordering it, which is the prep step before fuzzing.

Where does the cleaned list go on Trickest?

The duplicut node writes a FILE and a FOLDER. Wire that output into ffuf, gobuster, or a cracking node.

Updated Jul 14, 2026

Utilities

Strip wordlist duplicates without sorting away probability

Deduplicate a wordlist without sorting so probability order stays intact.

Clean a merged wordlist before fuzzing Deduplicate multi-gigabyte lists under a memory cap Normalize case and strip junk in one pass

overview

What duplicut does

Before ffuf or a cracker, run duplicut on the merged wordlist so downstream nodes do not burn requests or hashes on repeats.

Reach for it when you concatenated SecLists or custom lists and need duplicates gone without a sort that wrecks guess order.

Input is a FILE wordlist; outputs are FILE and FOLDER. Set --threads and --memlimit when the list is multi-gigabyte.

source github.com/nil0x42/duplicut

use cases

Where duplicut fits

Clean a merged wordlist before fuzzing

Concatenate several lists into one, then run duplicut to strip the overlap while keeping probability order, so ffuf or gobuster never spends a request on a repeated guess.

Deduplicate multi-gigabyte lists under a memory cap

Set --threads and --memlimit so a huge wordlist dedupes fast on a worker without swapping or getting killed for exceeding RAM.

Normalize case and strip junk in one pass

Fold entries with --lowercase or --uppercase and drop non-printable lines with --printable during the dedup, producing a tidy list without reaching for a second tool.

Prep password lists for cracking

Dedupe a rockyou-plus-custom merge before feeding hashcat or john, cutting cracking time by removing candidates the tool would otherwise try twice.

reference

duplicut inputs and flags

7 inputs

Name	Type	Flag	Description
input	FILE	·	Wordlist file to deduplicate.
threads	STRING	--threads	Max threads to use (default max).
memory-limit	STRING	--memlimit	Cap peak memory so large lists do not exhaust RAM (default max).
maximum-line-size	STRING	--line-max-size	Max line size (default 64). Lines longer than this are dropped.
lowercase	BOOLEAN	--lowercase	Convert every entry to lowercase during the dedup.
uppercase	BOOLEAN	--uppercase	Convert every entry to uppercase during the dedup.
printable	BOOLEAN	--printable	Keep only ASCII-printable lines, dropping the rest.

Showing key inputs. duplicut exposes 7 inputs in total.

example

Run duplicut

duplicut · command

# concatenate lists, then dedupe without reorderingcat rockyou.txt custom-paths.txt > merged.txtduplicut merged.txt -o clean.txt --threads 8 --memlimit 2G

sample output

123456password123456789qwertyabc123password1iloveyouadminletmein…  (3,221,225 duplicate lines removed; probability order preserved)

guidance

Choosing duplicut

Use duplicut to dedupe a wordlist while keeping probability order, right before fuzzing or brute forcing. To filter domains by level instead, use dsieve. To append only new lines to a growing file, use anew.

anew

Appends only lines not already present in a file. duplicut deduplicates a whole list at once without reordering it.

dsieve

Filters and extracts domains by subdomain level. Different list-shaping job than bulk wordlist dedupe.

faq

duplicut questions

Sorting destroys the probability order a good wordlist depends on. duplicut removes duplicate lines while preserving the original order, so a fuzzer or password cracker still tries likely guesses first.

Run duplicut yourself

A raw wordlist feeds duplicut, which strips duplicates while keeping probability order and passes the clean list to ffuf so discovered paths land as output.

Open duplicut in the library Build a workflow with it

Facts on this page come from the live Trickest tool library.