Utilities
Prep a wordlist so fuzzers waste fewer requests
Regex-clean a wordlist before the slow stage reads it.
overview
What clean-wordlist does
Sit clean-wordlist between a list producer and a consumer. After CeWL, a merge, or an archive dump, it writes a tidied file so ffuf or a DNS resolver does not burn budget on garbage lines.
Reach for it when a list came from a crawl, scrape, or merge. Malformed tokens, blank lines, and encoding artifacts inflate request counts without adding hits.
Input is a single wordlist file; output is a FILE plus a FOLDER. It is a regex cleaner, not a deduper. For uniqueness alone, prefer duplicut or anew. Trickest runs it as a managed Utilities node.
source github.com/BonJarber/SecUtils/tree/master/clean_wordlist
use cases
Where clean-wordlist fits
Strip junk before fuzzing
Run a harvested wordlist through clean-wordlist so a content fuzzer like ffuf or gobuster spends its requests on plausible paths instead of malformed lines.
Tidy a generated list
Clean the output of a generator like CeWL to remove crawl artifacts and noise before the list reaches a cracker or resolver.
Normalize merged sources
After combining several wordlists, run clean-wordlist to drop the malformed and encoding-broken entries that merging tends to introduce.
Prep a subdomain wordlist
Clean a subdomain wordlist before a DNS brute-force node reads it, so a resolver like puredns spends its lookups on real candidate labels rather than junk.
reference
clean-wordlist inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | · | Input wordlist file to clean (positional; the tool exposes no flags). |
Showing key inputs. clean-wordlist exposes 1 inputs in total.
example
Run clean-wordlist
# clean a harvested wordlist, cleaned entries to a new fileclean_wordlist.sh raw-wordlist.txt > clean-wordlist.txtadminapiapi/v1assetsbackupconfigloginrobots.txtuploadswp-login.phpguidance
Choosing clean-wordlist
Use clean-wordlist when a list comes from a crawl, archive, or merge and you want junk gone before a slow stage reads it. For deduplication alone, duplicut is sharper. For generating words, use CeWL, then clean afterward.
duplicut
Removes duplicates from very large wordlists fast. clean-wordlist drops malformed entries via regex.
anew
Appends only new lines to a file. Useful for dedupe across runs, not for cleaning malformed entries.
cewl
Generates a wordlist from a site. Run clean-wordlist after it to tidy harvested words.
faq
clean-wordlist questions
related
More Utilities tools
browser-fetch
Headless Chromium render: requested URL, final URL, status, and title next to the page.
cewl
Spider a URL and return a wordlist for password crackers.
csvkit
SQL on one CSV. The table name is the file stem.
dnsgen
Wordlist and mined-word subdomain permutation.
duckdb
In-process SQL on CSV, JSON, and Parquet. No server.
edge-tts
Microsoft Edge neural speech from text or a script file. No API key.
Run clean-wordlist yourself
A raw wordlist feeds clean-wordlist, which runs its regexes over each line and writes a tidied list ready for fuzzing or cracking.
Facts on this page come from the live Trickest tool library.