Utilities
Prep a wordlist so fuzzers waste fewer requests
Regex-clean a wordlist before the slow stage reads it.
overview
What clean-wordlist does
Sit clean-wordlist between a list producer and a consumer. After CeWL, a merge, or an archive dump, it writes a tidied file so ffuf or a DNS resolver does not burn budget on garbage lines.
Reach for it when a list came from a crawl, scrape, or merge. Malformed tokens, blank lines, and encoding artifacts inflate request counts without adding hits.
Input is a single wordlist file; output is a FILE plus a FOLDER. It is a regex cleaner, not a deduper. For uniqueness alone, prefer duplicut or anew. Trickest runs it as a managed Utilities node.
source github.com/BonJarber/SecUtils/tree/master/clean_wordlist
use cases
Where clean-wordlist fits
Strip junk before fuzzing
Run a harvested wordlist through clean-wordlist so a content fuzzer like ffuf or gobuster spends its requests on plausible paths instead of malformed lines.
Tidy a generated list
Clean the output of a generator like CeWL to remove crawl artifacts and noise before the list reaches a cracker or resolver.
Normalize merged sources
After combining several wordlists, run clean-wordlist to drop the malformed and encoding-broken entries that merging tends to introduce.
Prep a subdomain wordlist
Clean a subdomain wordlist before a DNS brute-force node reads it, so a resolver like puredns spends its lookups on real candidate labels rather than junk.
reference
clean-wordlist inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | · | Input wordlist file to clean (positional; the tool exposes no flags). |
Showing key inputs. clean-wordlist exposes 1 inputs in total.
example
Run clean-wordlist
# clean a harvested wordlist, cleaned entries to a new fileclean_wordlist.sh raw-wordlist.txt > clean-wordlist.txtadminapiapi/v1assetsbackupconfigloginrobots.txtuploadswp-login.phpguidance
Choosing clean-wordlist
Use clean-wordlist when a list comes from a crawl, archive, or merge and you want junk gone before a slow stage reads it. For deduplication alone, duplicut is sharper. For generating words, use CeWL, then clean afterward.
duplicut
Removes duplicates from very large wordlists fast. clean-wordlist drops malformed entries via regex.
anew
Appends only new lines to a file. Useful for dedupe across runs, not for cleaning malformed entries.
cewl
Generates a wordlist from a site. Run clean-wordlist after it to tidy harvested words.
faq
clean-wordlist questions
related
More Utilities tools
cewl
Spider a URL and return a wordlist for password crackers.
dnsgen
Wordlist and mined-word subdomain permutation.
pup
CSS selectors over HTML, the jq counterpart for markup.
unfurl
Extract chosen URL parts from stdin into clean line lists.
whisper
Speech to txt, vtt, srt, and json from an audio file or folder.
youtube-transcript
Public caption tracks to transcript.txt and results.jsonl.
Run clean-wordlist yourself
A raw wordlist feeds clean-wordlist, which runs its regexes over each line and writes a tidied list ready for fuzzing or cracking.
Facts on this page come from the live Trickest tool library.