loading
Loading content
loading
Utilities
Regex-clean a wordlist before the slow stage reads it.
overview
Sit clean-wordlist between a list producer and a consumer. After CeWL, a merge, or an archive dump, it writes a tidied file so ffuf or a DNS resolver does not burn budget on garbage lines.
Reach for it when a list came from a crawl, scrape, or merge. Malformed tokens, blank lines, and encoding artifacts inflate request counts without adding hits.
Input is a single wordlist file; output is a FILE plus a FOLDER. It is a regex cleaner, not a deduper. For uniqueness alone, prefer duplicut or anew. Trickest runs it as a managed Utilities node.
source github.com/BonJarber/SecUtils/tree/master/clean_wordlist
use cases
Run a harvested wordlist through clean-wordlist so a content fuzzer like ffuf or gobuster spends its requests on plausible paths instead of malformed lines.
Clean the output of a generator like CeWL to remove crawl artifacts and noise before the list reaches a cracker or resolver.
After combining several wordlists, run clean-wordlist to drop the malformed and encoding-broken entries that merging tends to introduce.
Clean a subdomain wordlist before a DNS brute-force node reads it, so a resolver like puredns spends its lookups on real candidate labels rather than junk.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | · | Input wordlist file to clean (positional; the tool exposes no flags). |
Showing key inputs. clean-wordlist exposes 1 inputs in total.
example
# clean a harvested wordlist, cleaned entries to a new fileclean_wordlist.sh raw-wordlist.txt > clean-wordlist.txtadminapiapi/v1assetsbackupconfigloginrobots.txtuploadswp-login.phpguidance
Use clean-wordlist when a list comes from a crawl, archive, or merge and you want junk gone before a slow stage reads it. For deduplication alone, duplicut is sharper. For generating words, use CeWL, then clean afterward.
Removes duplicates from very large wordlists fast. clean-wordlist drops malformed entries via regex.
Appends only new lines to a file. Useful for dedupe across runs, not for cleaning malformed entries.
Generates a wordlist from a site. Run clean-wordlist after it to tidy harvested words.
faq
related
Spider a URL and return a wordlist for password crackers.
Wordlist and mined-word subdomain permutation.
CSS selectors over HTML, the jq counterpart for markup.
Extract chosen URL parts from stdin into clean line lists.
Import, export, and upsert workflow data against an Airtable base.
Decode Android APK files into smali sources and resources.
A raw wordlist feeds clean-wordlist, which runs its regexes over each line and writes a tidied list ready for fuzzing or cracking.
Facts on this page come from the live Trickest tool library.