loading
Loading content
loading
Utilities
Clean up a wordlist by running a series of regexes against it.
overview
clean-wordlist takes a wordlist and runs a fixed series of regexes over it to drop entries that waste a fuzzing or cracking run. Lists scraped from crawls, pulled from archives, or merged from several sources carry noise: malformed lines, encoding artifacts, blank lines, and tokens that will never be a valid path or word. Clearing that out up front means the next tool spends its requests on candidates that can hit.
The tool is deliberately small. Point it at an input file and it writes back the cleaned list, with no flags and no tuning. That shape makes it a tidy glue node between a step that produces a list and the tool that consumes it.
On Trickest, clean-wordlist is a Utilities node that reads a file and writes a file plus a folder of results. Drop it after a generator like CeWL or a wordlist merge, then feed the tidied list into a fuzzer such as ffuf or a DNS resolver so the heavy stage runs on clean input.
source github.com/BonJarber/SecUtils/tree/master/clean_wordlist
use cases
Run a harvested wordlist through clean-wordlist so a content fuzzer like ffuf or gobuster spends its requests on plausible paths instead of malformed lines.
Clean the output of a generator like CeWL to remove crawl artifacts and noise before the list reaches a cracker or resolver.
After combining several wordlists, run clean-wordlist to drop the malformed and encoding-broken entries that merging tends to introduce.
Clean a subdomain wordlist before a DNS brute-force node reads it, so a resolver like puredns spends its lookups on real candidate labels rather than junk.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | · | Input wordlist file to clean (positional; the tool exposes no flags). |
Showing key inputs. clean-wordlist exposes 1 inputs in total.
example
# clean a harvested wordlist, cleaned entries to a new fileclean_wordlist.sh raw-wordlist.txt > clean-wordlist.txtadminapiapi/v1assetsbackupconfigloginrobots.txtuploadswp-login.phpguidance
Use clean-wordlist whenever a list comes from a crawl, an archive, or a merge and you want the junk gone before a slow stage reads it. For deduplication alone, a tool like duplicut is sharper. For generating words in the first place, use CeWL, then clean afterward.
Removes duplicates from very large wordlists fast. clean-wordlist also drops malformed entries via regex.
Appends only new lines to a file. Useful for dedupe across runs, not for cleaning malformed entries.
Generates a wordlist from a site. Run clean-wordlist after it to tidy the harvested words.
faq
related
Import, export, and link workflow data with Airtable.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Extract all hosted zones from AWS Route53.
Output file lines by batch size, given START_LINE and END_LINE.
Extract a batch range from a file's lines or a folder's files, with parallel processing.
A raw wordlist feeds clean-wordlist, which runs its regexes over each line and writes a tidied list ready for fuzzing or cracking.
Facts on this page come from the live Trickest tool library.