Introducing Agents.·Read
OverviewAgentsWorkflowsExecutionDeveloper ToolsSecurityLibrary
Updated Jul 14, 2026

Utilities

Prep a wordlist so fuzzers waste fewer requests

Regex-clean a wordlist before the slow stage reads it.

Python · GPL-3.0 license

Agent
Strip junk before fuzzingTidy a generated listNormalize merged sources

overview

What clean-wordlist does

Sit clean-wordlist between a list producer and a consumer. After CeWL, a merge, or an archive dump, it writes a tidied file so ffuf or a DNS resolver does not burn budget on garbage lines.

Reach for it when a list came from a crawl, scrape, or merge. Malformed tokens, blank lines, and encoding artifacts inflate request counts without adding hits.

Input is a single wordlist file; output is a FILE plus a FOLDER. It is a regex cleaner, not a deduper. For uniqueness alone, prefer duplicut or anew. Trickest runs it as a managed Utilities node.

source github.com/BonJarber/SecUtils/tree/master/clean_wordlist

use cases

Where clean-wordlist fits

Strip junk before fuzzing

Run a harvested wordlist through clean-wordlist so a content fuzzer like ffuf or gobuster spends its requests on plausible paths instead of malformed lines.

Tidy a generated list

Clean the output of a generator like CeWL to remove crawl artifacts and noise before the list reaches a cracker or resolver.

Normalize merged sources

After combining several wordlists, run clean-wordlist to drop the malformed and encoding-broken entries that merging tends to introduce.

Prep a subdomain wordlist

Clean a subdomain wordlist before a DNS brute-force node reads it, so a resolver like puredns spends its lookups on real candidate labels rather than junk.

reference

clean-wordlist inputs and flags

1 inputs
NameTypeFlagDescription
inputFILE·Input wordlist file to clean (positional; the tool exposes no flags).

Showing key inputs. clean-wordlist exposes 1 inputs in total.

example

Run clean-wordlist

clean-wordlist · command
# clean a harvested wordlist, cleaned entries to a new file
clean_wordlist.sh raw-wordlist.txt > clean-wordlist.txt
sample output
admin
api
api/v1
assets
backup
config
login
robots.txt
uploads
wp-login.php

guidance

Choosing clean-wordlist

Use clean-wordlist when a list comes from a crawl, archive, or merge and you want junk gone before a slow stage reads it. For deduplication alone, duplicut is sharper. For generating words, use CeWL, then clean afterward.

duplicut

Removes duplicates from very large wordlists fast. clean-wordlist drops malformed entries via regex.

anew

Appends only new lines to a file. Useful for dedupe across runs, not for cleaning malformed entries.

cewl

Generates a wordlist from a site. Run clean-wordlist after it to tidy harvested words.

faq

clean-wordlist questions

The fixed regex series drops malformed lines, non-printable and encoding artifacts, blank lines, and tokens that will never be a valid path or word. What remains is a tidier set of plausible candidates.

It is a regex cleaner first, not a dedicated deduplicator. For removing duplicates from very large lists, duplicut or anew is more focused. Clean malformed entries here, then dedupe if you need strict uniqueness.

No. The only input is the file to clean. There are no flags and no tuning; the regex series runs automatically and writes a cleaned FILE plus a FOLDER.

Between a step that produces a list and one that reads it. Put it after CeWL or a wordlist merge, and before ffuf, gobuster, or a DNS resolver, so the slow stage only sees clean input.

Yes. The tool does not care what the list is for, so it cleans subdomain, content-discovery, and password lists the same way. Choosing which wordlist to start from is a separate selection step.

CeWL generates a wordlist by crawling a site and harvesting words. clean-wordlist generates nothing. Run a generator first, then pass its output through clean-wordlist to strip crawl artifacts.

related

More Utilities tools

  1. browser-fetch

    Headless Chromium render: requested URL, final URL, status, and title next to the page.

  2. cewl

    Spider a URL and return a wordlist for password crackers.

  3. csvkit

    SQL on one CSV. The table name is the file stem.

  4. dnsgen

    Wordlist and mined-word subdomain permutation.

  5. duckdb

    In-process SQL on CSV, JSON, and Parquet. No server.

  6. edge-tts

    Microsoft Edge neural speech from text or a script file. No API key.

Run clean-wordlist yourself

A raw wordlist feeds clean-wordlist, which runs its regexes over each line and writes a tidied list ready for fuzzing or cracking.

Open clean-wordlist in the libraryBuild a workflow with it

Facts on this page come from the live Trickest tool library.