Directory & Content Discovery

Overview

The interesting part of a web application is usually the part nobody linked to. Backup archives, staging routes left enabled, admin panels on a path that was never meant to be guessable, an old API version still mounted next to the current one. None of it appears in a crawl, because nothing points at it.

This is the brute-force pass that finds it. You get the paths that answered, with status and size, per host, which is the raw material for deciding what deserves a closer look. It is one of the standing offensive security passes on any surface that changes, because a path only has to be added once to stay reachable for years.

A fuzzer on its own is one host, one wordlist on your disk, one machine. Here the target is probed first, so requests are only spent on hosts that actually serve HTTP, the wordlist is fetched at run time rather than baked in, so coverage improves when the upstream list does without you editing anything, and the fuzzing is sharded across the fleet, so a large list against many hosts stays a single run rather than an afternoon.

Pipeline

1. Read the target.
2. httpx confirms which hosts are live and worth fuzzing.
3. A wordlist is fetched at run time so the workflow is not pinned to a stale copy.
4. ffuf-multi brute-forces paths against every live host, distributed across the fleet.
5. Responses are extracted and normalized into a single set of discovered URLs.
6. Findings are written to a PDF report.

Inputs

• Target. The web host to assess, as a single domain or hostname. The safe default is example.com, which runs clean and small. Replace it with a host you are authorized to test.
• Wordlist URL. The raw list pulled at run time. The default is the SecLists common.txt set, roughly 4.7k paths, which is the right first pass. Point it at any other raw wordlist URL when you want more depth or a stack-specific list.
• Request rate and threads. The pace the fuzzer holds to, rate-limited to 50 requests per second by default with a per-process time cap. That default is chosen to be polite on a target you do not own. Raise it only on your own.

Outputs

• Discovered paths. Every path that answered, with status, size and content type, with the interesting ones flagged.
• Probed-host inventory. The hosts confirmed to serve HTTP, with status, title and server. This is worth having even on a run that finds no paths.
• Content Discovery PDF. One report collecting every fuzz shard plus the methodology used, so a result is reproducible months later.

Sample output

From a completed run against scanme.nmap.org, the host Nmap publishes for this purpose.

httpx confirms the one live host before any brute force starts:

Roughly 4.7k paths were requested against it. Ten answered:

/.hta          403    286
/.htaccess     403    291
/.htpasswd     403    291
/.svn          403    287
/.svn/entries  403    294
/favicon.ico   403    293
/shared        403    289
/images        200    947
/index         200   6974
/index.html    200   6974

Size matters as much as status. Those seven 403s land within nine bytes of each other, which is one Apache deny page rather than seven findings, and /index and /index.html are the same 6,974-byte document counted twice. Reading the sizes is what stops you filing all ten.

FAQ

Why probe the hosts before fuzzing them?

Only hosts that answer over HTTP get fuzzed. A wordlist against a name that resolves but serves nothing spends thousands of requests to learn what one probe would have told you.

Can I use my own wordlist?

Yes. The list is fetched from a URL at run time, so swapping it is a matter of changing that URL to any raw list you host. Nothing is baked into the workflow.

Will this overwhelm the target?

The fuzzer is rate-limited to 50 requests per second by default and carries a per-process time cap, and it stops on a host that answers mostly 403. Treat those defaults as the ceiling on a target you do not own.

Why do so many paths come back as 200?

Some applications answer every path with a soft 404. Auto-calibration filters the obvious wildcard pages, and the response sizes in the report are what let you recognise the rest as one handler rather than a hundred findings.

What does running it on a schedule buy?

New routes ship with releases and staging paths get left enabled. A recurring run against the same host turns the path list into a diff, so a directory that appeared this month is visible without rereading the whole report.

Related workflows

• Find Secrets in Wayback Responses. Reach for this instead when the paths you want are already recorded in web archives, so you can find them without sending brute-force traffic.
• Scan WordPress for Known CVEs. Reach for this instead when the host fingerprints as WordPress and a CVE-matched scan beats guessing paths from a wordlist.
• API IDOR and BOLA Scanner (OpenAPI). Reach for this instead when the brute force turned up an OpenAPI spec and the interesting surface is now the API rather than the filesystem.