Discovery
Pull archived URLs without touching the target
Passive known-URL fetch from public web archives.
overview
What gau does
gau (getallurls) builds a URL list for a domain without contacting the host. It queries Wayback, Common Crawl, AlienVault OTX, and urlscan for paths, parameters, and endpoints already recorded in public archives.
Use it early when the scope is sensitive or when you need history an active crawler will miss. Narrow with --providers, date windows via --from and --to, --blacklist for static assets, and --fp to collapse parameter variants.
Pass a domain or domains file, then take FILE and FOLDER URLs into httpx for liveness and a fuzzer. Pair with katana when you also need a live crawl of the current site.
source github.com/lc/gau
use cases
Where gau fits
Seed recon passively before touching the target
Pull every archived URL for a domain from Wayback, Common Crawl, OTX, and urlscan without sending a single request to the host, so reconnaissance starts wide and silent.
Recover endpoints removed from the live site
Surface old paths, deprecated APIs, and forgotten parameters that no longer appear on the running application but still live in public archives.
Build a fuzzing and injection candidate list
Collect parameterized URLs across subdomains, then collapse duplicates with parameter filtering so a fuzzer or injection scanner works from a clean, deduplicated set.
Combine passive and active discovery
Run gau alongside an active crawler such as Katana, then probe the merged URLs with httpx so the workflow covers both archive history and the current attack surface.
reference
gau inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| target | STRING | · | Target domain to fetch known URLs for. |
| targets | FILE | · | File listing multiple target domains to process. |
| subs | BOOLEAN | --subs | Include subdomains of the target domain in results. |
| urls-providers | STRING | --providers | Providers to query (wayback, commoncrawl, otx, urlscan). |
| json | BOOLEAN | --json | Output results as JSON records instead of plain URLs. |
| from | STRING | --from | Fetch URLs from this date onward (format YYYYMM). |
| blacklist | STRING | --blacklist | File extensions to skip, for example png, jpg, css. |
| filter-parameters | BOOLEAN | --fp | Remove different parameters of the same endpoint. |
Showing key inputs. gau exposes 18 inputs in total.
Full flag reference (18 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| target | STRING | · | target domain |
| targets | FILE | · | list of targets |
| subs | BOOLEAN | --subs | include subdomains of target domain |
| urls-providers | STRING | --providers | list of providers to use (wayback,commoncrawl,otx,urlscan) |
| from | STRING | --from | fetch urls from date (format: YYYYMM) |
| to | STRING | --to | fetch urls to date (format: YYYYMM) |
| json | BOOLEAN | --json | output as json |
| blacklist | STRING | --blacklist | list of extensions to skip |
| mime-types | STRING | --mt | list of mime-types to match |
| filter-mime-types | STRING | --ft | list of mime-types to filter |
| status-codes | STRING | --mc | list of status codes to match |
| filter-status-codes | STRING | --fc | list of status codes to filter |
| filter-parameters | BOOLEAN | --fp | remove different parameters of the same endpoint |
| proxy | STRING | --p | http proxy to use |
| threads | STRING | --threads | number of workers to spawn (default 1) |
| retries | STRING | --retries | retries for the HTTP client |
| timeout | STRING | --timeout | timeout (in seconds) for HTTP client (default 45) |
| verbose | BOOLEAN | --verbose | show verbose output |
example
Run gau
# archived URLs for a domain and its subdomains, static assets droppedgau example.com --subs --providers wayback,commoncrawl --blacklist png,jpg,gif,css --fphttps://example.com/https://example.com/index.php?id=42https://example.com/loginhttps://api.example.com/v1/users?page=2https://example.com/wp-content/uploads/2019/report.pdfhttps://example.com/old/checkout.aspx?cart=3https://dev.example.com/debug?verbose=truehttps://example.com/search?q=test&lang=enhttps://example.com/.git/configguidance
Choosing gau
Use gau when you want a wide URL list without sending traffic to the target, or when archive history matters. It is passive only, so it never confirms a URL is still live. For an active crawl that follows links and parses JavaScript, use katana. For probing which archived URLs respond, pass gau output to httpx.
gauplus
Maintained gau fork with the same archive providers and a few extra options. Same passive URL discovery job.
katana
Active crawler that follows links and parses JavaScript. Covers the current site, not archive history; complements gau.
httpx
HTTP prober, not a discovery tool. Run it after gau to find which archived URLs still respond.
workflows
Workflows using gau
Find Open Redirects on a Domain
Mine a domain's archived URLs for redirect parameters, then confirm which ones send a browser to an attacker-controlled host.
Find Reflected XSS on a Domain
Collect a domain's archived URLs, keep parameters that reflect input, and fuzz each one for reflected and DOM cross-site scripting.
Find Secrets in Wayback Responses
Pull a domain's archived URLs from web archives, fetch the stored response bodies, and scan them for leaked secrets without touching the live site.
faq
gau questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run gau yourself
A domain feeds gau, which pulls known URLs from public archives and passes them to httpx for live probing before they land as a queryable output.
Facts on this page come from the live Trickest tool library.