Which providers does gau query, and can I limit them?

By default gau pulls from wayback, commoncrawl, otx, and urlscan. Pass --providers with a comma-separated subset, for example wayback,commoncrawl, when you want a narrower archive set.

How does gau compare to gauplus?

gauplus is a community fork with the same archive providers and a few extra options. Both harvest passive URLs the same way. The library ships both as separate Discovery nodes when you want to compare output shapes.

Can I include subdomains in the results?

Yes. Enable --subs to fetch URLs for subdomains of the target, not the apex domain alone. Combine it with --blacklist and --fp to keep the output focused.

What should run after gau in a workflow?

Probe the URLs with httpx to see which still respond, then send parameterized endpoints to a fuzzer or scanner. Use --fp first to collapse near-duplicate parameter variants.

Updated Jul 14, 2026

Discovery

Pull archived URLs without touching the target

Passive known-URL fetch from public web archives.

Seed recon passively before touching the target Recover endpoints removed from the live site Build a fuzzing and injection candidate list

overview

What gau does

gau (getallurls) builds a URL list for a domain without contacting the host. It queries Wayback, Common Crawl, AlienVault OTX, and urlscan for paths, parameters, and endpoints already recorded in public archives.

Use it early when the scope is sensitive or when you need history an active crawler will miss. Narrow with --providers, date windows via --from and --to, --blacklist for static assets, and --fp to collapse parameter variants.

Pass a domain or domains file, then take FILE and FOLDER URLs into httpx for liveness and a fuzzer. Pair with katana when you also need a live crawl of the current site.

source github.com/lc/gau

use cases

Where gau fits

Seed recon passively before touching the target

Pull every archived URL for a domain from Wayback, Common Crawl, OTX, and urlscan without sending a single request to the host, so reconnaissance starts wide and silent.

Recover endpoints removed from the live site

Surface old paths, deprecated APIs, and forgotten parameters that no longer appear on the running application but still live in public archives.

Build a fuzzing and injection candidate list

Collect parameterized URLs across subdomains, then collapse duplicates with parameter filtering so a fuzzer or injection scanner works from a clean, deduplicated set.

Combine passive and active discovery

Run gau alongside an active crawler such as Katana, then probe the merged URLs with httpx so the workflow covers both archive history and the current attack surface.

reference

gau inputs and flags

18 inputs

Name	Type	Flag	Description
target	STRING	·	Target domain to fetch known URLs for.
targets	FILE	·	File listing multiple target domains to process.
subs	BOOLEAN	--subs	Include subdomains of the target domain in results.
urls-providers	STRING	--providers	Providers to query (wayback, commoncrawl, otx, urlscan).
json	BOOLEAN	--json	Output results as JSON records instead of plain URLs.
from	STRING	--from	Fetch URLs from this date onward (format YYYYMM).
blacklist	STRING	--blacklist	File extensions to skip, for example png, jpg, css.
filter-parameters	BOOLEAN	--fp	Remove different parameters of the same endpoint.

Showing key inputs. gau exposes 18 inputs in total.

Full flag reference (18 inputs)

Name	Type	Flag	Description
target	STRING	·	target domain
targets	FILE	·	list of targets
subs	BOOLEAN	--subs	include subdomains of target domain
urls-providers	STRING	--providers	list of providers to use (wayback,commoncrawl,otx,urlscan)
from	STRING	--from	fetch urls from date (format: YYYYMM)
to	STRING	--to	fetch urls to date (format: YYYYMM)
json	BOOLEAN	--json	output as json
blacklist	STRING	--blacklist	list of extensions to skip
mime-types	STRING	--mt	list of mime-types to match
filter-mime-types	STRING	--ft	list of mime-types to filter
status-codes	STRING	--mc	list of status codes to match
filter-status-codes	STRING	--fc	list of status codes to filter
filter-parameters	BOOLEAN	--fp	remove different parameters of the same endpoint
proxy	STRING	--p	http proxy to use
threads	STRING	--threads	number of workers to spawn (default 1)
retries	STRING	--retries	retries for the HTTP client
timeout	STRING	--timeout	timeout (in seconds) for HTTP client (default 45)
verbose	BOOLEAN	--verbose	show verbose output

example

Run gau

gau · command

# archived URLs for a domain and its subdomains, static assets droppedgau example.com --subs --providers wayback,commoncrawl --blacklist png,jpg,gif,css --fp

sample output

https://example.com/https://example.com/index.php?id=42https://example.com/loginhttps://api.example.com/v1/users?page=2https://example.com/wp-content/uploads/2019/report.pdfhttps://example.com/old/checkout.aspx?cart=3https://dev.example.com/debug?verbose=truehttps://example.com/search?q=test&lang=enhttps://example.com/.git/config

guidance

Choosing gau

Use gau when you want a wide URL list without sending traffic to the target, or when archive history matters. It is passive only, so it never confirms a URL is still live. For an active crawl that follows links and parses JavaScript, use katana. For probing which archived URLs respond, pass gau output to httpx.

gauplus

Maintained gau fork with the same archive providers and a few extra options. Same passive URL discovery job.

katana

Active crawler that follows links and parses JavaScript. Covers the current site, not archive history; complements gau.

httpx

HTTP prober, not a discovery tool. Run it after gau to find which archived URLs still respond.

faq

gau questions

No. gau is passive. It queries public archives such as the Wayback Machine, Common Crawl, OTX, and urlscan, so the target host never receives a request from the scan.

Run gau yourself

A domain feeds gau, which pulls known URLs from public archives and passes them to httpx for live probing before they land as a queryable output.

Run gau in Trickest View the workflow

Facts on this page come from the live Trickest tool library.

Name

Type

Flag

Description

target

STRING

Target domain to fetch known URLs for.

targets

FILE

File listing multiple target domains to process.

subs

BOOLEAN

--subs

Include subdomains of the target domain in results.

urls-providers

STRING

--providers

Providers to query (wayback, commoncrawl, otx, urlscan).

json

BOOLEAN

--json

Output results as JSON records instead of plain URLs.

from

STRING

--from

Fetch URLs from this date onward (format YYYYMM).

blacklist

STRING

--blacklist

File extensions to skip, for example png, jpg, css.

filter-parameters

BOOLEAN

--fp

Remove different parameters of the same endpoint.

Name

Type

Flag

Description

target

STRING

target domain

targets

FILE

list of targets

subs

BOOLEAN

--subs

include subdomains of target domain

urls-providers

STRING

--providers

list of providers to use (wayback,commoncrawl,otx,urlscan)

from

STRING

--from

fetch urls from date (format: YYYYMM)

STRING

--to

fetch urls to date (format: YYYYMM)

json

BOOLEAN

--json

output as json

blacklist

STRING

--blacklist

list of extensions to skip

mime-types

STRING

--mt

list of mime-types to match

filter-mime-types

STRING

--ft

list of mime-types to filter

status-codes

STRING

--mc

list of status codes to match

filter-status-codes

STRING

--fc

list of status codes to filter

filter-parameters

BOOLEAN

--fp

remove different parameters of the same endpoint

proxy

STRING

--p

http proxy to use

threads

STRING

--threads

number of workers to spawn (default 1)

retries

STRING

--retries

retries for the HTTP client

timeout

STRING

--timeout

timeout (in seconds) for HTTP client (default 45)

verbose

BOOLEAN

--verbose

show verbose output

https://example.com/https://example.com/index.php?id=42https://example.com/loginhttps://api.example.com/v1/users?page=2https://example.com/wp-content/uploads/2019/report.pdfhttps://example.com/old/checkout.aspx?cart=3https://dev.example.com/debug?verbose=truehttps://example.com/search?q=test&lang=enhttps://example.com/.git/config

Pull archived URLs without touching the target

What gau does

Where gau fits

Seed recon passively before touching the target

Recover endpoints removed from the live site

Build a fuzzing and injection candidate list

Combine passive and active discovery

gau inputs and flags

Run gau

Choosing gau

gauplus

katana

httpx

gau questions

More Discovery tools

apkurlgrep

dirsearch

fallparams

feroxbuster

gauplus

getJS

Run gau yourself

Pull archived URLs without touching the target

What gau does

Where gau fits

Seed recon passively before touching the target

Recover endpoints removed from the live site

Build a fuzzing and injection candidate list

Combine passive and active discovery

gau inputs and flags

Run gau

Choosing gau

gauplus

katana

httpx

gau questions

More Discovery tools

apkurlgrep

dirsearch

fallparams

feroxbuster

gauplus

getJS

Run gau yourself