loading
Loading content
loading
Discovery
A modified version of gau that pulls known URLs for a domain from public web archives.
overview
gauplus collects URLs that the web already knows about. Instead of crawling a target, it queries public archives and threat-intel feeds, the Wayback Machine, OTX, and Common Crawl, and returns every path those sources have recorded for a domain. The work is passive, so the target sees no traffic while you build a wide list of historical and current URLs.
It is a maintained fork of gau (Get All URLs), tuned for throughput and provider control. You pick which archives to query, run several workers in parallel, and optionally fold in subdomains so a single run covers a whole footprint rather than one host. Skipping noisy extensions such as images and fonts keeps the output focused on the URLs worth testing.
On Trickest, gauplus is a Discovery node that takes a file of target domains and writes a file and a folder of results. Run it early to seed an attack surface from archive data, then hand the URLs to a prober such as httpx to find which still respond before scanning and fuzzing stages run.
source github.com/bp0lr/gauplus
use cases
Pull every known URL for a domain from Wayback, OTX, and Common Crawl so a workflow starts with historical paths and parameters no live crawl would surface.
Surface old admin panels, retired API routes, and stale parameters that archives still hold but the current site no longer links to.
Enable subdomain inclusion with -subs and run several workers so a single pass gathers URLs across every subdomain of a target domain instead of one host at a time.
Skip image and font extensions with -b to keep the list lean, then pass it to httpx for liveness and on to parameter and content fuzzing stages.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| targets-domains | FILE | · | File of target domains to fetch known URLs for. |
| providers | STRING | -providers | Providers to fetch URLs from (default wayback,otx,commoncrawl). |
| include-subs | BOOLEAN | -subs | Include subdomains of the target domain in results. |
| extension-to-skip | STRING | -b | Extensions to skip, e.g. ttf,woff,svg,png,jpg. |
| threads | STRING | -t | Number of parallel workers fetching from providers (default 5). |
| output-json | BOOLEAN | -json | Write output as JSON. |
Showing key inputs. gauplus exposes 8 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| targets-domains | FILE | · | File of target domains to fetch known URLs for. |
| providers | STRING | -providers | Providers to fetch URLs from (default wayback,otx,commoncrawl). |
| include-subs | BOOLEAN | -subs | Include subdomains of the target domain in results. |
| extension-to-skip | STRING | -b | Extensions to skip, e.g. ttf,woff,svg,png,jpg. |
| threads | STRING | -t | Number of parallel workers fetching from providers (default 5). |
| output-json | BOOLEAN | -json | Write output as JSON. |
| se-random-agent | BOOLEAN | -random-agent | Send a random user-agent on each request to a provider. |
| verbose | BOOLEAN | -v | Enable verbose mode. |
example
# pull known URLs for a domain and its subdomains, skipping noisy extensionsecho example.com | gauplus -subs -t 10 -b ttf,woff,svg,png,jpg -o urls.txthttps://example.com/robots.txthttps://example.com/index.php?id=1http://example.com/wp-login.phphttps://api.example.com/v1/users?id=42https://example.com/assets/app.min.jshttps://blog.example.com/2019/06/release-noteshttps://example.com/admin/login.aspxhttps://example.com/search?q=test&page=2https://dev.example.com/uploads/report.pdfguidance
Reach for gauplus when you want a domain's known URLs without touching the target. It reads archives, so it has no idea what is live now, pair it with httpx to filter responders. For an active crawl that walks the current site and parses JavaScript, use katana instead.
The upstream tool gauplus forks. Same archive sources, gauplus adds provider control and parallel workers.
Active crawler that walks the live site and parses JavaScript. Run alongside gauplus for current plus archived URLs.
Wayback-only URL fetcher. gauplus adds OTX and Common Crawl and more tuning.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A file of domains feeds gauplus, which pulls known URLs from web archives and passes them to httpx, which keeps the ones still responding as a queryable output.
Facts on this page come from the live Trickest tool library.