loading
Loading content
loading
Discovery
Passive historic robots.txt path enumeration from Wayback.
overview
waybackrobots queries the Wayback Machine for historical robots.txt copies and extracts listed or disallowed paths that may no longer appear on the live site.
Control volume with -limit (default 50, -1 unlimited) and -recent for latest snapshots only. Pass a single target domain or a targets FILE for a multi-host sweep.
Trickest provides waybackrobots as a managed Discovery node. Hand paths to httpx or ffuf to see what still resolves. For broader archive URLs use waymore or gau; for the live robots.txt use katana.
use cases
Pull every disallowed entry from a target's archived robots.txt history to find admin and staging routes the current file no longer lists.
Collect historical paths without touching the live target, then hand them to a fuzzer as a curated, target-specific wordlist.
Pass a file of targets so the enumeration runs across a whole scope in one node.
Use the snapshot limit to control how many archived robots.txt versions the run pulls and how far back it reaches.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| limit | STRING | -limit | limit the number crawled snapshots. Use -1 for unlimited (default 50) |
| recent | BOOLEAN | -recent | use the most recent snapshots without evenly distributing them |
| target | STRING | · | Target domain |
| targets | FILE | · | List of targets |
Showing key inputs. waybackrobots exposes 4 inputs in total.
example
# waybackrobots: historic robots.txt paths for example.comwaybackrobots example.com -limit 50/admin/backup/internal/legacy/staging/debug/old-apiguidance
Use waybackrobots early in recon when you want historic robots.txt paths without touching the target. For a wider archive of all URLs, use waymore or gau. For the live robots.txt and sitemap, use katana.
Pulls a far wider set of historical URLs from many archive sources, not just robots.txt.
Fetches known URLs from Wayback, Common Crawl, and others. Broader than robots.txt history.
Active crawler that reads the live robots.txt and sitemap rather than the archive.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A domain feeds waybackrobots, which mines historical robots.txt paths and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.