Discovery
Mine archived robots.txt for forgotten paths
Passive historic robots.txt path enumeration from Wayback.
overview
What waybackrobots does
waybackrobots queries the Wayback Machine for historical robots.txt copies and extracts listed or disallowed paths that may no longer appear on the live site.
Control volume with -limit (default 50, -1 unlimited) and -recent for latest snapshots only. Pass a single target domain or a targets FILE for a multi-host sweep.
Trickest provides waybackrobots as a managed Discovery node. Hand paths to httpx or ffuf to see what still resolves. For broader archive URLs use waymore or gau; for the live robots.txt use katana.
use cases
Where waybackrobots fits
Recover forgotten disallowed paths
Pull every disallowed entry from a target's archived robots.txt history to find admin and staging routes the current file no longer lists.
Seed content discovery passively
Collect historical paths without touching the live target, then hand them to a fuzzer as a curated, target-specific wordlist.
Sweep many targets at once
Pass a file of targets so the enumeration runs across a whole scope in one node.
Bound the archive lookback
Use the snapshot limit to control how many archived robots.txt versions the run pulls and how far back it reaches.
reference
waybackrobots inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| limit | STRING | -limit | limit the number crawled snapshots. Use -1 for unlimited (default 50) |
| recent | BOOLEAN | -recent | use the most recent snapshots without evenly distributing them |
| target | STRING | · | Target domain |
| targets | FILE | · | List of targets |
Showing key inputs. waybackrobots exposes 4 inputs in total.
example
Run waybackrobots
# waybackrobots: historic robots.txt paths for example.comwaybackrobots example.com -limit 50/admin/backup/internal/legacy/staging/debug/old-apiguidance
Choosing waybackrobots
Use waybackrobots early in recon when you want historic robots.txt paths without touching the target. For a wider archive of all URLs, use waymore or gau. For the live robots.txt and sitemap, use katana.
waymore
Pulls a far wider set of historical URLs from many archive sources, not just robots.txt.
gau
Fetches known URLs from Wayback, Common Crawl, and others. Broader than robots.txt history.
katana
Active crawler that reads the live robots.txt and sitemap rather than the archive.
faq
waybackrobots questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run waybackrobots yourself
A domain feeds waybackrobots, which mines historical robots.txt paths and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.