loading
Loading content
loading
Discovery
Enumerate old versions of robots.txt from the Wayback Machine for content discovery.
overview
waybackrobots queries the Wayback Machine for historical robots.txt copies and extracts disallowed or listed paths that may no longer appear on the live site.
Limit and recent controls keep the lookback window practical when a domain has years of snapshots.
Use it in content discovery when old crawl rules still point at admin, backup, or staging paths worth requesting.
use cases
Pull every disallowed entry from a target's archived robots.txt history to find admin and staging routes the current file no longer lists.
Collect historical paths without touching the live target, then hand them to a fuzzer as a curated, target-specific wordlist.
Pass a file of targets so the enumeration runs across a whole scope in one node.
Use the snapshot limit to control how many archived robots.txt versions the run pulls and how far back it reaches.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| limit | STRING | -limit | limit the number crawled snapshots. Use -1 for unlimited (default 50) |
| recent | BOOLEAN | -recent | use the most recent snapshots without evenly distributing them |
| target | STRING | · | Target domain |
| targets | FILE | · | List of targets |
Showing key inputs. waybackrobots exposes 4 inputs in total.
example
# waybackrobots: historic robots.txt paths for example.comwaybackrobots example.com -limit 50/admin/backup/internal/legacy/staging/debug/old-apiguidance
Use waybackrobots early in recon to mine historical robots.txt for hidden paths, passively and without touching the target. For a broader archive of all URLs a domain ever exposed, use waymore or gau alongside it.
Pulls a far wider set of historical URLs from many archive sources, not just robots.txt.
Fetches known URLs from Wayback, Common Crawl, and others. Broader than robots.txt history.
Active crawler that reads the live robots.txt and sitemap rather than the archive.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A domain feeds waybackrobots, which mines historical robots.txt paths and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.