Updated Jul 15, 2026
Recon
Harvest historical URLs and archived responses
Multi-archive URL harvest with optional response download.
overview
What waymore does
waymore harvests a domain's historical URLs from Wayback, Common Crawl, URLScan, AlienVault OTX, VirusTotal, and Intelligence X, then merges them into one set larger than any single source.
Choose -mode U (URLs), R (responses), or B (both). Narrow with --providers, --from-date and --to-date, -ft and -fc, plus --limit so large domains stay workable.
Trickest provides waymore as a managed Recon node that writes a FOLDER. Seed early, then probe with httpx. Prefer gau for a lighter URL-only pass; waybackrobots when you only need historic robots.txt paths.
use cases
Where waymore fits
Build a wide historical URL map
Pull a domain's URLs from six archive sources at once to recover endpoints and parameters the live site no longer links.
Mine archived responses for secrets
Download saved responses with -mode R or B, optionally combine inline JavaScript, then grep for API keys, tokens, and hidden routes.
Filter by date, MIME, and status
Constrain a noisy domain with --from-date, --to-date, -ft, and -fc so only relevant URLs survive.
Pick your archive sources
Use --providers to choose exactly which archives to query, trading coverage against speed and rate limits.
reference
waymore inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| input | FILE | --input | The list of domains to find links for (domain only or domain with path). |
| mode | STRING | -mode | U (URLs only), R (responses only), or B (both). |
| providers | STRING | --providers | Comma-separated sources: wayback, commoncrawl, otx, urlscan, virustotal, intelx. |
| from-date | STRING | --from-date | What date to get responses from (e.g. 2016, 201805). |
| to-date | STRING | --to-date | What date to get responses to. |
| filter-mime-type | STRING | -ft | Filter MIME types for retrieved URLs and responses. |
| filter-status-code | STRING | -fc | Filter HTTP status codes for retrieved URLs and responses. |
| limit | STRING | --limit | How many responses will be saved. 0 gets all (default 5000). |
Showing key inputs. waymore exposes 38 inputs in total.
Full flag reference (38 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| mode | STRING | -mode | The mode to run: U (retrieve URLs only), R (download Responses only) or B (Both). If -i is a domain only, then -mode will default to B. If -i is a domain with path then -mode will default to R. |
| input | FILE | --input | The list of domains to find links for. This can be a domain only, or a domain with a specific path. If it is a domain only to get everything for that domain, don't prefix with www. You can also specify a TLD only by prefixing with a period, e.g. .mil, which will get all subs for all domains with that TLD (NOTE: The Alien Vault OTX source is excluded if searching for a TLD because it requires a full domain). |
| limit | STRING | --limit | How many responses will be saved (if -b is not passed). A positive value will get the first N results, a negative value will will get the last N results. A value of 0 will get ALL responses (default: 5000)) |
| config | FILE | --config | Path to the YML config file |
| no-subs | BOOLEAN | --no-subs | Don't include subdomains of the target domain (only used if input is not a domain with a specific path). |
| retries | STRING | --retries | The number of retries for requests that get connection error or rate limited (default: 1) |
| timeout | STRING | --timeout | For archived responses only, how many seconds to wait for the server to send data before giving up (default: 30) |
| to-date | STRING | --to-date | What date to get responses to. If not specified it will get to the latest possible results. A partial value can be passed, e.g. 2021, 202112, etc. |
| verbose | BOOLEAN | --verbose | Verbose output |
| from-date | STRING | --from-date | What date to get responses from. If not specified it will get from the earliest possible results. A partial value can be passed, e.g. 2016, 201805, etc. |
| processes | STRING | --processes | The number of processes (threads) used (default: 1) |
| providers | STRING | --providers | A comma separated list of source providers that you want to get URLs from. The values can be wayback,commoncrawl,otx,urlscan,virustotal and intelx. Passing this will override any exclude arguments (e.g. -xwm,-xcc, etc.) passed to exclude sources, and reset those based on what was passed with this argument. |
| check-only | BOOLEAN | --check-only | This will make a few minimal requests to show you how many requests, and roughly how long it could take, to get URLs from the sources and downloaded responses from Wayback Machine. |
| regex-after | STRING | --regex-after | RegEx for filtering purposes against links found from archive.org/commoncrawl.org AND responses downloaded. Only positive matches will be output. |
| input-domain | STRING | --input | The target domain to find links for. This can be a domain only, or a domain with a specific path. If it is a domain only to get everything for that domain, don't prefix with www. You can also specify a TLD only by prefixing with a period, e.g. .mil, which will get all subs for all domains with that TLD (NOTE: The Alien Vault OTX source is excluded if searching for a TLD because it requires a full domain). |
| url-filename | BOOLEAN | -url-filename | Set the file name of downloaded responses to the URL that generated the response, otherwise it will be set to the hash value of the response. Using the hash value means multiple URLs that generated the same response will only result in one file being saved for that response. |
| keywords-only | BOOLEAN | --keywords-only | Only return links and responses that contain keywords that you are interested in. This can reduce the time it takes to get results. Keywords are given in the "config.yml" file with the "FILTER_KEYWORDS" key |
| exclude-intelx | BOOLEAN | -xix | Exclude checks for links from intelx.io |
| limit-requests | STRING | --limit-requests | Limit the number of requests that will be made when getting links from a source (this doesn't apply to Common Crawl). Some targets can return a huge amount of requests that are not feasible to fetch, so this can be used to manage that situation. This defaults to 0 (Zero) which means there is no limit. |
| notify-discord | BOOLEAN | --notify-discord | Whether to send a notification to Discord when waymore completes. It requires WEBHOOK_DISCORD to be provided in the config.yml file. |
| match-mime-type | STRING | -mt | Only MIME Types for retrieved URLs and responses. Comma separated list of MIME types. Passing this argument overrides the config FILTER_MIME and -ft. NOTE: This will NOT be applied to Alien Vault OTX, Virus Total and Intelligence X because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined - these will always be included. Consider excluding sources if this matters to you.. |
| capture-interval | STRING | --capture-interval | Filters the search on archive.org to only get at most 1 capture per hour (h), day (d) or month (m). This filter is used for responses only. The default is 'd' but can also be set to 'none' to not filter anything and get all responses. |
| exclude-url-scan | BOOLEAN | -xus | Exclude checks for links from urlscan.io |
| filter-mime-type | STRING | -ft | Filter MIME Types for retrieved URLs and responses. Comma separated list of MIME Types (default: the FILTER_MIME values from config.yml). Passing this argument will override the value from config.yml. NOTE: This will NOT be applied to Alien Vault OTX, Virus Total and Intelligence X because they don't have the ability to filter on MIME Type. Sometimes URLScan does not have a MIME Type defined - these will always be included. Consider excluding sources if this matters to you.. |
| memory-threshold | STRING | --memory-threshold | The memory threshold percentage. If the machines memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95) |
| output-inline-js | BOOLEAN | --output-inline-js | Whether to save combined inline javascript of all relevant files in the response directory when -mode R (or -mode B) has been used. The files are saved with the name combinedInline{}.js where {} is the number of the file, saving 1000 unique scripts per file. The file combinedInlineSrc.txt will also be created, containing the src value of all external scripts referenced in the files. |
| match-status-code | STRING | -mc | Only Match HTTP status codes for retrieved URLs and responses. Comma separated list of codes. Passing this argument overrides the config FILTER_CODE and -fc |
| exclude-virustotal | BOOLEAN | -xvt | Exclude checks for links from virustotal.com |
| filter-status-code | STRING | -fc | Filter HTTP status codes for retrieved URLs and responses. Comma separated list of codes (default: the FILTER_CODE values from config.yml). Passing this argument will override the value from config.yml |
| limit-common-crawl | STRING | -lcc | Limit the number of Common Crawl index collections searched, e.g. -lcc 10 will search only the latest 10 collections (default: 3). As of July 2023 there are currently 95 collections. Setting to 0 (default) will search ALL collections. If you don't want to search Common Crawl at all, use the -xcc option. |
| match-keywords-only | STRING | --keywords-only | Only return links and responses that contain keywords that you are interested in. This can reduce the time it takes to get results. you can pass a specific Regex value to use, e.g. -ko "admin" to only get links containing the word admin, or -ko "\.js(\?|$)" to only get JS files. The Regex check is NOT case sensitive. |
| exclude-alient-vault | BOOLEAN | -xav | Exclude checks for links from alienvault.com |
| exclude-common-crawl | BOOLEAN | -xcc | Exclude checks for links from commoncrawl.org |
| filter-responses-only | BOOLEAN | --filter-responses-only | The initial links from Wayback Machine will not be filtered, only the responses that are downloaded, , e.g. it maybe useful to still see all available paths from the links even if you don't want to check the content. |
| limit-common-crawl-year | STRING | -lcy | Limit the number of Common Crawl index collections searched by the year of the index data. The earliest index has data from 2008. Setting to 0 (default) will search collections or any year (but in conjuction with -lcc). For example, if you are only interested in data from 2015 and after, pass -lcy 2015. This will override the value of -lcc if passed. If you don't want to search Common Crawl at all, use the -xcc option. |
| exclude-wayback-matchine | BOOLEAN | -xwm | Exclude checks for links from Wayback Machine (archive.org) |
| urlscan-rate-limit-retry | STRING | --urlscan-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a 429 error (default: 1) |
| wayback-rate-limit-retry | STRING | --wayback-rate-limit-retry | The number of minutes the user wants to wait for a rate limit pause on Watback Machine (archive.org) instead of stopping with a 429 error (default: 3). |
example
Run waymore
# URLs only from selected archives, date-boundedwaymore --input domains.txt -mode U --providers wayback,commoncrawl,urlscan --from-date 2020 --to-date 2024 --limit 2000https://www.example.com/https://www.example.com/loginhttps://api.example.com/v1/users?id=1https://cdn.example.com/static/app.jshttps://www.example.com/admin/config.bakhttps://staging.example.com/api/healthhttps://www.example.com/search?q=testhttps://app.example.com/oauth/callback… (1842 unique URLs after filters)guidance
Choosing waymore
Use waymore when one archive source is not enough and you want historical URLs plus optional archived responses. Apply date and MIME filters on large domains. Prefer gau for a lighter URL-only fetch; waybackrobots for robots.txt history only.
gau
Lighter known-URL fetcher from fewer archives. waymore queries more sources and can download responses.
waybackurls
Wayback-only URL fetcher. waymore adds Common Crawl, URLScan, OTX, VirusTotal, and more.
waybackrobots
Narrow archive miner for historical robots.txt paths only.
faq
waymore questions
Wayback Machine, Common Crawl, URLScan, AlienVault OTX, VirusTotal, and Intelligence X. Pass --providers with a comma-separated subset, or exclude sources with flags such as -xwm and -xcc.
related
More Recon tools
amass
Multi-source subdomain discovery; hand names to httpx.
amass-intel
OWASP Amass intel: map an organization's root domains and ranges.
assetfinder
Find domains and subdomains potentially related to a given domain.
bbot
Modular OSINT recon that chains modules from a seed target.
bevigil
CLI client for the BeVigil OSINT API, keyed by domain or app package.
cloud-enum
Multi-cloud public name enumeration for AWS, Azure, and GCP.
Run waymore yourself
A domain list feeds waymore, which harvests historical URLs from many archives and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.