waymore

-modeThe mode to run: U (retrieve URLs only), R (download Responses only) or B (Both). If -i is a domain only, then -mode will default to B. If -i is a domain with path then -mode will default to R.

--inputThe list of domains to find links for. This can be a domain only, or a domain with a specific path. If it is a domain only to get everything for that domain, don't prefix with www. You can also specify a TLD only by prefixing with a period, e.g. .mil, which will get all subs for all domains with that TLD (NOTE: The Alien Vault OTX source is excluded if searching for a TLD because it requires a full domain).

--limitHow many responses will be saved (if -b is not passed). A positive value will get the first N results, a negative value will will get the last N results. A value of 0 will get ALL responses (default: 5000))

--configPath to the YML config file

--no-subsDon't include subdomains of the target domain (only used if input is not a domain with a specific path).

--retriesThe number of retries for requests that get connection error or rate limited (default: 1)

--timeoutFor archived responses only, how many seconds to wait for the server to send data before giving up (default: 30)

--to-dateWhat date to get responses to. If not specified it will get to the latest possible results. A partial value can be passed, e.g. 2021, 202112, etc.

--verboseVerbose output

--from-dateWhat date to get responses from. If not specified it will get from the earliest possible results. A partial value can be passed, e.g. 2016, 201805, etc.

--processesThe number of processes (threads) used (default: 1)

--check-onlyThis will make a few minimal requests to show you how many requests, and roughly how long it could take, to get URLs from the sources and downloaded responses from Wayback Machine.

--regex-afterRegEx for filtering purposes against links found from archive.org/commoncrawl.org AND responses downloaded. Only positive matches will be output.

--inputThe target domain to find links for. This can be a domain only, or a domain with a specific path. If it is a domain only to get everything for that domain, don't prefix with www. You can also specify a TLD only by prefixing with a period, e.g. .mil, which will get all subs for all domains with that TLD (NOTE: The Alien Vault OTX source is excluded if searching for a TLD because it requires a full domain).

-url-filenameSet the file name of downloaded responses to the URL that generated the response, otherwise it will be set to the hash value of the response. Using the hash value means multiple URLs that generated the same response will only result in one file being saved for that response.

--keywords-onlyOnly return links and responses that contain keywords that you are interested in. This can reduce the time it takes to get results. Keywords are given in the config.yml file with the FILTER_KEYWORDS key

--limit-requestsLimit the number of requests that will be made when getting links from a source (this doesn't apply to Common Crawl). Some targets can return a huge amount of requests needed that are just not feasible to get, so this can be used to manage that situation. This defaults to 0 (Zero) which means there is no limit.

--notify-discordWhether to send a notification to Discord when waymore completes. It requires WEBHOOK_DISCORD to be provided in the config.yml file.

--capture-intervalFilters the search on archive.org to only get at most 1 capture per hour (h), day (d) or month (m). This filter is used for responses only. The default is 'd' but can also be set to 'none' to not filter anything and get all responses.

-xusExclude checks for links from urlscan.io

--memory-thresholdThe memory threshold percentage. If the machines memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95)

--output-inline-jsWhether to save combined inline javascript of all relevant files in the response directory when -mode R (or -mode B) has been used. The files are saved with the name combinedInline{}.js where {} is the number of the file, saving 1000 unique scripts per file. The file combinedInlineSrc.txt will also be created, containing the src value of all external scripts referenced in the files.

-mcOnly Match HTTP status codes for retrieved URLs and responses. Comma separated list of codes. Passing this argument overrides the config FILTER_CODE and -fc

-fcFilter HTTP status codes for retrieved URLs and responses. Comma separated list of codes (default: the FILTER_CODE values from config.yml). Passing this argument will override the value from config.yml

-lccLimit the number of Common Crawl index collections searched, e.g. -lcc 10 will just search the latest 10 collections (default: 3). As of July 2023 there are currently 95 collections. Setting to 0 (default) will search ALL collections. If you don't want to search Common Crawl at all, use the -xcc option.

--keywords-onlyOnly return links and responses that contain keywords that you are interested in. This can reduce the time it takes to get results. you can pass a specific Regex value to use, e.g. -ko admin to only get links containing the word admin, or -ko .js(?|$) to only get JS files. The Regex check is NOT case sensitive.

-xavExclude checks for links from alienvault.com

-xccExclude checks for links from commoncrawl.org

--filter-responses-onlyThe initial links from Wayback Machine will not be filtered, only the responses that are downloaded, , e.g. it maybe useful to still see all available paths from the links even if you don't want to check the content.

-lcyLimit the number of Common Crawl index collections searched by the year of the index data. The earliest index has data from 2008. Setting to 0 (default) will search collections or any year (but in conjuction with -lcc). For example, if you are only interested in data from 2015 and after, pass -lcy 2015. This will override the value of -lcc if passed. If you don't want to search Common Crawl at all, use the -xcc option.

-xwmExclude checks for links from Wayback Machine (archive.org)

--urlscan-rate-limit-retryThe number of minutes the user wants to wait for a rate limit pause on URLScan.io instead of stopping with a 429 error (default: 1)

--wayback-rate-limit-retryThe number of minutes the user wants to wait for a rate limit pause on Watback Machine (archive.org) instead of stopping with a 429 error (default: 3).