loading
Loading content
loading
Discovery
Identify known URLs of given domains by tapping into a multitude of curated online passive sources.
overview
xurlfind3r collects URLs that already exist in public archives and curated feeds for a domain, without sending any traffic to the target. It queries sources such as the Wayback Machine and other passive providers, then returns every historical and indexed URL it can find. That gives you a fast, quiet picture of an application's known endpoints before you touch it.
Because the data is passive, you control the scope with regex. The match and filter options keep or drop URLs by pattern, include-subdomains widens the search beyond the apex name, and the Wayback parsers can read robots.txt snapshots and archived source code for paths that no longer appear in a live crawl.
On Trickest, xurlfind3r is a Discovery node that reads a domain or a list of domains and writes a file and a folder of URLs. Run it at the start of a workflow to seed content discovery, or feed its output to httpx so probing runs only against URLs that already have a history.
use cases
Pull a domain's historical and indexed URLs from passive sources to build a starting set of endpoints before any active crawling or probing begins.
Enable the Wayback robots.txt and source-code parsers to surface paths and endpoints that have been removed from the live site but remain in archives.
Apply match and filter patterns to keep only the URLs you care about, so the output feeds the next stage clean instead of full of noise.
Hand the discovered URLs to httpx to find which still respond, turning an archive dump into a list of live targets worth testing.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| domain | STRING | --domain | Target domain to find known URLs for. |
| list | FILE | --list | File of target domains, for bulk URL discovery. |
| match | STRING | --match | Regex to keep only URLs that match. |
| filter | STRING | --filter | Regex to drop URLs that match. |
| include-subdomains | BOOLEAN | --include-subdomains | Also match URLs on subdomains of the target. |
| use-sources | STRING | --use-sources | Comma-separated passive sources to use. |
| parse-wayback-source | BOOLEAN | --parse-wayback-source | With Wayback, parse archived source code snapshots. |
| silent | BOOLEAN | --silent | Display output URLs only, no status lines. |
Showing key inputs. xurlfind3r exposes 13 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| list | FILE | --list | target domains' list file path |
| match | STRING | --match | regex to match URLs |
| domain | STRING | --domain | target domain |
| filter | STRING | --filter | regex to filter URLs |
| silent | BOOLEAN | --silent | display output subdomains only |
| verbose | BOOLEAN | --verbose | display verbose output |
| no-color | BOOLEAN | --no-color | disable colored output |
| use-sources | STRING | --use-sources | comma(,) separated sources to use |
| configuration | FILE | --configuration | configuration file |
| exclude-sources | STRING | --exclude-sources | comma(,) separated sources to exclude |
| include-subdomains | BOOLEAN | --include-subdomains | match subdomain's URLs |
| parse-wayback-robots | BOOLEAN | --parse-wayback-robots | with wayback, parse robots.txt snapshots |
| parse-wayback-source | BOOLEAN | --parse-wayback-source | with wayback, parse source code snapshots |
example
# known URLs for a domain and its subdomains, JS files onlyxurlfind3r --domain example.com --include-subdomains --match '\.js(\?|$)' --silent > urls.txthttps://example.com/https://example.com/loginhttps://www.example.com/static/app.4c1f9e.jshttps://api.example.com/v1/users?id=1042https://example.com/robots.txthttps://blog.example.com/2019/06/release-noteshttps://example.com/wp-json/wp/v2/pageshttps://staging.example.com/admin/config.php… (612 URLs collected for the real run behind this page)guidance
Use xurlfind3r when you want a passive list of a domain's known URLs without touching the target. It collects names from archives, it does not test them, so follow it with httpx to find which are live. For active crawling of a live app, use katana instead.
Pulls URLs from Wayback, Common Crawl, and other archives. A close passive alternative.
Deeper Wayback harvesting with more sources. Slower, broader coverage.
Active crawler that finds live endpoints. Run it when you want fresh paths, not archived ones.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A domain feeds xurlfind3r, which collects known URLs from passive sources and passes them to httpx so only live URLs land as output.
Facts on this page come from the live Trickest tool library.