loading
Loading content
loading
Discovery
Passive known-URL discovery from archive feeds.
overview
xurlfind3r queries passive archives and curated feeds for URLs already known for a domain. Pass --domain or --list; it returns historical and indexed endpoints without sending traffic to the target.
Scope with --match and --filter regexes, widen with --include-subdomains, and optionally parse Wayback robots.txt or source snapshots for paths a live crawl will miss. Narrow providers with --use-sources or --exclude-sources.
Trickest provides xurlfind3r as a managed Discovery node. Seed FILE and FOLDER URLs into httpx for liveness. Prefer katana when you need an active crawl of the live app instead of archive history.
use cases
Pull a domain's historical and indexed URLs from passive sources to build a starting set of endpoints before any active crawling or probing begins.
Enable the Wayback robots.txt and source-code parsers to surface paths and endpoints that have been removed from the live site but remain in archives.
Apply match and filter patterns to keep only the URLs you care about, so the output feeds the next stage clean instead of full of noise.
Hand the discovered URLs to httpx to find which still respond, turning an archive dump into a list of live targets worth testing.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| domain | STRING | --domain | Target domain to find known URLs for. |
| list | FILE | --list | File of target domains, for bulk URL discovery. |
| match | STRING | --match | Regex to keep only URLs that match. |
| filter | STRING | --filter | Regex to drop URLs that match. |
| include-subdomains | BOOLEAN | --include-subdomains | Also match URLs on subdomains of the target. |
| use-sources | STRING | --use-sources | Comma-separated passive sources to use. |
| parse-wayback-source | BOOLEAN | --parse-wayback-source | With Wayback, parse archived source code snapshots. |
| silent | BOOLEAN | --silent | Display output URLs only, no status lines. |
Showing key inputs. xurlfind3r exposes 13 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| list | FILE | --list | target domains' list file path |
| match | STRING | --match | regex to match URLs |
| domain | STRING | --domain | target domain |
| filter | STRING | --filter | regex to filter URLs |
| silent | BOOLEAN | --silent | display output subdomains only |
| verbose | BOOLEAN | --verbose | display verbose output |
| no-color | BOOLEAN | --no-color | disable colored output |
| use-sources | STRING | --use-sources | comma(,) separated sources to use |
| configuration | FILE | --configuration | configuration file |
| exclude-sources | STRING | --exclude-sources | comma(,) separated sources to exclude |
| include-subdomains | BOOLEAN | --include-subdomains | match subdomain's URLs |
| parse-wayback-robots | BOOLEAN | --parse-wayback-robots | with wayback, parse robots.txt snapshots |
| parse-wayback-source | BOOLEAN | --parse-wayback-source | with wayback, parse source code snapshots |
example
# known URLs for a domain and its subdomains, JS files onlyxurlfind3r --domain example.com --include-subdomains --match '\.js(\?|$)' --silent > urls.txthttps://example.com/https://example.com/loginhttps://www.example.com/static/app.4c1f9e.jshttps://api.example.com/v1/users?id=1042https://example.com/robots.txthttps://blog.example.com/2019/06/release-noteshttps://example.com/wp-json/wp/v2/pageshttps://staging.example.com/admin/config.php… (612 URLs collected for the real run behind this page)guidance
Use xurlfind3r when you want a quiet list of known URLs before any active probing. It collects from archives; it does not confirm liveness. Follow with httpx. For live crawling and JS parsing, use katana. gau and waymore cover similar passive ground with different source mixes.
Passive URLs from Wayback, Common Crawl, and related feeds. Close peer; pick by source set and flags.
Deeper Wayback-focused harvest. Broader archive coverage, usually slower.
Active crawler for live endpoints. Use when you need current paths, not archived ones.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A domain feeds xurlfind3r, which collects known URLs from passive sources and passes them to httpx so only live URLs land as output.
Facts on this page come from the live Trickest tool library.