Discovery
Collect known URLs from passive archives
Passive known-URL discovery from archive feeds.
overview
What xurlfind3r does
xurlfind3r queries passive archives and curated feeds for URLs already known for a domain. Pass --domain or --list; it returns historical and indexed endpoints without sending traffic to the target.
Scope with --match and --filter regexes, widen with --include-subdomains, and optionally parse Wayback robots.txt or source snapshots for paths a live crawl will miss. Narrow providers with --use-sources or --exclude-sources.
Trickest provides xurlfind3r as a managed Discovery node. Seed FILE and FOLDER URLs into httpx for liveness. Prefer katana when you need an active crawl of the live app instead of archive history.
use cases
Where xurlfind3r fits
Seed a workflow with known URLs
Pull a domain's historical and indexed URLs from passive sources to build a starting set of endpoints before any active crawling or probing begins.
Recover paths from archived snapshots
Enable the Wayback robots.txt and source-code parsers to surface paths and endpoints that have been removed from the live site but remain in archives.
Scope results with regex
Apply match and filter patterns to keep only the URLs you care about, so the output feeds the next stage clean instead of full of noise.
Feed a probing pipeline
Hand the discovered URLs to httpx to find which still respond, turning an archive dump into a list of live targets worth testing.
reference
xurlfind3r inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| domain | STRING | --domain | Target domain to find known URLs for. |
| list | FILE | --list | File of target domains, for bulk URL discovery. |
| match | STRING | --match | Regex to keep only URLs that match. |
| filter | STRING | --filter | Regex to drop URLs that match. |
| include-subdomains | BOOLEAN | --include-subdomains | Also match URLs on subdomains of the target. |
| use-sources | STRING | --use-sources | Comma-separated passive sources to use. |
| parse-wayback-source | BOOLEAN | --parse-wayback-source | With Wayback, parse archived source code snapshots. |
| silent | BOOLEAN | --silent | Display output URLs only, no status lines. |
Showing key inputs. xurlfind3r exposes 13 inputs in total.
Full flag reference (13 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| list | FILE | --list | target domains' list file path |
| match | STRING | --match | regex to match URLs |
| domain | STRING | --domain | target domain |
| filter | STRING | --filter | regex to filter URLs |
| silent | BOOLEAN | --silent | display output subdomains only |
| verbose | BOOLEAN | --verbose | display verbose output |
| no-color | BOOLEAN | --no-color | disable colored output |
| use-sources | STRING | --use-sources | comma(,) separated sources to use |
| configuration | FILE | --configuration | configuration file |
| exclude-sources | STRING | --exclude-sources | comma(,) separated sources to exclude |
| include-subdomains | BOOLEAN | --include-subdomains | match subdomain's URLs |
| parse-wayback-robots | BOOLEAN | --parse-wayback-robots | with wayback, parse robots.txt snapshots |
| parse-wayback-source | BOOLEAN | --parse-wayback-source | with wayback, parse source code snapshots |
example
Run xurlfind3r
# known URLs for a domain and its subdomains, JS files onlyxurlfind3r --domain example.com --include-subdomains --match '\.js(\?|$)' --silent > urls.txthttps://example.com/https://example.com/loginhttps://www.example.com/static/app.4c1f9e.jshttps://api.example.com/v1/users?id=1042https://example.com/robots.txthttps://blog.example.com/2019/06/release-noteshttps://example.com/wp-json/wp/v2/pageshttps://staging.example.com/admin/config.php… (612 URLs collected for the real run behind this page)guidance
Choosing xurlfind3r
Use xurlfind3r when you want a quiet list of known URLs before any active probing. It collects from archives; it does not confirm liveness. Follow with httpx. For live crawling and JS parsing, use katana. gau and waymore cover similar passive ground with different source mixes.
gau
Passive URLs from Wayback, Common Crawl, and related feeds. Close peer; pick by source set and flags.
waymore
Deeper Wayback-focused harvest. Broader archive coverage, usually slower.
katana
Active crawler for live endpoints. Use when you need current paths, not archived ones.
faq
xurlfind3r questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run xurlfind3r yourself
A domain feeds xurlfind3r, which collects known URLs from passive sources and passes them to httpx so only live URLs land as output.
Facts on this page come from the live Trickest tool library.