loading
Loading content
loading
Discovery
Passive URL collection from public discovery sources.
overview
urlfinder builds a URL map for a domain from public sources. Pass -list with a target or domain file; optionally enable -all sources, then trim with -match and -filter before active stages see the set.
Use it early when you want archive and OSINT coverage without crawling the live site. Scope with -field-scope, -url-scope, and -sources; emit -jsonl when the next node expects structured lines.
Trickest provides urlfinder as a managed Discovery node. Take FILE and FOLDER into urldedupe, then httpx or a fuzzer. Prefer katana when you need an active crawl instead of passive collection.
use cases
Gather every URL the archives and OSINT sources know for a domain, building a wide endpoint map with no requests to the target.
Set field-scope to domain, root domain, or FQDN so the URL set stays inside the assets you actually care about.
Use match and filter regexes to keep only URLs of interest, dropping the long tail of noise before it reaches the next stage.
Emit JSONL and pass the URLs to urldedupe and then a prober or fuzzer, so active testing runs against a clean, deduplicated set.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| all | BOOLEAN | -all | use all sources for enumeration (slow) |
| list | FILE | -list | target domain / list to find urls for |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| match | STRING | -match | url to match (comma separated) |
| proxy | STRING | -proxy | http proxy to use with urlfinder |
| stats | BOOLEAN | -stats | report source statistics |
| config | FILE | -config | flag config file |
Showing key inputs. urlfinder exposes 29 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| all | BOOLEAN | -all | use all sources for enumeration (slow) |
| list | FILE | -list | target domain / list to find urls for |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| match | STRING | -match | url to match (comma separated) |
| proxy | STRING | -proxy | http proxy to use with urlfinder |
| stats | BOOLEAN | -stats | report source statistics |
| config | FILE | -config | flag config file |
| filter | STRING | -filter | url to filter (comma separated) |
| silent | BOOLEAN | -silent | show only urls in output |
| sources | STRING | -sources | specific sources for discovery (e.g., -s alienvault,commoncrawl) |
| timeout | STRING | -timeout | seconds to wait before timing out (default 30) |
| verbose | BOOLEAN | -v | show verbose output |
| max-time | STRING | -max-time | minutes to wait for enumeration results (default 10) |
| no-color | BOOLEAN | -no-color | disable color in output |
| no-scope | BOOLEAN | -no-scope | disables host based default scope |
| url-scope | STRING | -url-scope | in scope url regex to be followed by urlfinder |
| match-file | FILE | -match | list of urls to match |
| rate-limit | STRING | -rate-limit | maximum number of http requests to send per second (global) |
| field-scope | STRING | -field-scope | pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn") |
| filter-file | FILE | -filter | list of urls to filter |
| list-sources | BOOLEAN | -list-sources | list all available sources |
| url-out-scope | STRING | -url-out-scope | out of scope url regex to be excluded by urlfinder |
| url-scope-file | FILE | -url-scope | in scope url regex to be followed by urlfinder |
| collect-sources | BOOLEAN | -collect-sources | include all sources in the output (-json only) |
| exclude-sources | STRING | -exclude-sources | sources to exclude (e.g., -es alienvault,commoncrawl) |
| provider-config | FILE | -provider-config | provider config file |
| display-out-scope | BOOLEAN | -display-out-scope | display external endpoint from scoped crawling |
| rate-limit-source | STRING | -rate-limits | maximum number of http requests to send per second for providers in key=value format (-rls hackertarget=10/m) (default ["waybackarchive=15/m"]) |
| url-out-scope-file | FILE | -url-out-scope | out of scope url regex to be excluded by urlfinder |
example
# urlfinder: passively collect URLs for example.comurlfinder -list domains.txt -all -jsonl -o urls.jsonlhttps://www.example.com/https://www.example.com/loginhttps://api.example.com/v1/healthhttps://dev.example.com/debughttps://static.example.com/app.jshttps://www.example.com/adminguidance
Use urlfinder to build a passive URL map of a domain from archives and OSINT before any active scanning. It sends no traffic to the target. Run it early, then deduplicate with urldedupe and probe or fuzz the survivors. For active in-browser crawling instead of passive collection, use katana.
Pulls archive URLs from Wayback and Common Crawl. urlfinder spans more sources with built-in scoping.
Deep Wayback harvesting. urlfinder covers a broader OSINT source set with built-in scoping.
Active crawler that renders the app. urlfinder is passive and touches nothing.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A domain feeds urlfinder, which gathers known URLs from passive sources and passes them to urldedupe for collapsing before they land as a queryable output.
Facts on this page come from the live Trickest tool library.