Discovery
Collect known URLs without touching the target
Passive URL collection from public discovery sources.
overview
What urlfinder does
urlfinder builds a URL map for a domain from public sources. Pass -list with a target or domain file; optionally enable -all sources, then trim with -match and -filter before active stages see the set.
Use it early when you want archive and OSINT coverage without crawling the live site. Scope with -field-scope, -url-scope, and -sources; emit -jsonl when the next node expects structured lines.
Trickest provides urlfinder as a managed Discovery node. Take FILE and FOLDER into urldedupe, then httpx or a fuzzer. Prefer katana when you need an active crawl instead of passive collection.
use cases
Where urlfinder fits
Seed an endpoint list passively
Gather every URL the archives and OSINT sources know for a domain, building a wide endpoint map with no requests to the target.
Scope results to the right domain boundary
Set field-scope to domain, root domain, or FQDN so the URL set stays inside the assets you actually care about.
Filter and match by regex
Use match and filter regexes to keep only URLs of interest, dropping the long tail of noise before it reaches the next stage.
Feed a probe-and-fuzz pipeline
Emit JSONL and pass the URLs to urldedupe and then a prober or fuzzer, so active testing runs against a clean, deduplicated set.
reference
urlfinder inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| all | BOOLEAN | -all | use all sources for enumeration (slow) |
| list | FILE | -list | target domain / list to find urls for |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| match | STRING | -match | url to match (comma separated) |
| proxy | STRING | -proxy | http proxy to use with urlfinder |
| stats | BOOLEAN | -stats | report source statistics |
| config | FILE | -config | flag config file |
Showing key inputs. urlfinder exposes 29 inputs in total.
Full flag reference (29 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| all | BOOLEAN | -all | use all sources for enumeration (slow) |
| list | FILE | -list | target domain / list to find urls for |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| match | STRING | -match | url to match (comma separated) |
| proxy | STRING | -proxy | http proxy to use with urlfinder |
| stats | BOOLEAN | -stats | report source statistics |
| config | FILE | -config | flag config file |
| filter | STRING | -filter | url to filter (comma separated) |
| silent | BOOLEAN | -silent | show only urls in output |
| sources | STRING | -sources | specific sources for discovery (e.g., -s alienvault,commoncrawl) |
| timeout | STRING | -timeout | seconds to wait before timing out (default 30) |
| verbose | BOOLEAN | -v | show verbose output |
| max-time | STRING | -max-time | minutes to wait for enumeration results (default 10) |
| no-color | BOOLEAN | -no-color | disable color in output |
| no-scope | BOOLEAN | -no-scope | disables host based default scope |
| url-scope | STRING | -url-scope | in scope url regex to be followed by urlfinder |
| match-file | FILE | -match | list of urls to match |
| rate-limit | STRING | -rate-limit | maximum number of http requests to send per second (global) |
| field-scope | STRING | -field-scope | pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn") |
| filter-file | FILE | -filter | list of urls to filter |
| list-sources | BOOLEAN | -list-sources | list all available sources |
| url-out-scope | STRING | -url-out-scope | out of scope url regex to be excluded by urlfinder |
| url-scope-file | FILE | -url-scope | in scope url regex to be followed by urlfinder |
| collect-sources | BOOLEAN | -collect-sources | include all sources in the output (-json only) |
| exclude-sources | STRING | -exclude-sources | sources to exclude (e.g., -es alienvault,commoncrawl) |
| provider-config | FILE | -provider-config | provider config file |
| display-out-scope | BOOLEAN | -display-out-scope | display external endpoint from scoped crawling |
| rate-limit-source | STRING | -rate-limits | maximum number of http requests to send per second for providers in key=value format (-rls hackertarget=10/m) (default ["waybackarchive=15/m"]) |
| url-out-scope-file | FILE | -url-out-scope | out of scope url regex to be excluded by urlfinder |
example
Run urlfinder
# urlfinder: passively collect URLs for example.comurlfinder -list domains.txt -all -jsonl -o urls.jsonlhttps://www.example.com/https://www.example.com/loginhttps://api.example.com/v1/healthhttps://dev.example.com/debughttps://static.example.com/app.jshttps://www.example.com/adminguidance
Choosing urlfinder
Use urlfinder to build a passive URL map of a domain from archives and OSINT before any active scanning. It sends no traffic to the target. Run it early, then deduplicate with urldedupe and probe or fuzz the survivors. For active in-browser crawling instead of passive collection, use katana.
gau
Pulls archive URLs from Wayback and Common Crawl. urlfinder spans more sources with built-in scoping.
waymore
Deep Wayback harvesting. urlfinder covers a broader OSINT source set with built-in scoping.
katana
Active crawler that renders the app. urlfinder is passive and touches nothing.
faq
urlfinder questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run urlfinder yourself
A domain feeds urlfinder, which gathers known URLs from passive sources and passes them to urldedupe for collapsing before they land as a queryable output.
Facts on this page come from the live Trickest tool library.