loading
Loading content
loading
Discovery
A high-speed tool for passively gathering URLs for web asset discovery.
overview
urlfinder passively gathers URLs from public sources so you expand content discovery without crawling the live site first.
Match and filter flags trim the set to the paths or hosts you care about before you hand them to httpx or a fuzzer.
Use it beside wayback and gau-style collectors when you want ProjectDiscovery's take on passive URL intel.
use cases
Gather every URL the archives and OSINT sources know for a domain, building a wide endpoint map with no requests to the target.
Set field-scope to domain, root domain, or FQDN so the URL set stays inside the assets you actually care about.
Use match and filter regexes to keep only URLs of interest, dropping the long tail of noise before it reaches the next stage.
Emit JSONL and pass the URLs to urldedupe and then a prober or fuzzer, so active testing runs against a clean, deduplicated set.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| all | BOOLEAN | -all | use all sources for enumeration (slow) |
| list | FILE | -list | target domain / list to find urls for |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| match | STRING | -match | url to match (comma separated) |
| proxy | STRING | -proxy | http proxy to use with urlfinder |
| stats | BOOLEAN | -stats | report source statistics |
| config | FILE | -config | flag config file |
Showing key inputs. urlfinder exposes 29 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| all | BOOLEAN | -all | use all sources for enumeration (slow) |
| list | FILE | -list | target domain / list to find urls for |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| match | STRING | -match | url to match (comma separated) |
| proxy | STRING | -proxy | http proxy to use with urlfinder |
| stats | BOOLEAN | -stats | report source statistics |
| config | FILE | -config | flag config file |
| filter | STRING | -filter | url to filter (comma separated) |
| silent | BOOLEAN | -silent | show only urls in output |
| sources | STRING | -sources | specific sources for discovery (e.g., -s alienvault,commoncrawl) |
| timeout | STRING | -timeout | seconds to wait before timing out (default 30) |
| verbose | BOOLEAN | -v | show verbose output |
| max-time | STRING | -max-time | minutes to wait for enumeration results (default 10) |
| no-color | BOOLEAN | -no-color | disable color in output |
| no-scope | BOOLEAN | -no-scope | disables host based default scope |
| url-scope | STRING | -url-scope | in scope url regex to be followed by urlfinder |
| match-file | FILE | -match | list of urls to match |
| rate-limit | STRING | -rate-limit | maximum number of http requests to send per second (global) |
| field-scope | STRING | -field-scope | pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn") |
| filter-file | FILE | -filter | list of urls to filter |
| list-sources | BOOLEAN | -list-sources | list all available sources |
| url-out-scope | STRING | -url-out-scope | out of scope url regex to be excluded by urlfinder |
| url-scope-file | FILE | -url-scope | in scope url regex to be followed by urlfinder |
| collect-sources | BOOLEAN | -collect-sources | include all sources in the output (-json only) |
| exclude-sources | STRING | -exclude-sources | sources to exclude (e.g., -es alienvault,commoncrawl) |
| provider-config | FILE | -provider-config | provider config file |
| display-out-scope | BOOLEAN | -display-out-scope | display external endpoint from scoped crawling |
| rate-limit-source | STRING | -rate-limits | maximum number of http requests to send per second for providers in key=value format (-rls hackertarget=10/m) (default ["waybackarchive=15/m"]) |
| url-out-scope-file | FILE | -url-out-scope | out of scope url regex to be excluded by urlfinder |
example
# urlfinder: passively collect URLs for example.comurlfinder -list domains.txt -all -jsonl -o urls.jsonlhttps://www.example.com/https://www.example.com/loginhttps://api.example.com/v1/healthhttps://dev.example.com/debughttps://static.example.com/app.jshttps://www.example.com/adminguidance
Use urlfinder to build a passive URL map of a domain from archives and OSINT before any active scanning. It sends no traffic to the target. Run it early, then deduplicate with urldedupe and probe or fuzz the survivors. For active in-browser crawling instead of passive collection, use katana.
Pulls archive URLs from Wayback and Common Crawl. urlfinder spans more sources with built-in scoping.
Deep Wayback harvesting. urlfinder is faster and broader across OSINT sources.
Active crawler that renders the app. urlfinder is passive and touches nothing.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A domain feeds urlfinder, which gathers known URLs from passive sources and passes them to urldedupe for collapsing before they land as a queryable output.
Facts on this page come from the live Trickest tool library.