Discovery
Gather URLs and JS paths from live hosts
Go crawler for URLs, forms, and JavaScript locations.
overview
What hakrawler does
hakrawler crawls a list of seed URLs and emits links, forms, and JavaScript file locations. Bound how far it walks with -d. -subs widens scope to related hostnames; -u keeps only unique URLs.
Enable -s when you need the source of each URL (href, form, or script). -json makes the stream easier to pipe. Without live seeds, the crawl has little to walk.
Trickest runs hakrawler as a managed Discovery node that takes a URLs file and writes file/folder output. Prefer katana when you need headless JS rendering; use gau for passive archive URLs only.
source github.com/hakluke/hakrawler
use cases
Where hakrawler fits
Gather URLs from a host list
Crawl a set of live hosts to depth and collect the links and forms each one exposes for the next stage to probe.
Find JavaScript file locations
Let hakrawler surface the script files a site loads so a downstream linkfinder can mine them for endpoints.
Trace where each URL came from
Show the source of every URL, whether href, form, or script, so you know how a path was discovered before testing it.
Crawl across subdomains
Include subdomains so a crawl seeded from one host reaches related names within the same organization.
reference
hakrawler inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of urls |
| depth-to-crawl | STRING | -d | Depth to crawl. (default 2) |
| threads | STRING | -t | Number of threads to utilise. (default 8) |
| json-output | BOOLEAN | -json | Output as json |
| include-subdomains | BOOLEAN | -subs | Include subdomains for crawling. |
| show-source-url | BOOLEAN | -s | Show the source of URL based on where it was found (href, form, script, etc.) |
| only-unique-urls | BOOLEAN | -u | Show only unique urls. |
| proxy | STRING | -proxy | Proxy URL. E.g. -proxy http://127.0.0.1:8080 |
Showing key inputs. hakrawler exposes 10 inputs in total.
Full flag reference (10 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of urls |
| proxy | STRING | -proxy | Proxy URL. E.g. -proxy http://127.0.0.1:8080 |
| threads | STRING | -t | Number of threads to utilise. (default 8) |
| insecure | BOOLEAN | -insecure | Disable TLS verification. |
| json-output | BOOLEAN | -json | Output as json |
| custom-header | STRING | -h | Custom headers separated by two semi-colons. E.g. -h "Cookie: foo=bar;;Referer: http://example.com/" |
| depth-to-crawl | STRING | -d | Depth to crawl. (default 2) |
| show-source-url | BOOLEAN | -s | Show the source of URL based on where it was found (href, form, script, etc.) |
| only-unique-urls | BOOLEAN | -u | Show only unique urls. |
| include-subdomains | BOOLEAN | -subs | Include subdomains for crawling. |
example
Run hakrawler
# crawl seeds to depth 2, unique URLs only, include subdomainscat urls.txt | hakrawler -d 2 -u -subs -t 8https://example.com/https://example.com/loginhttps://example.com/api/v1/statushttps://example.com/static/app.jshttps://www.example.com/docshttps://api.example.com/healthhttps://example.com/products?id=1https://staging.example.com/https://cdn.example.com/bundle.jsguidance
Choosing hakrawler
Use for a lightweight crawl that gathers URLs and JavaScript locations from a host list. For deeper crawling with headless JS rendering and richer scope controls, use katana. For passive-only archive URLs, use gau.
katana
Deeper crawler with headless JS rendering and more scope controls. hakrawler stays lighter.
gospider
Go spider with archive sources and more filtering. Close sibling with a larger option set.
gau
Passive URLs from web archives with no live requests. Complements an active hakrawler crawl.
faq
hakrawler questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run hakrawler yourself
A host list feeds hakrawler, which crawls for URLs and JavaScript locations and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.