loading
Loading content
loading
Discovery
Go crawler for URLs, forms, and JavaScript locations.
overview
hakrawler crawls a list of seed URLs and emits links, forms, and JavaScript file locations. Bound how far it walks with -d. -subs widens scope to related hostnames; -u keeps only unique URLs.
Enable -s when you need the source of each URL (href, form, or script). -json makes the stream easier to pipe. Without live seeds, the crawl has little to walk.
Trickest runs hakrawler as a managed Discovery node that takes a URLs file and writes file/folder output. Prefer katana when you need headless JS rendering; use gau for passive archive URLs only.
source github.com/hakluke/hakrawler
use cases
Crawl a set of live hosts to depth and collect the links and forms each one exposes for the next stage to probe.
Let hakrawler surface the script files a site loads so a downstream linkfinder can mine them for endpoints.
Show the source of every URL, whether href, form, or script, so you know how a path was discovered before testing it.
Include subdomains so a crawl seeded from one host reaches related names within the same organization.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of urls |
| depth-to-crawl | STRING | -d | Depth to crawl. (default 2) |
| threads | STRING | -t | Number of threads to utilise. (default 8) |
| json-output | BOOLEAN | -json | Output as json |
| include-subdomains | BOOLEAN | -subs | Include subdomains for crawling. |
| show-source-url | BOOLEAN | -s | Show the source of URL based on where it was found (href, form, script, etc.) |
| only-unique-urls | BOOLEAN | -u | Show only unique urls. |
| proxy | STRING | -proxy | Proxy URL. E.g. -proxy http://127.0.0.1:8080 |
Showing key inputs. hakrawler exposes 10 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of urls |
| proxy | STRING | -proxy | Proxy URL. E.g. -proxy http://127.0.0.1:8080 |
| threads | STRING | -t | Number of threads to utilise. (default 8) |
| insecure | BOOLEAN | -insecure | Disable TLS verification. |
| json-output | BOOLEAN | -json | Output as json |
| custom-header | STRING | -h | Custom headers separated by two semi-colons. E.g. -h "Cookie: foo=bar;;Referer: http://example.com/" |
| depth-to-crawl | STRING | -d | Depth to crawl. (default 2) |
| show-source-url | BOOLEAN | -s | Show the source of URL based on where it was found (href, form, script, etc.) |
| only-unique-urls | BOOLEAN | -u | Show only unique urls. |
| include-subdomains | BOOLEAN | -subs | Include subdomains for crawling. |
example
# crawl seeds to depth 2, unique URLs only, include subdomainscat urls.txt | hakrawler -d 2 -u -subs -t 8https://example.com/https://example.com/loginhttps://example.com/api/v1/statushttps://example.com/static/app.jshttps://www.example.com/docshttps://api.example.com/healthhttps://example.com/products?id=1https://staging.example.com/https://cdn.example.com/bundle.jsguidance
Use for a lightweight crawl that gathers URLs and JavaScript locations from a host list. For deeper crawling with headless JS rendering and richer scope controls, use katana. For passive-only archive URLs, use gau.
Deeper crawler with headless JS rendering and more scope controls. hakrawler stays lighter.
Go spider with archive sources and more filtering. Close sibling with a larger option set.
Passive URLs from web archives with no live requests. Complements an active hakrawler crawl.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A host list feeds hakrawler, which crawls for URLs and JavaScript locations and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.