loading
Loading content
loading
Discovery
A fast Go web crawler for gathering URLs and JavaScript file locations.
overview
hakrawler crawls a list of URLs and pulls out the links, forms, and JavaScript file locations it finds, built on the Gocolly library. It is intentionally minimal and fast: feed it hosts, set a depth, and it returns a flat stream of discovered URLs ready for the next stage of a pipeline.
Depth bounds how far it walks from each seed, subdomain inclusion widens scope when you want it, and showing the source of each URL tells you whether a link came from an href, a form, or a script. JSON output and a proxy option make it easy to wire into automation.
In a Trickest workflow it takes a URLs file and writes a file and a folder of results. Run it after subdomain discovery and HTTP probing so it crawls live hosts, then pass the gathered URLs to a prober or a scanner downstream.
source github.com/hakluke/hakrawler
use cases
Crawl a set of live hosts to depth and collect the links and forms each one exposes for the next stage to probe.
Let hakrawler surface the script files a site loads so a downstream linkfinder can mine them for endpoints.
Show the source of every URL, whether href, form, or script, so you know how a path was discovered before testing it.
Include subdomains so a crawl seeded from one host reaches related names within the same organization.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of urls |
| depth-to-crawl | STRING | -d | Depth to crawl. (default 2) |
| threads | STRING | -t | Number of threads to utilise. (default 8) |
| json-output | BOOLEAN | -json | Output as json |
| include-subdomains | BOOLEAN | -subs | Include subdomains for crawling. |
| show-source-url | BOOLEAN | -s | Show the source of URL based on where it was found (href, form, script, etc.) |
| only-unique-urls | BOOLEAN | -u | Show only unique urls. |
| proxy | STRING | -proxy | Proxy URL. E.g. -proxy http://127.0.0.1:8080 |
Showing key inputs. hakrawler exposes 10 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of urls |
| proxy | STRING | -proxy | Proxy URL. E.g. -proxy http://127.0.0.1:8080 |
| threads | STRING | -t | Number of threads to utilise. (default 8) |
| insecure | BOOLEAN | -insecure | Disable TLS verification. |
| json-output | BOOLEAN | -json | Output as json |
| custom-header | STRING | -h | Custom headers separated by two semi-colons. E.g. -h "Cookie: foo=bar;;Referer: http://example.com/" |
| depth-to-crawl | STRING | -d | Depth to crawl. (default 2) |
| show-source-url | BOOLEAN | -s | Show the source of URL based on where it was found (href, form, script, etc.) |
| only-unique-urls | BOOLEAN | -u | Show only unique urls. |
| include-subdomains | BOOLEAN | -subs | Include subdomains for crawling. |
example
# crawl seeds to depth 2, unique URLs only, include subdomainscat urls.txt | hakrawler -d 2 -u -subs -t 8https://example.com/https://example.com/loginhttps://example.com/api/v1/statushttps://example.com/static/app.jshttps://www.example.com/docshttps://api.example.com/healthhttps://example.com/products?id=1https://staging.example.com/https://cdn.example.com/bundle.jsguidance
Use hakrawler for a fast, lightweight crawl that gathers URLs and JavaScript locations from a host list. For deeper crawling with headless JS rendering and richer scope controls, use katana. For passive-only archive URLs, use gau.
Deeper crawler with headless JS rendering and more scope controls. hakrawler is lighter and quicker to run.
Go spider with archive sources and more filtering. A close sibling with a larger option set.
Passive URLs from web archives with no live requests. Complements an active hakrawler crawl.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A host list feeds hakrawler, which crawls for URLs and JavaScript locations and passes them to httpx so only live ones land as output.
Facts on this page come from the live Trickest tool library.