Can it feed a vulnerability scanner directly?

Yes. --push-to-proxy forwards every captured request to a proxy address so a passive scanner tests the crawl traffic as-is.

Does it fill out forms?

Yes. Provide --form-values and it fills and submits forms during the crawl, including login and search shapes you define.

How do I limit how far the crawl goes?

Set --max-crawled-count (default 200) and choose --filter-mode simple, smart, or strict so large targets stay manageable.

What does --fuzz-path do?

It probes the target with common paths. Point --fuzz-path-dict at a custom dictionary when the builtin list is not enough.

When should I pick katana instead?

When you need faster coverage over many hosts and can accept lighter JS handling. Keep crawlergo for browser-faithful request harvest on a focused app.

Updated Jul 15, 2026

Discovery

Crawl JavaScript apps with a real Chrome browser

Browser-driven crawler that harvests requests for downstream scanners.

Crawl JavaScript-heavy applications Capture full requests for a scanner Exercise forms during the crawl

overview

What crawlergo does

crawlergo drives Chrome against a same-host URL set so the crawl sees rendered pages, filled forms, and DOM-triggered requests that static link spiders miss. The output is a request map suited to single-page and JavaScript-heavy apps.

Bound scope with --max-crawled-count and --filter-mode (simple, smart, or strict). Enable --fuzz-path for common-path probes. Set --push-to-proxy when a passive scanner should test the exact traffic the crawl produced, including POST bodies.

Use it in discovery after you have a concrete app URL and need browser-faithful endpoints. Prefer katana when speed over a wide scope matters more than full rendering. Set --output-mode json when the next stage parses structured requests.

source github.com/Qianlitp/crawlergo

use cases

Where crawlergo fits

Crawl JavaScript-heavy applications

Render the target in Chrome and trigger DOM events to reach routes and API calls that live in bundled JavaScript, not in static anchor tags.

Capture full requests for a scanner

Push every collected request to a proxy address so a passive scanner downstream tests the exact traffic the crawl produced, including POST bodies.

Exercise forms during the crawl

Supply form values so login and search forms get filled and submitted, surfacing post-authentication and result pages the crawler would otherwise miss.

Bound a wide crawl

Set a max-crawled-count and a filter mode to keep large targets manageable, then feed the resulting endpoint set into content discovery or fuzzing.

reference

crawlergo inputs and flags

23 inputs

Name	Type	Flag	Description
url	STRING	·	URLs to scan, separated by spaces (must be same host).
output-mode	STRING	--output-mode	Serialize output: console, json, or none.
filter-mode	STRING	--filter-mode	How collected requests are filtered: simple, smart, or strict (default smart).
max-crawled-count	STRING	--max-crawled-count	Maximum number of URLs visited in this task (default 200).
fuzz-path	BOOLEAN	--fuzz-path	Fuzz the target with common paths (default false).
form-values	STRING	--form-values	Custom filling text per form type, for example username and password.
push-to-proxy	STRING	--push-to-proxy	Push every captured request to this proxy address for downstream scanning.
robots-path	BOOLEAN	--robots-path	Resolve paths from /robots.txt (default false).

Showing key inputs. crawlergo exposes 23 inputs in total.

Full flag reference (23 inputs)

Name	Type	Flag	Description
url	STRING	·	URLs to scan, separated by spaces (must be same host)
fuzz-path	BOOLEAN	--fuzz-path	whether to fuzz the target with common paths. (default: false)
log-level	STRING	--log-level	log print Level, options include debug, info, warn, error and fatal. (default: "Info")
post-data	STRING	--post-data	set PostData to target and use POST method.
encode-url	BOOLEAN	--encode-url	whether to encode url with detected charset. (default: false)
filter-mode	STRING	--filter-mode	filtering Mode used for collected requests. Allowed mode:"simple", "smart" or "strict". (default: "smart")
form-values	STRING	--form-values	custom filling text for each form type. e.g.: -fv username=crawlergo_nice -fv password=admin123
output-mode	STRING	--output-mode	console print or serialize output. Allowed mode:"console" ,"json" or "none". (default: "console")
robots-path	BOOLEAN	--robots-path	whether to resolve paths from /robots.txt. (default: false)
max-tab-count	STRING	--max-tab-count	maximum Number of tabs allowed. (default: 8)
push-pool-max	STRING	--push-pool-max	maximum Number of concurrency when pushing results to proxy. (default: 10)
push-to-proxy	STRING	--push-to-proxy	every request in 'req_list' will be pushed to the proxy Address"
request-proxy	STRING	--request-proxy	all requests connect through defined proxy server.
custom-headers	STRING	--custom-headers	add additional Headers to each request. The input string will be called json.Unmarshal (default: {"Spider-Name": "crawlergo", "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.0 Safari/537.36"})
fuzz-path-dict	FILE	--fuzz-path-dict	Fuzz dict
tab-run-timeout	STRING	--tab-run-timeout	the Timeout of a single tab task. (default: 20s)
before-exit-delay	STRING	--before-exit-delay	the Time of waiting before crawler exit. (default: 1s)
max-crawled-count	STRING	--max-crawled-count	the maximum Number of URLs visited by the crawler in this task. (default: 200)
event-trigger-mode	STRING	--event-trigger-mode	this Value determines how the crawler automatically triggers events.Allowed mode:"async" or "sync". (default: "async")
form-keyword-values	STRING	--form-keyword-values	custom filling text, fuzzy matched by keyword. e.g.: -fkv user=crawlergo_nice -fkv pass=admin123
ignore-url-keywords	STRING	--ignore-url-keywords	crawlergo will not crawl these URLs matched by Keywords. Default [logout quit exit])
event-trigger-interval	STRING	--event-trigger-interval	the Interval of triggering each event. (default: 100ms)
wait-dom-content-loaded-timeout	STRING	--wait-dom-content-loaded-timeout	the Timeout of waiting for a page dom ready. (default: 5s)

example

Run crawlergo

crawlergo · command

# crawl example.com with smart filtering and JSON outputcrawlergo --filter-mode smart --output-mode json --max-crawled-count 100 --fuzz-path https://example.com/

sample output

https://example.com/https://example.com/loginhttps://example.com/api/v1/usershttps://example.com/api/v1/products?id=1https://example.com/dashboardhttps://example.com/static/app.jshttps://example.com/search?q=testhttps://www.example.com/robots.txt

guidance

Choosing crawlergo

Use crawlergo when the target relies on JavaScript and you need a real browser to observe its endpoints and requests. It is heavier than a static crawler, so prefer katana when speed matters more than full rendering. Pair with a proxy-based scanner via --push-to-proxy.

katana

Fast crawler with optional JS parsing. Lighter than a full browser; quicker over wide scopes.

hakrawler

Lightweight active crawler. Faster to run; no headless rendering.

gospider

Fast Go spider for quick link discovery; no browser-driven event triggering.

faq

crawlergo questions

It drives Chrome, so it renders JavaScript and triggers DOM events. That reaches routes and requests a static HTTP crawler never observes.

Run crawlergo yourself

A target feeds crawlergo, which renders it in Chrome to harvest endpoints, then passes them to httpx for live probing before they land as a queryable output.

Open crawlergo in the library Build a workflow with it

Facts on this page come from the live Trickest tool library.