hakrawler
Fast golang web crawler for gathering URLs and JavaSript file locations. This is basically a simple implementation of the awesome Gocolly library.
Name:hakrawler
Category:Discovery
Publisher:trickest
Created:6/23/2021
Container:
quay.io/trickest/hakrawler:6190559Output Type:
License:Unknown
Source:View Source
Parameters
-proxyProxy URL. E.g. -proxy http://127.0.0.1:8080-tNumber of threads to utilise. (default 8)-insecureDisable TLS verification.-jsonOutput as json-hCustom headers separated by two semi-colons. E.g. -h Cookie: foo=bar;;Referer: http://example.com/-dDepth to crawl. (default 2)-sShow the source of URL based on where it was found (href, form, script, etc.)-uShow only unique urls.-subsInclude subdomains for crawling.