hakrawler
Fast golang web crawler for gathering URLs and JavaSript file locations. This is basically a simple implementation of the awesome Gocolly library.
Details
Category: Discovery
Publisher: trickest
Created Date: 6/23/2021
Container: quay.io/trickest/hakrawler:6190559
Source URL: https://github.com/hakluke/hakrawler
Parameters
urls
file
requiredCommand:
- List of urlsproxy
string
Command:
-proxy
- Proxy URL. E.g. -proxy http://127.0.0.1:8080threads
string
Command:
-t
- Number of threads to utilise. (default 8)insecure
boolean
Command:
-insecure
- Disable TLS verification.json-output
boolean
Command:
-json
- Output as jsoncustom-header
string
Command:
-h
- Custom headers separated by two semi-colons. E.g. -h Cookie: foo=bar;;Referer: http://example.com/depth-to-crawl
string
Command:
-d
- Depth to crawl. (default 2)show-source-url
boolean
Command:
-s
- Show the source of URL based on where it was found (href, form, script, etc.)only-unique-urls
boolean
Command:
-u
- Show only unique urls.include-subdomains
boolean
Command:
-subs
- Include subdomains for crawling.