hakrawler
Fast golang web crawler for gathering URLs and JavaSript file locations. This is basically a simple implementation of the awesome Gocolly library.
Details
Category: Discovery
Publisher: trickest
Created Date: 6/23/2021
Container: quay.io/trickest/hakrawler:6190559
Source URL: https://github.com/hakluke/hakrawler
Parameters
Command:
- List of urlsCommand:
-proxy
- Proxy URL. E.g. -proxy http://127.0.0.1:8080Command:
-t
- Number of threads to utilise. (default 8)Command:
-insecure
- Disable TLS verification.Command:
-json
- Output as jsonCommand:
-h
- Custom headers separated by two semi-colons. E.g. -h Cookie: foo=bar;;Referer: http://example.com/Command:
-d
- Depth to crawl. (default 2)Command:
-s
- Show the source of URL based on where it was found (href, form, script, etc.)Command:
-u
- Show only unique urls.Command:
-subs
- Include subdomains for crawling.