gospider

--rawTurn on raw

-pProxy (Ex: http://127.0.0.1:8080)

--lengthTurn on length

--verboseTurn on verbose

--debugTurn on debug mode

-uUser Agent to use (web: random web user-agent, mobi: random mobile user-agent)

--jsonEnable JSON output

-KRandomDelay is the extra randomized duration to wait added to Delay before creating a new request (second)

-aFind URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com)

--cookieCookie to use (testA=a; testB=b)

--filter-lengthTurn on length filter

--headerHeader to use (Use multiple flag to set multiple header)

-kDelay is the duration to wait before creating a new request to the matching domains (second)

-sSite to crawl

--robotsTry to crawl robots.txt (default true)

-mRequest timeout (second) (default 10)

--no-redirectDisable redirect

--whitelist-domainWhitelist Domain

--sitemapTry to crawl sitemap.xml

--baseDisable all and only use HTML content

-tNumber of threads (Run sites in parallel) (default 1)

--jsEnable linkfinder in javascript file (default true)

--subsInclude subdomains

--blacklistBlacklist URL Regex

-dMaxDepth limits the recursion depth of visited URLs. (Set it to 0 for infinite recursion) (default 1)

-SSite list to crawl

--whitelistWhitelist URL Regex

-rAlso include other-source's urls (still crawl and request)

-cThe number of the maximum allowed concurrent requests of the matching domains (default 5)

-wInclude subdomains crawled from 3rd party. Default is main domain

--burpLoad headers and cookie from burp raw http request