gospider
Basic Usage Examples
Crawl One Site
Pass a site (e.g. http://trickest.io) to the site-to-crawl input (type string).
Run gospider against single site
Crawl Multiple Sites
Pass a list of sites, contained within a file, to the sites-to-crawl input (type file).
https://trickest.com
http://trickest.io
Run gospider against multiple site
Improvements
Number of Concurrent Requests
The default maximum number of allowed concurrent requests per given site is 5. To modify this value pass a desired number of concurrent requests to the number-of-concurent-req input (type string).
Recursion Depth
To set up a max depth limits for recursion depth of visited sites, pass a desired value to the max-recursion-depth input (type string).
Multi-threaded Crawl
To crawl multiple sites in parallel, pass desired number of threads to the threads input (type string).
Use Special User-Agent
To set special User-Agent to be used, pass a desired mobile or web User-Agent value to the user-agent input (type string).
Include Subdomains
Use include-subdomains boolean input to include subdomains crawled from given sites.
