Getting started

Knowledge hub

Tutorials

Concepts

gospider

Trickest - GitHub

Basic Usage Examples

Crawl One Site

Pass a site (e.g. http://trickest.io) to the site-to-crawl input (type string).

Run gospider against single site

Crawl Multiple Sites

Pass a list of sites, contained within a file, to the sites-to-crawl input (type file).

https://trickest.com
http://trickest.io

Run gospider against multiple site

Improvements

Number of Concurrent Requests

Default maximum allowed concurrent requests per given site is 5. To modify this value pass a desired number of concurrent requests to the number-of-concurent-req input (type string).

Recursion Depth

To set up a max depth limits for recursion depth of visited sites, pass a desired value to the max-recursion-depth input (type string).

Multi-threaded Crawl

To crawl multiple sites in parallel, pass desired number of threads to the threads input (type string).

Use Special User-Agent

To set special User-Agent to be used, pass a desired mobile or web User-Agent value to the user-agent input (type string).

Include Subdomains

Use include-subdomains boolean input to include subdomains crawled from given sites.

Notable Workflows