loading
Loading content
loading
Discovery
A fast web spider written in Go.
overview
gospider walks a site from a seed URL and pulls out every link, form, and parameter it can reach, following references across pages to map the full URL surface. It is built for speed, with concurrent requests, configurable threads, and a recursion depth you set to trade coverage for time.
Beyond plain HTML it runs a built-in linkfinder over JavaScript files to recover API routes and endpoints that never appear in static markup, reads robots.txt and sitemap.xml, and can merge URLs from third-party archives like Archive.org, CommonCrawl, VirusTotal, and AlienVault. Whitelist and blacklist regexes, subdomain inclusion, and per-domain delays keep a large crawl focused and polite.
On Trickest, gospider is a Discovery node that takes a site or a file of sites and writes a folder of results. Turn on JSON output and chain it after subdomain discovery to crawl the live surface, then hand the URLs to a prober like httpx and a scanner.
use cases
Point gospider at a host to walk links and forms to a set depth and collect every reachable URL, including parameterized routes the next stage can probe or fuzz.
Keep linkfinder on so gospider parses script files for API routes and links that never appear in static HTML.
Enable third-party sources to merge URLs from Archive.org, CommonCrawl, VirusTotal, and AlienVault into the live crawl results.
Use whitelist and blacklist regexes plus subdomain controls so a deep, multi-site crawl stays inside the assets you are testing.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| site-to-crawl | STRING | -s | Single site URL to crawl. |
| sites-list-to-crawl | FILE | -S | File of sites to crawl in one run. |
| max-recursion-depth | STRING | -d | Maximum recursion depth of visited URLs (0 for infinite). |
| number-of-threads | STRING | -t | Number of threads, running sites in parallel. |
| enable-link-finder | BOOLEAN | --js | Run linkfinder over JavaScript to extract endpoints (on by default). |
| 3rd-party-url | BOOLEAN | -a | Pull URLs from Archive.org, CommonCrawl, VirusTotal, and AlienVault. |
| json-output | BOOLEAN | --json | Enable structured JSON output. |
| include-subdomains | BOOLEAN | --subs | Include subdomains while crawling. |
Showing key inputs. gospider exposes 31 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| raw | BOOLEAN | --raw | Print raw response output as received from each request. |
| proxy | STRING | -p | Proxy to route requests through (e.g. http://127.0.0.1:8080). |
| length | BOOLEAN | --length | Show the content length of each crawled URL. |
| verbose | BOOLEAN | --verbose | Enable verbose logging. |
| debug-mode | BOOLEAN | --debug | Enable debug logging. |
| user-agent | STRING | -u | User agent to send (web for a random desktop agent, mobi for a random mobile agent). |
| json-output | BOOLEAN | --json | Write results as structured JSON. |
| random-delay | STRING | -K | Extra randomized seconds added to the request delay before each new request. |
| 3rd-party-url | BOOLEAN | -a | Find URLs from third-party sources (Archive.org, CommonCrawl, VirusTotal, AlienVault). |
| cookie-to-use | STRING | --cookie | Cookie header to send with each request (testA=a; testB=b). |
| filter-length | BOOLEAN | --filter-length | Filter results by content length. |
| header-to-use | STRING | --header | Custom header to send; repeat the flag to set several. |
| request-delay | STRING | -k | Seconds to wait before each new request to matching domains. |
| site-to-crawl | STRING | -s | Single site URL to crawl. |
| crawl-robot-txt | BOOLEAN | --robots | Crawl robots.txt for additional paths (default true). |
| request-timeout | STRING | -m | Per-request timeout in seconds (default 10). |
| disable-redirect | BOOLEAN | --no-redirect | Do not follow HTTP redirects. |
| whitelist-domain | STRING | --whitelist-domain | Restrict crawling to this domain. |
| crawl-sitemap-xml | BOOLEAN | --sitemap | Crawl sitemap.xml for additional URLs. |
| html-only-content | BOOLEAN | --base | Disable extras and use only HTML content. |
| number-of-threads | STRING | -t | Number of threads, running sites in parallel (default 1). |
| enable-link-finder | BOOLEAN | --js | Run linkfinder over JavaScript files to extract endpoints (default true). |
| include-subdomains | BOOLEAN | --subs | Include subdomains while crawling. |
| blacklist-url-regex | STRING | --blacklist | Skip URLs matching this regex. |
| max-recursion-depth | STRING | -d | Maximum recursion depth of visited URLs (0 for infinite, default 1). |
| sites-list-to-crawl | FILE | -S | File of sites to crawl in one run. |
| whitelist-url-regex | STRING | --whitelist | Only crawl URLs matching this regex. |
| include-3rd-party-urls | BOOLEAN | -r | Also crawl and request URLs found from other sources. |
| number-of-concurent-req | STRING | -c | Maximum concurrent requests per matching domain (default 5). |
| include-3rd-party-subdomains | BOOLEAN | -w | Include subdomains discovered from third-party sources. |
| load-headers-and-strings-from-burp | STRING | --burp | Load headers and cookies from a Burp raw HTTP request. |
example
# crawl example.com to depth 2, parse JS, pull archive URLs, JSON outputgospider -s https://example.com -d 2 -c 5 -t 4 --js -a --json -o crawl-out[url] - [code-200] - https://example.com/[href] - https://example.com/assets/app.css[javascript] - https://example.com/static/js/main.4f2a.js[linkfinder] - /api/v2/users[form] - https://example.com/login[url] - [code-301] - https://example.com/blog[subdomains] - api.example.com[url] - [code-200] - https://cdn.example.com/img/logo.svgguidance
Use gospider for a fast active crawl that also reads JavaScript and archive sources. For a deeper crawler with more scoping controls and headless rendering, reach for katana. For passive-only archive URLs with no requests, use gau.
Deeper crawler with richer scope controls and headless JS rendering. gospider is lighter and quick to wire.
Minimal Go crawler from the same family. gospider adds archive sources and more filtering.
Passive URLs from web archives with no live requests. Complements an active gospider crawl.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A site feeds gospider, which crawls it for URLs and passes them to httpx so only live endpoints land as a queryable output.
Facts on this page come from the live Trickest tool library.