loading
Loading content
loading
Discovery
Go web crawler for links, forms, and JS endpoints.
overview
gospider walks a seed site (-s) or site list (-S) and collects reachable URLs, forms, and parameters to a set recursion depth (-d).
Keep --js on for linkfinder over scripts. -a merges Archive.org, CommonCrawl, VirusTotal, and AlienVault. Scope with --whitelist, --blacklist, and --subs; pace with -t, -c, and -k.
Trickest provides gospider as a managed Discovery node. Folder out, optionally --json. Chain after subdomain discovery into httpx. Prefer katana for headless rendering; gau for passive archives only.
use cases
Point gospider at a host to walk links and forms to a set depth and collect every reachable URL, including parameterized routes the next stage can probe or fuzz.
Keep linkfinder on so gospider parses script files for API routes and links that never appear in static HTML.
Enable third-party sources to merge URLs from Archive.org, CommonCrawl, VirusTotal, and AlienVault into the live crawl results.
Use whitelist and blacklist regexes plus subdomain controls so a deep, multi-site crawl stays inside the assets you are testing.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| site-to-crawl | STRING | -s | Single site URL to crawl. |
| sites-list-to-crawl | FILE | -S | File of sites to crawl in one run. |
| max-recursion-depth | STRING | -d | Maximum recursion depth of visited URLs (0 for infinite). |
| number-of-threads | STRING | -t | Number of threads, running sites in parallel. |
| enable-link-finder | BOOLEAN | --js | Run linkfinder over JavaScript to extract endpoints (on by default). |
| 3rd-party-url | BOOLEAN | -a | Pull URLs from Archive.org, CommonCrawl, VirusTotal, and AlienVault. |
| json-output | BOOLEAN | --json | Enable structured JSON output. |
| include-subdomains | BOOLEAN | --subs | Include subdomains while crawling. |
Showing key inputs. gospider exposes 31 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| raw | BOOLEAN | --raw | Print raw response output as received from each request. |
| proxy | STRING | -p | Proxy to route requests through (e.g. http://127.0.0.1:8080). |
| length | BOOLEAN | --length | Show the content length of each crawled URL. |
| verbose | BOOLEAN | --verbose | Enable verbose logging. |
| debug-mode | BOOLEAN | --debug | Enable debug logging. |
| user-agent | STRING | -u | User agent to send (web for a random desktop agent, mobi for a random mobile agent). |
| json-output | BOOLEAN | --json | Write results as structured JSON. |
| random-delay | STRING | -K | Extra randomized seconds added to the request delay before each new request. |
| 3rd-party-url | BOOLEAN | -a | Find URLs from third-party sources (Archive.org, CommonCrawl, VirusTotal, AlienVault). |
| cookie-to-use | STRING | --cookie | Cookie header to send with each request (testA=a; testB=b). |
| filter-length | BOOLEAN | --filter-length | Filter results by content length. |
| header-to-use | STRING | --header | Custom header to send; repeat the flag to set several. |
| request-delay | STRING | -k | Seconds to wait before each new request to matching domains. |
| site-to-crawl | STRING | -s | Single site URL to crawl. |
| crawl-robot-txt | BOOLEAN | --robots | Crawl robots.txt for additional paths (default true). |
| request-timeout | STRING | -m | Per-request timeout in seconds (default 10). |
| disable-redirect | BOOLEAN | --no-redirect | Do not follow HTTP redirects. |
| whitelist-domain | STRING | --whitelist-domain | Restrict crawling to this domain. |
| crawl-sitemap-xml | BOOLEAN | --sitemap | Crawl sitemap.xml for additional URLs. |
| html-only-content | BOOLEAN | --base | Disable extras and use only HTML content. |
| number-of-threads | STRING | -t | Number of threads, running sites in parallel (default 1). |
| enable-link-finder | BOOLEAN | --js | Run linkfinder over JavaScript files to extract endpoints (default true). |
| include-subdomains | BOOLEAN | --subs | Include subdomains while crawling. |
| blacklist-url-regex | STRING | --blacklist | Skip URLs matching this regex. |
| max-recursion-depth | STRING | -d | Maximum recursion depth of visited URLs (0 for infinite, default 1). |
| sites-list-to-crawl | FILE | -S | File of sites to crawl in one run. |
| whitelist-url-regex | STRING | --whitelist | Only crawl URLs matching this regex. |
| include-3rd-party-urls | BOOLEAN | -r | Also crawl and request URLs found from other sources. |
| number-of-concurent-req | STRING | -c | Maximum concurrent requests per matching domain (default 5). |
| include-3rd-party-subdomains | BOOLEAN | -w | Include subdomains discovered from third-party sources. |
| load-headers-and-strings-from-burp | STRING | --burp | Load headers and cookies from a Burp raw HTTP request. |
example
# crawl example.com to depth 2, parse JS, pull archive URLs, JSON outputgospider -s https://example.com -d 2 -c 5 -t 4 --js -a --json -o crawl-out[url] - [code-200] - https://example.com/[href] - https://example.com/assets/app.css[javascript] - https://example.com/static/js/main.4f2a.js[linkfinder] - /api/v2/users[form] - https://example.com/login[url] - [code-301] - https://example.com/blog[subdomains] - api.example.com[url] - [code-200] - https://cdn.example.com/img/logo.svgguidance
Use gospider for a light active crawl that also parses JavaScript and optional archive sources. For deeper scope controls and headless rendering, use katana. For passive archive URLs with no live requests, use gau.
Deeper crawler with richer scope controls and headless JS rendering. gospider stays lighter to wire.
Minimal Go crawler from the same family. gospider adds archive sources and more filtering.
Passive URLs from web archives with no live requests. Complements an active gospider crawl.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A site feeds gospider, which crawls it for URLs and passes them to httpx so only live endpoints land as a queryable output.
Facts on this page come from the live Trickest tool library.