Discovery
Map URLs, forms, and JS endpoints from a seed
Go web crawler for links, forms, and JS endpoints.
overview
What gospider does
gospider walks a seed site (-s) or site list (-S) and collects reachable URLs, forms, and parameters to a set recursion depth (-d).
Keep --js on for linkfinder over scripts. -a merges Archive.org, CommonCrawl, VirusTotal, and AlienVault. Scope with --whitelist, --blacklist, and --subs; pace with -t, -c, and -k.
Trickest provides gospider as a managed Discovery node. Folder out, optionally --json. Chain after subdomain discovery into httpx. Prefer katana for headless rendering; gau for passive archives only.
use cases
Where gospider fits
Crawl a site for its URL and parameter surface
Point gospider at a host to walk links and forms to a set depth and collect every reachable URL, including parameterized routes the next stage can probe or fuzz.
Recover endpoints from JavaScript
Keep linkfinder on so gospider parses script files for API routes and links that never appear in static HTML.
Pull URLs from third-party archives
Enable third-party sources to merge URLs from Archive.org, CommonCrawl, VirusTotal, and AlienVault into the live crawl results.
Keep a wide crawl in scope
Use whitelist and blacklist regexes plus subdomain controls so a deep, multi-site crawl stays inside the assets you are testing.
reference
gospider inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| site-to-crawl | STRING | -s | Single site URL to crawl. |
| sites-list-to-crawl | FILE | -S | File of sites to crawl in one run. |
| max-recursion-depth | STRING | -d | Maximum recursion depth of visited URLs (0 for infinite). |
| number-of-threads | STRING | -t | Number of threads, running sites in parallel. |
| enable-link-finder | BOOLEAN | --js | Run linkfinder over JavaScript to extract endpoints (on by default). |
| 3rd-party-url | BOOLEAN | -a | Pull URLs from Archive.org, CommonCrawl, VirusTotal, and AlienVault. |
| json-output | BOOLEAN | --json | Enable structured JSON output. |
| include-subdomains | BOOLEAN | --subs | Include subdomains while crawling. |
Showing key inputs. gospider exposes 31 inputs in total.
Full flag reference (31 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| raw | BOOLEAN | --raw | Print raw response output as received from each request. |
| proxy | STRING | -p | Proxy to route requests through (e.g. http://127.0.0.1:8080). |
| length | BOOLEAN | --length | Show the content length of each crawled URL. |
| verbose | BOOLEAN | --verbose | Enable verbose logging. |
| debug-mode | BOOLEAN | --debug | Enable debug logging. |
| user-agent | STRING | -u | User agent to send (web for a random desktop agent, mobi for a random mobile agent). |
| json-output | BOOLEAN | --json | Write results as structured JSON. |
| random-delay | STRING | -K | Extra randomized seconds added to the request delay before each new request. |
| 3rd-party-url | BOOLEAN | -a | Find URLs from third-party sources (Archive.org, CommonCrawl, VirusTotal, AlienVault). |
| cookie-to-use | STRING | --cookie | Cookie header to send with each request (testA=a; testB=b). |
| filter-length | BOOLEAN | --filter-length | Filter results by content length. |
| header-to-use | STRING | --header | Custom header to send; repeat the flag to set several. |
| request-delay | STRING | -k | Seconds to wait before each new request to matching domains. |
| site-to-crawl | STRING | -s | Single site URL to crawl. |
| crawl-robot-txt | BOOLEAN | --robots | Crawl robots.txt for additional paths (default true). |
| request-timeout | STRING | -m | Per-request timeout in seconds (default 10). |
| disable-redirect | BOOLEAN | --no-redirect | Do not follow HTTP redirects. |
| whitelist-domain | STRING | --whitelist-domain | Restrict crawling to this domain. |
| crawl-sitemap-xml | BOOLEAN | --sitemap | Crawl sitemap.xml for additional URLs. |
| html-only-content | BOOLEAN | --base | Disable extras and use only HTML content. |
| number-of-threads | STRING | -t | Number of threads, running sites in parallel (default 1). |
| enable-link-finder | BOOLEAN | --js | Run linkfinder over JavaScript files to extract endpoints (default true). |
| include-subdomains | BOOLEAN | --subs | Include subdomains while crawling. |
| blacklist-url-regex | STRING | --blacklist | Skip URLs matching this regex. |
| max-recursion-depth | STRING | -d | Maximum recursion depth of visited URLs (0 for infinite, default 1). |
| sites-list-to-crawl | FILE | -S | File of sites to crawl in one run. |
| whitelist-url-regex | STRING | --whitelist | Only crawl URLs matching this regex. |
| include-3rd-party-urls | BOOLEAN | -r | Also crawl and request URLs found from other sources. |
| number-of-concurent-req | STRING | -c | Maximum concurrent requests per matching domain (default 5). |
| include-3rd-party-subdomains | BOOLEAN | -w | Include subdomains discovered from third-party sources. |
| load-headers-and-strings-from-burp | STRING | --burp | Load headers and cookies from a Burp raw HTTP request. |
example
Run gospider
# crawl example.com to depth 2, parse JS, pull archive URLs, JSON outputgospider -s https://example.com -d 2 -c 5 -t 4 --js -a --json -o crawl-out[url] - [code-200] - https://example.com/[href] - https://example.com/assets/app.css[javascript] - https://example.com/static/js/main.4f2a.js[linkfinder] - /api/v2/users[form] - https://example.com/login[url] - [code-301] - https://example.com/blog[subdomains] - api.example.com[url] - [code-200] - https://cdn.example.com/img/logo.svgguidance
Choosing gospider
Use gospider for a light active crawl that also parses JavaScript and optional archive sources. For deeper scope controls and headless rendering, use katana. For passive archive URLs with no live requests, use gau.
katana
Deeper crawler with richer scope controls and headless JS rendering. gospider stays lighter to wire.
hakrawler
Minimal Go crawler from the same family. gospider adds archive sources and more filtering.
gau
Passive URLs from web archives with no live requests. Complements an active gospider crawl.
faq
gospider questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run gospider yourself
A site feeds gospider, which crawls it for URLs and passes them to httpx so only live endpoints land as a queryable output.
Facts on this page come from the live Trickest tool library.