Discovery
Crawl a web app for endpoints, including JavaScript and a headless mode
katana is a next-generation crawling and spidering framework from ProjectDiscovery that discovers endpoints in web applications with standard and headless-browser crawling and JavaScript parsing, output in pipe-friendly form for downstream pipelines.
installgo install github.com/projectdiscovery/katana/cmd/katana@latestdocker pull projectdiscovery/katana:latest
Go · MIT license
overview
What katana does
katana walks a web application from a seed URL, follows every link and form it reaches, and collects the endpoints it finds into a structured list. It crawls fast enough to cover wide scopes and controls scope tightly enough that the output feeds the next stage without manual cleanup.
Two things set it apart from a plain crawler. It parses JavaScript files for endpoints that never appear in static HTML, which is how it recovers API routes in single-page apps where the navigation lives in bundled script. A -headless hybrid mode drives a real Chrome instance for pages that only render under a browser, and -known-files pulls robots.txt and sitemap.xml for paths the crawl would otherwise skip.
In a Trickest workflow katana sits between discovery and testing. Point it at the hosts from a subdomain pass, emit -jsonl so every endpoint becomes one queryable record, and hand the results to httpx and a scanner so probing runs only against URLs that exist.
use cases
Where katana fits
Map an application's full endpoint surface
Crawl a target to depth and emit JSONL so every reachable path, parameter, and form is captured as structured data the rest of the workflow can query.
Pull endpoints out of JavaScript bundles
Enable JS crawling on single-page apps to recover API routes and links that static crawlers miss because the navigation is defined in script, not HTML.
Feed a prober and scanner pipeline
Run katana after subdomain discovery, then pass its output to httpx and a vulnerability scanner so probing and testing run only against real, reachable URLs.
Harvest known files and seed wordlists
Pull robots.txt and sitemap.xml entries, then use the discovered paths to seed content-discovery and parameter-fuzzing stages downstream.
reference
katana inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | -u | Target URL to crawl. |
| list | FILE | -list | File of target URLs, for crawling many hosts at once. |
| depth | STRING | -depth | Maximum depth to crawl (default 3). |
| js-crawl | BOOLEAN | -js-crawl | Parse JavaScript files for endpoints (short flag -jc). |
| headless | BOOLEAN | -headless | Enable headless hybrid crawling for rendered pages (experimental). |
| jsonl | BOOLEAN | -jsonl | Write output in JSONL format, one record per line. |
| field-scope | STRING | -field-scope | Pre-defined scope field (dn, rdn, fqdn) or custom regex (default rdn). |
| known-files | STRING | -known-files | Crawl known files (all, robotstxt, sitemapxml). |
Showing key inputs. katana exposes 64 inputs in total.
Full flag reference (64 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | -u | target url |
| list | FILE | -list | target url list |
| debug | BOOLEAN | -debug | display debug output |
| delay | STRING | -delay | request delay between each request in seconds |
| depth | STRING | -depth | maximum depth to crawl (default 3) |
| field | STRING | -field | field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) (Deprecated: use -output-template instead) |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| proxy | STRING | -proxy | http/socks5 proxy to use |
| retry | STRING | -retry | number of times to retry the request (default 1) |
| config | FILE | -config | path to the katana configuration file |
| resume | FILE | -resume | resume scan using resume.cfg |
| silent | BOOLEAN | -silent | display output only |
| exclude | STRING | -exclude | exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex) |
| headers | STRING | -headers | custom header/cookie to include in request |
| jsluice | BOOLEAN | -jsluice | enable jsluice parsing in javascript file (memory intensive) |
| timeout | STRING | -timeout | time to wait for request in seconds (default 10) |
| verbose | BOOLEAN | -verbose | display verbose output |
| headless | BOOLEAN | -headless | enable headless hybrid crawling (experimental) |
| js-crawl | BOOLEAN | -js-crawl | enable endpoint parsing / crawling in javascript file |
| no-scope | BOOLEAN | -no-scope | disables host based default scope |
| omit-raw | BOOLEAN | -omit-raw | omit raw requests/responses from jsonl output |
| strategy | STRING | -strategy | Visit strategy (depth-first, breadth-first) (default "depth-first") |
| omit-body | BOOLEAN | -omit-body | omit response body from jsonl output |
| resolvers | STRING | -resolvers | list of custom resolver (file or comma separated) |
| no-sandbox | BOOLEAN | -no-sandbox | start headless chrome in --no-sandbox mode |
| path-climb | BOOLEAN | -path-climb | enable path climb (auto crawl parent paths) |
| rate-limit | STRING | -rate-limit | maximum requests to send per second (default 150) |
| concurrency | STRING | -concurrency | number of concurrent fetchers to use (default 10) |
| crawl-scope | STRING | -crawl-scope | in scope url regex to be followed by crawler |
| field-scope | STRING | -field-scope | pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn") |
| form-config | FILE | -form-config | path to custom form configuration file |
| known-files | STRING | -known-files | enable crawling of known files (all,robotstxt,sitemapxml) |
| match-regex | STRING | -match-regex | regex to match on output url |
| parallelism | STRING | -parallelism | number of concurrent inputs to process (default 10) |
| store-field | STRING | -store-field | field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) |
| tech-detect | BOOLEAN | -tech-detect | enable technology detection (works with jsonl output) |
| field-config | FILE | -field-config | path to custom field configuration file |
| filter-regex | STRING | -filter-regex | regex to filter on output url |
| headers-file | FILE | -headers | custom header/cookie to include in request |
| health-check | BOOLEAN | -health-check | run diagnostic check up |
| no-incognito | BOOLEAN | -no-incognito | start headless chrome without incognito mode |
| crawl-duration | STRING | -crawl-duration | maximum duration to crawl the target for (s, m, h, d) (default s) |
| xhr-extraction | BOOLEAN | -xhr-extraction | extract xhr request url,method in jsonl output |
| chrome-data-dir | FOLDER | -chrome-data-dir | path to store chrome browser data |
| crawl-out-scope | STRING | -crawl-out-scope | out of scope url regex to be excluded by crawler |
| extension-match | STRING | -extension-match | match output for given extension (eg, -em php,html,js) |
| form-extraction | BOOLEAN | -form-extraction | extract form, input, textarea & select elements in jsonl output |
| match-condition | STRING | -match-condition | match response with dsl based condition |
| output-template | STRING | -output-template | custom output template |
| tls-impersonate | BOOLEAN | -tls-impersonate | enable experimental client hello (ja3) tls randomization |
| crawl-scope-file | FILE | -crawl-scope | in scope url regex to be followed by crawler |
| extension-filter | STRING | -extension-filter | filter output for given extension (eg, -ef png,css) |
| filter-condition | STRING | -filter-condition | filter response with dsl based condition |
| headless-options | STRING | -headless-options | start headless chrome with additional options |
| match-regex-file | FILE | -match-regex | list of regex to match on output url |
| disable-redirects | BOOLEAN | -disable-redirects | disable following redirects (default false) |
| display-out-scope | BOOLEAN | -display-out-scope | display external endpoint from scoped crawling |
| filter-regex-file | FILE | -filter-regex | list of regex to filter on output url |
| max-response-size | STRING | -max-response-size | maximum response size to read (default 9223372036854775807) |
| rate-limit-minute | STRING | -rate-limit-minute | maximum number of requests to send per minute |
| automatic-form-fill | BOOLEAN | -automatic-form-fill | enable automatic form filling (experimental) |
| ignore-query-params | BOOLEAN | -ignore-query-params | Ignore crawling same path with different query-param values |
| crawl-out-scope-file | FILE | -crawl-out-scope | out of scope url regex to be excluded by crawler |
| disable-unique-filter | BOOLEAN | -disable-unique-filter | disable duplicate content filtering |
example
Run katana
# crawl to depth 3, parse JavaScript, include known files, write JSONL
katana -u https://example.com -depth 3 -js-crawl -known-files all -jsonl -o endpoints.jsonlhttps://example.com/
https://example.com/about
https://example.com/login
https://example.com/api/v1/users
https://example.com/api/v1/products?id=1
https://example.com/assets/app.min.js
https://example.com/robots.txt
https://example.com/sitemap.xml
https://example.com/dashboard/settingsguidance
Choosing katana
Reach for katana when you need depth and JavaScript awareness, not just a flat URL dump. For passive, archive-sourced URLs with no requests, use gau. For a lighter, simpler crawl, hakrawler or gospider are quicker to wire but see less.
gau
Passive URLs from archives like Wayback and Common Crawl. No live requests, complements an active katana crawl.
hakrawler
Lightweight active crawler. Faster to run, less coverage, no JS rendering.
gospider
Fast Go spider. Good for quick link discovery, fewer scoping controls than katana.
workflows
Workflows using katana
Find SSRF in HTTP Parameters
Crawl a target and mine its archived URLs, then test every parameter that takes a URL for server-side request forgery using out-of-band detection.
SPA Endpoint & Parameter Map
Map the route, API and parameter attack surface of a modern JS app (Next, Nuxt, React) from its JS bundles. No browser, no auth.
faq
katana questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run katana yourself
A target feeds katana, which crawls it and passes the endpoints to httpx for live probing before they land as a queryable output.
Facts on this page come from the live Trickest tool library.