loading
Loading content
loading
Discovery
A next-generation crawling and spidering framework.
overview
katana walks a web application from a seed URL, follows every link and form it reaches, and collects the endpoints it finds into a structured list. It crawls fast enough to cover wide scopes and controls scope tightly enough that the output feeds the next stage without manual cleanup.
Two things set it apart from a plain crawler. It parses JavaScript files for endpoints that never appear in static HTML, which is how it recovers API routes in single-page apps where the navigation lives in bundled script. A -headless hybrid mode drives a real Chrome instance for pages that only render under a browser, and -known-files pulls robots.txt and sitemap.xml for paths the crawl would otherwise skip.
In a Trickest workflow katana sits between discovery and testing. Point it at the hosts from a subdomain pass, emit -jsonl so every endpoint becomes one queryable record, and hand the results to httpx and a scanner so probing runs only against URLs that exist.
use cases
Crawl a target to depth and emit JSONL so every reachable path, parameter, and form is captured as structured data the rest of the workflow can query.
Enable JS crawling on single-page apps to recover API routes and links that static crawlers miss because the navigation is defined in script, not HTML.
Run katana after subdomain discovery, then pass its output to httpx and a vulnerability scanner so probing and testing run only against real, reachable URLs.
Pull robots.txt and sitemap.xml entries, then use the discovered paths to seed content-discovery and parameter-fuzzing stages downstream.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | -u | Target URL to crawl. |
| list | FILE | -list | File of target URLs, for crawling many hosts at once. |
| depth | STRING | -depth | Maximum depth to crawl (default 3). |
| js-crawl | BOOLEAN | -js-crawl | Parse JavaScript files for endpoints (short flag -jc). |
| headless | BOOLEAN | -headless | Enable headless hybrid crawling for rendered pages (experimental). |
| jsonl | BOOLEAN | -jsonl | Write output in JSONL format, one record per line. |
| field-scope | STRING | -field-scope | Pre-defined scope field (dn, rdn, fqdn) or custom regex (default rdn). |
| known-files | STRING | -known-files | Crawl known files (all, robotstxt, sitemapxml). |
Showing key inputs. katana exposes 64 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | -u | target url |
| list | FILE | -list | target url list |
| debug | BOOLEAN | -debug | display debug output |
| delay | STRING | -delay | request delay between each request in seconds |
| depth | STRING | -depth | maximum depth to crawl (default 3) |
| field | STRING | -field | field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) (Deprecated: use -output-template instead) |
| jsonl | BOOLEAN | -jsonl | write output in JSONL(ines) format |
| proxy | STRING | -proxy | http/socks5 proxy to use |
| retry | STRING | -retry | number of times to retry the request (default 1) |
| config | FILE | -config | path to the katana configuration file |
| resume | FILE | -resume | resume scan using resume.cfg |
| silent | BOOLEAN | -silent | display output only |
| exclude | STRING | -exclude | exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex) |
| headers | STRING | -headers | custom header/cookie to include in request |
| jsluice | BOOLEAN | -jsluice | enable jsluice parsing in javascript file (memory intensive) |
| timeout | STRING | -timeout | time to wait for request in seconds (default 10) |
| verbose | BOOLEAN | -verbose | display verbose output |
| headless | BOOLEAN | -headless | enable headless hybrid crawling (experimental) |
| js-crawl | BOOLEAN | -js-crawl | enable endpoint parsing / crawling in javascript file |
| no-scope | BOOLEAN | -no-scope | disables host based default scope |
| omit-raw | BOOLEAN | -omit-raw | omit raw requests/responses from jsonl output |
| strategy | STRING | -strategy | Visit strategy (depth-first, breadth-first) (default "depth-first") |
| omit-body | BOOLEAN | -omit-body | omit response body from jsonl output |
| resolvers | STRING | -resolvers | list of custom resolver (file or comma separated) |
| no-sandbox | BOOLEAN | -no-sandbox | start headless chrome in --no-sandbox mode |
| path-climb | BOOLEAN | -path-climb | enable path climb (auto crawl parent paths) |
| rate-limit | STRING | -rate-limit | maximum requests to send per second (default 150) |
| concurrency | STRING | -concurrency | number of concurrent fetchers to use (default 10) |
| crawl-scope | STRING | -crawl-scope | in scope url regex to be followed by crawler |
| field-scope | STRING | -field-scope | pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn") |
| form-config | FILE | -form-config | path to custom form configuration file |
| known-files | STRING | -known-files | enable crawling of known files (all,robotstxt,sitemapxml) |
| match-regex | STRING | -match-regex | regex to match on output url |
| parallelism | STRING | -parallelism | number of concurrent inputs to process (default 10) |
| store-field | STRING | -store-field | field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) |
| tech-detect | BOOLEAN | -tech-detect | enable technology detection (works with jsonl output) |
| field-config | FILE | -field-config | path to custom field configuration file |
| filter-regex | STRING | -filter-regex | regex to filter on output url |
| headers-file | FILE | -headers | custom header/cookie to include in request |
| health-check | BOOLEAN | -health-check | run diagnostic check up |
| no-incognito | BOOLEAN | -no-incognito | start headless chrome without incognito mode |
| crawl-duration | STRING | -crawl-duration | maximum duration to crawl the target for (s, m, h, d) (default s) |
| xhr-extraction | BOOLEAN | -xhr-extraction | extract xhr request url,method in jsonl output |
| chrome-data-dir | FOLDER | -chrome-data-dir | path to store chrome browser data |
| crawl-out-scope | STRING | -crawl-out-scope | out of scope url regex to be excluded by crawler |
| extension-match | STRING | -extension-match | match output for given extension (eg, -em php,html,js) |
| form-extraction | BOOLEAN | -form-extraction | extract form, input, textarea & select elements in jsonl output |
| match-condition | STRING | -match-condition | match response with dsl based condition |
| output-template | STRING | -output-template | custom output template |
| tls-impersonate | BOOLEAN | -tls-impersonate | enable experimental client hello (ja3) tls randomization |
| crawl-scope-file | FILE | -crawl-scope | in scope url regex to be followed by crawler |
| extension-filter | STRING | -extension-filter | filter output for given extension (eg, -ef png,css) |
| filter-condition | STRING | -filter-condition | filter response with dsl based condition |
| headless-options | STRING | -headless-options | start headless chrome with additional options |
| match-regex-file | FILE | -match-regex | list of regex to match on output url |
| disable-redirects | BOOLEAN | -disable-redirects | disable following redirects (default false) |
| display-out-scope | BOOLEAN | -display-out-scope | display external endpoint from scoped crawling |
| filter-regex-file | FILE | -filter-regex | list of regex to filter on output url |
| max-response-size | STRING | -max-response-size | maximum response size to read (default 9223372036854775807) |
| rate-limit-minute | STRING | -rate-limit-minute | maximum number of requests to send per minute |
| automatic-form-fill | BOOLEAN | -automatic-form-fill | enable automatic form filling (experimental) |
| ignore-query-params | BOOLEAN | -ignore-query-params | Ignore crawling same path with different query-param values |
| crawl-out-scope-file | FILE | -crawl-out-scope | out of scope url regex to be excluded by crawler |
| disable-unique-filter | BOOLEAN | -disable-unique-filter | disable duplicate content filtering |
example
# crawl to depth 3, parse JavaScript, include known files, write JSONLkatana -u https://example.com -depth 3 -js-crawl -known-files all -jsonl -o endpoints.jsonlhttps://example.com/https://example.com/abouthttps://example.com/loginhttps://example.com/api/v1/usershttps://example.com/api/v1/products?id=1https://example.com/assets/app.min.jshttps://example.com/robots.txthttps://example.com/sitemap.xmlhttps://example.com/dashboard/settingsguidance
Reach for katana when you need depth and JavaScript awareness, not just a flat URL dump. For passive, archive-sourced URLs with no requests, use gau. For a lighter, simpler crawl, hakrawler or gospider are quicker to wire but see less.
Passive URLs from archives like Wayback and Common Crawl. No live requests, complements an active katana crawl.
Lightweight active crawler. Faster to run, less coverage, no JS rendering.
Fast Go spider. Good for quick link discovery, fewer scoping controls than katana.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A target feeds katana, which crawls it and passes the endpoints to httpx for live probing before they land as a queryable output.
Facts on this page come from the live Trickest tool library.