Content Discovery
Discover Paths via Crawling
Crawl live web server URLs, follow links and parse JavaScript, and emit a path map per asset. Nest when the surface is linked, not guessed or archived.
What Discover Paths via Crawling does
Web crawling maps the paths an application actually links, so you can scope targeted scans instead of guessing from a wordlist. Hidden path discovery starts from the linked surface: feed it a list of live URLs and the module walks what the app exposes, including JavaScript the crawl can see, and emits a path map per asset. Unlinked paths and archived URLs belong on the sibling modules, not this one.
You pass a file of live web server URLs and the module follows linked content, including JavaScript the crawl can see, then emits a path map per asset. Depth defaults to 5; headless browser mode defaults to false, so routes that exist only after render stay hidden unless you turn it on. This maps what the app exposes; it does not guess unlinked paths and it does not query archives.
Where it fits in the pipeline
Pipeline guidance
Upstream: Probe for Web Servers, or any inventory that already emits live URLs. Nest after liveness.
Downstream: The path map feeds Scan for Sensitive Files, Scan for Exposed Admin Panels, Fuzz Web Applications for Vulnerabilities, and Analyze JavaScript Code for deeper script review.
Parallel: Discover Paths via Directory Brute Force and Discover Paths via OSINT Sources. Same urls / url-details shape, different data_source. Run them beside this, not instead of it.
When to use this
Nest this after you have live web server URLs. Crawling a dead host returns nothing. You drop the subgraph into a workflow on a typed FILE face; you do not run a crawler from a shell. Use it when you want the linked surface before you guess unlinked paths.
Data
The tables this module produces. Sample rows are illustrative.
Crawled path map: one URL per line, linked content only.
urlurl | |
|---|---|
| 1 | https://example.com/ |
| 2 | https://example.com/login |
| 3 | https://example.com/assets/app.js |
JSONLines per crawled URL with response metadata.
urlurl | methodtext | status_codeint | titletext | content_lengthint | hostnametext | |
|---|---|---|---|---|---|---|
| 1 | https://example.com/ | GET | 200 | Example Domain | 1256 | example.com |
| 2 | https://example.com/login | GET | 200 | Sign In | 3421 | example.com |
| 3 | https://example.com/api/users | POST | 201 | 87 | example.com |
How it differs from similar modules
Artifact class it owns
Unlinked paths from a wordlist; masked 404s dropped
Upstream it needs
Live web server URLs
Discover Paths via Crawling
Artifact class it owns
Linked paths, including JavaScript the crawl can see
Upstream it needs
Live web server URLs
Artifact class it owns
Third-party and archived URLs for those hosts
Upstream it needs
Hostnames
questions
Discover Paths via Crawling questions
Nest Discover Paths via Crawling in a workflow
Drop this module into a Trickest workflow and connect it to your upstream and downstream nodes. The face contract above is the wiring.
Get a personalized demo
See Trickest in Action
A 30-minute walkthrough. We map the platform to your stack and answer pricing and deployment questions for your environment.