loading
Loading content
loading
Content Discovery
Crawl live web server URLs, follow links and parse JavaScript, and emit a path map per asset. Nest when the surface is linked, not guessed or archived.
Web crawling maps the paths an application actually links, so you can scope targeted scans instead of guessing from a wordlist. Hidden path discovery starts from the linked surface: feed it a list of live URLs and the module walks what the app exposes, including JavaScript the crawl can see, and emits a path map per asset. Unlinked paths and archived URLs belong on the sibling modules, not this one.
You pass a file of live web server URLs and the module follows linked content, including JavaScript the crawl can see, then emits a path map per asset. Depth defaults to 5; headless browser mode defaults to false, so routes that exist only after render stay hidden unless you turn it on. This maps what the app exposes; it does not guess unlinked paths and it does not query archives.
Pipeline guidance
Upstream: Probe for Web Servers, or any inventory that already emits live URLs. Nest after liveness.
Downstream: The path map feeds Scan for Sensitive Files, Scan for Exposed Admin Panels, Fuzz Web Applications for Vulnerabilities, and Analyze JavaScript Code for deeper script review.
Parallel: Discover Paths via Directory Brute Force and Discover Paths via OSINT Sources. Same urls / url-details shape, different data_source. Run them beside this, not instead of it.
Nest this after you have live web server URLs. Crawling a dead host returns nothing. You drop the subgraph into a workflow on a typed FILE face; you do not run a crawler from a shell. Use it when you want the linked surface before you guess unlinked paths.
The tables this module produces. Sample rows are illustrative.
Crawled path map: one URL per line, linked content only.
urlurl | |
|---|---|
| 1 | https://example.com/ |
| 2 | https://example.com/login |
| 3 | https://example.com/assets/app.js |
JSONLines per crawled URL with response metadata.
urlurl | methodtext | status_codeint | titletext | content_lengthint | hostnametext | |
|---|---|---|---|---|---|---|
| 1 | https://example.com/ | GET | 200 | Example Domain | 1256 | example.com |
| 2 | https://example.com/login | GET | 200 | Sign In | 3421 | example.com |
| 3 | https://example.com/api/users | POST | 201 | 87 | example.com |
Artifact class it owns
Unlinked paths from a wordlist; masked 404s dropped
Upstream it needs
Live web server URLs
Discover Paths via Crawling
Artifact class it owns
Linked paths, including JavaScript the crawl can see
Upstream it needs
Live web server URLs
Artifact class it owns
Third-party and archived URLs for those hosts
Upstream it needs
Hostnames
questions
Drop this module into a Trickest workflow and connect it to your upstream and downstream nodes. The face contract above is the wiring.
Get a personalized demo
A 30-minute walkthrough. We map the platform to your stack and answer pricing and deployment questions for your environment.