> ## Documentation Index
> Fetch the complete documentation index at: https://trickest.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Discover Paths via Crawling
> Crawl a list of web server URLs to discover endpoints and form a comprehensive map of each asset on your attack surface
export const ModuleOverview = ({category, inputs, outputs, author, createdDate, iframeUrl, long_description}) =>
{category}
Created by
{author}
Last updated
{new Date(createdDate).toLocaleDateString()}
{long_description}
Input Parameters
{Object.entries(inputs).map(([key, value]) =>
{value.description}
)}
Outputs
{outputs.map((output, index) =>
{output}
)}
;
# Discover Paths via Crawling
## Description
Crawl a list of web server URLs to discover endpoints and form a comprehensive map of each asset on your attack surface.
## Features
* Supports **headless browser crawling** for more accurate spidering.
* Parses **JavaScript code to discover additional endpoints** and hidden paths.
* Can crawl **thousands of web servers** simulataneously.
## Inputs
### Required
* **urls:** a list of URLs
```
https://foo.example.com
https://bar.example.com
```
### Optional
* **depth**: Maximum crawling depth (default: 5)
* **headless**: Enable headless browser mode (default: false)
* **header**: Header(s) to include in HTTP requests
* **header-file**: File with header(s) to include in HTTP requests
* **rate-limit**: Maximum number of requests to send per second per machine (default: 300)
### Outputs
* **urls:** List of discovered URLs.
```
https://foo.example.com/about
https://foo.example.com/login
https://bar.example.com/app.js
https://bar.example.com/admin
```
* **url-details**: JSONLines records of URL discovery details.
```json theme={null}
{"url": "https://foo.example.com/about", "hostname": "foo.example.com", "domain_name": "example.com", "data_source": "crawling", "status_code": 200, "content_length": 9283}
{"url": "https://foo.example.com/login", "hostname": "foo.example.com", "domain_name": "example.com", "data_source": "crawling", "status_code": 200, "content_length": 2031}
{"url": "https://bar.example.com/app.js", "hostname": "bar.example.com", "domain_name": "example.com", "data_source": "crawling", "status_code": 200, "content_length": 4212}
{"url": "https://bar.example.com/admin", "hostname": "bar.example.com", "domain_name": "example.com", "data_source": "crawling", "status_code": 403, "content_length": 385}
```
## Changelog
* v1.0.0
* Initial release
* v1.1.0
* Added `header-file` input
* v1.2.0
* Support exporting non-GET requests to the `url-details` output
* Automatically extract forms and fetch requests from crawled pages
* Submit POST requests using sample data
* Extend `url-details` format to support `method`, `body`, `body_parameters`, and `query_parameters` fields