Attack Surface Management
Enumerate Hostnames via Crawling
Crawl live web servers and pull hostnames from links, script, and headers. Input is URLs, not root domains. Last in the hostname enum family; nest after probe.
What Enumerate Hostnames via Crawling does
Some hostnames never appear in public records or DNS wordlists. They show up only in the HTML, JavaScript, and response headers of live web servers, which is why crawling live sites after a probe is the last step in subdomain enumeration and the one that catches what DNS brute force and OSINT miss. Internal aliases, analytics hosts, and linked third parties often surface here first.
You feed live web-server URLs after a probe. The module walks those sites and keeps hostnames found in HTML, JavaScript, and response headers, names that often never appear in public records or a wordlist.
Where it fits in the pipeline
Upstream
Live web-server URLs from probe for web servers. Dead names have nothing to parse.
Downstream
New subdomains back into probe for liveness, then into permutations and recursive DNS brute force if you want more names, and into DNS records once you have names to query.
Parallel
The other hostname enumerators run on domains or known hosts, not on live URLs. Path crawl is a different family: same walk, different artifact (paths, not hostnames).
When to use this
Nest this last in the hostname family, after Probe for Web Servers has turned names into live URLs. Crawling a dead host returns nothing. New names you keep go back through probe, then optionally into Enumerate Hostnames via DNS Permutations Brute Force and Enumerate Hostnames via Recursive DNS Brute Force. Family order: Enumerate Hostnames via OSINT Sources and Enumerate Hostnames via Root Domain DNS Brute Force first, then permutations and recursive, then probe, then this crawl.
Inputs and outputs
- Optional
depth: max crawl depth. Default 2.header/header-file: request headers when the site sits behind an auth wall.subdomains: hostnames found in the crawl.subdomain-details: JSONL with hostname, domain, data source (crawling), context (where it was found: an href, script text, a header), andlinked_asset. The same name can appear more than once when it shows up in more than one place.
Data
The tables this module produces. Sample rows are illustrative.
Hostnames pulled from the HTML, JavaScript, and response headers of live web servers.
hostnametext | domain_nametext | initial_assettext | initial_asset_typetext | discovery_methodtext | evidencetext | evidence_typetext | evidence_reproductiontext | |
|---|---|---|---|---|---|---|---|---|
| 1 | analytics.example.com | example.com | https://www.example.com | url | crawl | <script src="https://analytics.example.com/tracker.js"> | html script src | https://www.example.com/ |
| 2 | cdn.example.net | example.net | https://www.example.com | url | http response | Link: <https://cdn.example.net/assets/style.css>; rel=preload | http header | curl -I https://www.example.com/ |
| 3 | internal-admin.example.com | example.com | https://app.example.com | url | crawl | href="https://internal-admin.example.com/login" in HTML | html href | https://app.example.com/dashboard |
How it differs from similar modules
Artifact class it owns
Names third parties already know
Upstream it needs
domains
Enumerate Hostnames via Crawling
Artifact class it owns
Names in HTML, JS, headers of live sites
Upstream it needs
live URLs after probe
Artifact class it owns
Altered spellings of known hostnames
Upstream it needs
known hostnames
Artifact class it owns
First-level names against the apex
Upstream it needs
domains + wordlist
Artifact class it owns
Sub-subdomains of names you already have
Upstream it needs
known hostnames + level-2/3/4 wordlists
questions
Enumerate Hostnames via Crawling questions
Nest Enumerate Hostnames via Crawling in a workflow
Drop this module into a Trickest workflow and connect it to your upstream and downstream nodes. The face contract above is the wiring.
Get a personalized demo
See Trickest in Action
A 30-minute walkthrough. We map the platform to your stack and answer pricing and deployment questions for your environment.