---
title: "Web Crawling"
canonical: https://trickest.com/glossary/web-crawling
description: "Web crawling follows links and parses pages to map an application's URLs, parameters, and endpoints, building the inventory that later testing checks."
---

# Web Crawling

Web crawling, also called spidering, walks an application by following its links. A crawler starts from a seed URL, fetches the page, extracts every link, form, and script reference, then repeats on each new URL until it has traced the reachable surface. Modern crawlers parse JavaScript and inspect API responses, since single-page apps hide most of their routes behind client-side code rather than plain anchor tags.

This matters because you cannot test what you have not found. Before a scanner checks for injection or access-control flaws, it needs the full set of endpoints and the parameters each one accepts. A crawl that misses a route leaves that route untested, so coverage of the map drives coverage of the assessment. Crawling also exposes parameters that feed [fuzzing](/glossary/fuzzing) later.

Crawling discovers what the application links to. [Content discovery](/glossary/content-discovery) and [directory brute forcing](/glossary/directory-brute-forcing) go further by guessing paths that exist but appear nowhere in the markup. Together they build the endpoint inventory that [web application security](/glossary/web-application-security) testing depends on.

In Trickest you run a crawler as a workflow node, pipe its output of URLs and parameters straight into vulnerability checks, and crawl many hosts in parallel without managing each tool by hand.

---
_Markdown source of https://trickest.com/glossary/web-crawling._