loading
Loading content
loading
Recon
Extract URLs for a specific target from the results of commoncrawl.org.
overview
ccpy queries Common Crawl, the petabyte-scale open web archive, for every URL it has recorded under a target domain. Because the data already sits in the archive, you collect a site's paths, parameters, and long-dead endpoints without sending a single request to the host, so the recon stays passive and quiet.
Point it at a domain and it walks Common Crawl's monthly indexes. Narrow the pull to one year, crawl a single index for a deeper sweep, or list and update the available indexes when you want to control which snapshots you read from. The result is a broad list of historical URLs, including paths that no longer appear on the live site.
As a Recon node on Trickest it takes a domain and writes a file and a folder of URLs. Run it early to seed an endpoint inventory, then pass the results to a prober like httpx and a fuzzer so both the live and the forgotten paths get tested.
source github.com/si9int/cc.py
use cases
Pull a domain's historical URLs from Common Crawl to build a broad starting set of paths and parameters without sending any traffic to the target.
Surface URLs the archive recorded that no longer appear on the live site, often the stale endpoints worth probing for old bugs.
Limit results to a single year to focus on a time window, or crawl one Common Crawl index in full when you want every recorded path from that snapshot.
Hand the recovered URLs to httpx to find which still respond, then to a fuzzer or nuclei so both live and historical paths get tested.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| domain | STRING | · | Domain to extract archived URLs for. |
| crawl-index | STRING | --index | Crawl one specific Common Crawl index, pulling all its pages. |
| year | STRING | --year | Limit results to a specific year (default all). |
| list-indexes | BOOLEAN | --list | List all available Common Crawl indexes. |
| update-index-file | BOOLEAN | --update | Refresh the local index file before querying. |
Showing key inputs. ccpy exposes 5 inputs in total.
example
# list available indexes, then extract 2021 URLs for a domaincc.py --listcc.py --domain example.com --year 2021https://example.com/https://example.com/index.phphttps://example.com/login?redirect=%2Faccounthttps://example.com/blog/2019/03/release-noteshttps://example.com/wp-content/uploads/2018/report.pdfhttps://example.com/api/v1/users?id=1042https://static.example.com/js/app.min.jshttps://example.com/old/admin/config.bak… (2,317 URLs saved to output/example.com.txt)guidance
Use ccpy for passive URL discovery from Common Crawl at the start of a workflow. It reads an archive, not the target, so nothing reaches the host. It finds URLs, it does not test them, so follow it with httpx, and pair it with gau or waymore for wider archive coverage.
Pulls URLs from Wayback, Common Crawl, and more in one tool. ccpy focuses purely on Common Crawl data.
Aggressive archive harvester across many sources. ccpy is the lighter, Common Crawl-only option.
Pulls historical robots.txt entries from Wayback. ccpy returns the broader archived URL set.
faq
related
Asynchronous DNS brute-force tool for fast subdomain enumeration.
OWASP Amass: network mapping and external asset discovery.
OWASP Amass intel: find an organization's root domains and ranges.
OWASP Amass enumeration that produces structured JSON output.
Find related domains and subdomains via shared Google Analytics IDs.
Find domains and subdomains potentially related to a given domain.
A domain feeds ccpy, which pulls its archived URLs from Common Crawl and hands them to httpx so only the still-live paths land as a queryable output.
Facts on this page come from the live Trickest tool library.