loading
Loading content
loading
Recon
Extract URLs for a specific target from commoncrawl.org indexes.
overview
ccpy queries Common Crawl for every URL recorded under a target domain. The data already sits in the archive, so you collect paths, parameters, and long-dead endpoints without sending a request to the host.
Point it at a domain and it walks monthly indexes. Narrow with --year, crawl one snapshot with --index, or use --list and --update when you want control over which indexes you read. Output is a broad historical URL list, including paths gone from the live site.
Trickest runs ccpy as a managed Recon node: domain in, file and folder of URLs out. Seed an endpoint inventory early, then pass results to httpx and a fuzzer so live and forgotten paths both get tested.
source github.com/si9int/cc.py
use cases
Pull a domain's historical URLs from Common Crawl to build a broad starting set of paths and parameters without sending any traffic to the target.
Surface URLs the archive recorded that no longer appear on the live site, often the stale endpoints worth probing for old bugs.
Limit results to a single year to focus on a time window, or crawl one Common Crawl index in full when you want every recorded path from that snapshot.
Hand the recovered URLs to httpx to find which still respond, then to a fuzzer or nuclei so both live and historical paths get tested.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| domain | STRING | · | Domain to extract archived URLs for. |
| crawl-index | STRING | --index | Crawl one specific Common Crawl index, pulling all its pages. |
| year | STRING | --year | Limit results to a specific year (default all). |
| list-indexes | BOOLEAN | --list | List all available Common Crawl indexes. |
| update-index-file | BOOLEAN | --update | Refresh the local index file before querying. |
Showing key inputs. ccpy exposes 5 inputs in total.
example
# list available indexes, then extract 2021 URLs for a domaincc.py --listcc.py --domain example.com --year 2021https://example.com/https://example.com/index.phphttps://example.com/login?redirect=%2Faccounthttps://example.com/blog/2019/03/release-noteshttps://example.com/wp-content/uploads/2018/report.pdfhttps://example.com/api/v1/users?id=1042https://static.example.com/js/app.min.jshttps://example.com/old/admin/config.bak… (2,317 URLs saved to output/example.com.txt)guidance
Use ccpy for passive URL discovery from Common Crawl at the start of a workflow. It reads an archive, not the target, so nothing reaches the host. It finds URLs; it does not test them. Follow with httpx, and pair with gau or waymore for wider archive coverage.
Pulls URLs from Wayback, Common Crawl, and more in one tool. ccpy focuses purely on Common Crawl.
Aggressive archive harvester across many sources. ccpy is the lighter, Common Crawl-only option.
Pulls historical robots.txt entries from Wayback. ccpy returns the broader archived URL set.
faq
related
Multi-source subdomain discovery; hand names to httpx.
Find domains and subdomains potentially related to a given domain.
CLI client for the BeVigil OSINT API, keyed by domain or app package.
Multi-cloud public name enumeration for AWS, Azure, and GCP.
List assets from multiple cloud providers in one inventory.
Active multi-technique DNS enumeration for assessments.
A domain feeds ccpy, which pulls its archived URLs from Common Crawl and hands them to httpx so only the still-live paths land as a queryable output.
Facts on this page come from the live Trickest tool library.