Recon
Pull a domain archive from Common Crawl without touching the host
Extract URLs for a specific target from commoncrawl.org indexes.
overview
What ccpy does
ccpy queries Common Crawl for every URL recorded under a target domain. The data already sits in the archive, so you collect paths, parameters, and long-dead endpoints without sending a request to the host.
Point it at a domain and it walks monthly indexes. Narrow with --year, crawl one snapshot with --index, or use --list and --update when you want control over which indexes you read. Output is a broad historical URL list, including paths gone from the live site.
Trickest runs ccpy as a managed Recon node: domain in, file and folder of URLs out. Seed an endpoint inventory early, then pass results to httpx and a fuzzer so live and forgotten paths both get tested.
source github.com/si9int/cc.py
use cases
Where ccpy fits
Seed an endpoint inventory passively
Pull a domain's historical URLs from Common Crawl to build a broad starting set of paths and parameters without sending any traffic to the target.
Recover forgotten and removed paths
Surface URLs the archive recorded that no longer appear on the live site, often the stale endpoints worth probing for old bugs.
Scope a crawl by year or index
Limit results to a single year to focus on a time window, or crawl one Common Crawl index in full when you want every recorded path from that snapshot.
Feed probing and fuzzing stages
Hand the recovered URLs to httpx to find which still respond, then to a fuzzer or nuclei so both live and historical paths get tested.
reference
ccpy inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| domain | STRING | · | Domain to extract archived URLs for. |
| crawl-index | STRING | --index | Crawl one specific Common Crawl index, pulling all its pages. |
| year | STRING | --year | Limit results to a specific year (default all). |
| list-indexes | BOOLEAN | --list | List all available Common Crawl indexes. |
| update-index-file | BOOLEAN | --update | Refresh the local index file before querying. |
Showing key inputs. ccpy exposes 5 inputs in total.
example
Run ccpy
# list available indexes, then extract 2021 URLs for a domaincc.py --listcc.py --domain example.com --year 2021https://example.com/https://example.com/index.phphttps://example.com/login?redirect=%2Faccounthttps://example.com/blog/2019/03/release-noteshttps://example.com/wp-content/uploads/2018/report.pdfhttps://example.com/api/v1/users?id=1042https://static.example.com/js/app.min.jshttps://example.com/old/admin/config.bak… (2,317 URLs saved to output/example.com.txt)guidance
Choosing ccpy
Use ccpy for passive URL discovery from Common Crawl at the start of a workflow. It reads an archive, not the target, so nothing reaches the host. It finds URLs; it does not test them. Follow with httpx, and pair with gau or waymore for wider archive coverage.
gau
Pulls URLs from Wayback, Common Crawl, and more in one tool. ccpy focuses purely on Common Crawl.
waymore
Aggressive archive harvester across many sources. ccpy is the lighter, Common Crawl-only option.
waybackrobots
Pulls historical robots.txt entries from Wayback. ccpy returns the broader archived URL set.
faq
ccpy questions
related
More Recon tools
amass
Multi-source subdomain discovery; hand names to httpx.
amass-intel
OWASP Amass intel: map an organization's root domains and ranges.
assetfinder
Find domains and subdomains potentially related to a given domain.
bbot
Modular OSINT recon that chains modules from a seed target.
bevigil
CLI client for the BeVigil OSINT API, keyed by domain or app package.
cloud-enum
Multi-cloud public name enumeration for AWS, Azure, and GCP.
Run ccpy yourself
A domain feeds ccpy, which pulls its archived URLs from Common Crawl and hands them to httpx so only the still-live paths land as a queryable output.
Facts on this page come from the live Trickest tool library.