Find Secrets in Wayback Responses

Overview

Secrets get committed, deployed, noticed, and removed. What survives is the copy a crawler took before anyone noticed. An API key that was live for one afternoon in 2021 is still in the archive, still valid if it was never rotated, and completely invisible to a scan of the site as it exists today.

This reads that history instead of the present. The run collects the archived URLs a domain has accumulated, pulls the stored response bodies back out of the archive, and scans those bodies with a full pattern and entropy detector set, checking each candidate credential against its provider where the detector supports it. You finish with findings that say whether the key still authenticates, redacted in the report so the artifact is safe to circulate, plus an inventory of the parameterized, admin and API endpoints the archive remembers even when no secret comes back.

Because every fetch goes to the archive rather than the target, nothing you run here shows up in the target's logs, which makes it usable for third-party exposure intelligence on vendors you have no agreement to test. On a schedule it picks up whatever the crawlers archived since the last run, which is where the next leaked key will appear.

Pipeline

1. Read the target domain.
2. gau collects every URL the web archives hold for it.
3. The URL list is normalized, deduplicated and capped so the fetch stays bounded.
4. Archived response bodies are fetched from the archive, not from the target.
5. trufflehog scans the bodies for credentials and verifies the ones it can.
6. Findings are redacted and written to a PDF report.

Inputs

• Target domain. The domain whose archived history is mined, with the subdomain switch on to cover *.domain rather than the apex alone. The default is example.com, which runs clean end to end.
• Archive providers. Which archives are queried. The default is the Wayback Machine on its own, which is the fast path. Adding Common Crawl, OTX and urlscan widens coverage and lengthens the run.
• Asset blacklist. The file extensions dropped before anything is fetched, so images, fonts and media do not spend the fetch budget. The default list is already tuned for this.
• URL cap. The ceiling on how many URLs reach the fetch stage after deduplication. It defaults to 1500 so one very large target cannot wedge the fleet.

Outputs

• Secrets report (PDF and HTML). Every finding redacted to its first and last four characters plus length, split into verified live credentials and unverified matches.
• Interesting endpoint inventory. The archived URLs worth a second look, grouped as parameterized, admin, API, sensitive file and secret hint. This is the artifact when no credential comes back.
• Normalized archive URL set. The deduplicated, asset-filtered list of historical URLs the scan actually ran over, which doubles as a parameter corpus for other work.

Sample output

The scan reports the detector that fired and where, with the secret itself redacted. Verified findings are the ones to act on first, because they mean the credential still authenticates:

url	detector	verified	severity
https://example.com/js/app.min.js	entropy	false	medium
https://example.com/config.json	aws-access-key	true	critical
https://example.com/.env.bak	dotenv	false	medium
https://dev.example.com/static/bundle.js	github-token	true	critical
https://example.com/api/docs	stripe-key	false	medium
https://staging.example.com/config.yml	private-key	false	high

gau returns the archived URL set, which on an established domain runs to tens of thousands of entries before capping. The rows above are the ones where a detector fired after that harvest.

FAQ

Why scan archived bodies instead of the live site?

Because the live site is the version somebody already cleaned. A key that shipped in a bundle for one afternoon is gone from today's response and still sitting in the snapshot a crawler took that afternoon. Scanning the present finds only the secrets nobody has noticed yet.

Does the target see this scan?

No. The URL harvest reads archive indexes, and every response body is pulled from the archive's copy, so the origin serves nothing and logs nothing. That is what makes the workflow usable against vendors and acquisition targets where you have no testing agreement.

How is a verified finding different from a match?

A match is a pattern or entropy hit: it looks like a credential. A verified finding is one the scan presented to the issuing provider at run time and got an accepted answer for, so the credential is still live. Verified findings rank critical and matches rank medium, and that split is your triage order.

What if the domain has almost nothing archived?

The run still completes and hands back the endpoint inventory rather than failing on an empty set. Widening the provider list beyond the Wayback Machine is the next thing to try before you conclude the history is thin.

How many archived URLs does one run cover?

The harvest is deduplicated and stripped of static assets first, then capped before fetching, with the cap at 1500 by default. On an established domain the raw archive set runs to tens of thousands of entries, so the cap is what keeps a single run bounded.

Related workflows

• Scan GitHub for Leaked Secrets. Reach for this one when the code is public, because commit history is a richer source of leaked keys than archived responses.
• Find Reflected XSS on a Domain. Reach for this one when the archived URLs are more useful as a parameter corpus to fuzz than as bodies to read.
• Directory & Content Discovery. Reach for this one when the archive turns out to be thin and you are willing to brute-force paths against the live site.