Discovery
Strip masked 404s before probing a URL list
Headless check for URLs that return 200 but render not-found.
overview
What 404checker does
404checker takes a file of candidate URLs and keeps only pages that render as real content. Soft 404s answer HTTP 200 with a not-found body; status filters in ffuf or gobuster miss them. Headless browsers render each URL so a 200 cannot hide a decoy page.
Point --input_file at one URL per line. Raise --threads and --processes for large sets, and set --max-urls so overflow spills to the next pass instead of stalling. Sorted input groups similar responses and helps the browsers move faster.
In a workflow the managed node sits after content discovery and before probing or scanning. Wire a URL file to --input_file; it writes a file and folder of survivors. Use httpx for liveness and title checks; use 404checker when the failure mode is a masked not-found page.
use cases
Where 404checker fits
Clean a content-discovery URL set
Run 404checker on the output of a fuzzer or crawler to drop URLs that return 200 but render a not-found page, so only real endpoints reach the next stage.
Catch what status-code filters miss
ffuf and gobuster drop hard 404s by status code, but a soft 404 answers 200 and slips through. Run 404checker on the survivors to remove the masked pages those filters leave behind.
Cut false positives before scanning
Filter masked 404s out of a candidate list so a vulnerability scanner spends its time on pages that exist instead of decoy responses.
Validate redirect behavior at scale
Push a large sorted URL list through multiple browser processes to flag soft-404 redirects across a whole estate in one pass.
reference
404checker inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| input-file | FILE | --input_file | Input file with URLs, one per line. |
| threads | STRING | --threads | Number of threads (default 50). |
| processes | STRING | --processes | Number of browser processes (default number of CPUs). |
| max-urls | STRING | --max-urls | Max URLs per pass; the rest spill over (default 50000). |
| user-agent | STRING | --user-agent | User-Agent string to send with each request. |
| verbose | BOOLEAN | --verbose | Print verbose output. |
Showing key inputs. 404checker exposes 6 inputs in total.
example
Run 404checker
# keep only URLs that resolve to a real page, dropping masked 404spython3 404checker.py --input_file urls.txt --output_file real-urls.txt --threads 50 --processes 4https://example.com/admin/loginhttps://example.com/api/v1/statushttps://example.com/dashboardhttps://example.com/uploads/https://example.com/.git/confighttps://example.com/backup/db.sqlhttps://203.0.113.24/portal/… (4,812 URLs in, 2,190 real, 2,622 masked 404s dropped)guidance
Choosing 404checker
Use 404checker between content discovery and probing when a URL list may include soft 404s that answer 200. It does not discover or crawl; feed it ffuf, feroxbuster, or crawler output. Prefer httpx when you need status, title, and size rather than soft-404 detection.
httpx
Probes hosts for status, title, and size. Use for liveness and classification, not soft-404 detection.
ffuf
Discovers content and can filter by size or words. 404checker is purpose-built for the masked-404 case after discovery.
faq
404checker questions
related
More Discovery tools
apkurlgrep
Extract URLs and endpoints from Android APK files.
cariddi
Crawl a domain list and scan responses for endpoints, secrets, tokens, and juicy files.
crawlergo
Browser-driven crawler that harvests requests for downstream scanners.
dirsearch
Web path scanner.
fallparams
Crawl pages, harvest potential parameters, write a custom wordlist.
feroxbuster
Recursive content discovery with smart defaults and rich response filters.
Run 404checker yourself
A URL list feeds 404checker, which renders each one and writes the URLs that resolve to real pages as a queryable output.
Facts on this page come from the live Trickest tool library.