loading
Loading content
loading
Discovery
Headless check for URLs that return 200 but render not-found.
overview
404checker takes a file of candidate URLs and keeps only pages that render as real content. Soft 404s answer HTTP 200 with a not-found body; status filters in ffuf or gobuster miss them. Headless browsers render each URL so a 200 cannot hide a decoy page.
Point --input_file at one URL per line. Raise --threads and --processes for large sets, and set --max-urls so overflow spills to the next pass instead of stalling. Sorted input groups similar responses and helps the browsers move faster.
In a workflow the managed node sits after content discovery and before probing or scanning. Wire a URL file to --input_file; it writes a file and folder of survivors. Use httpx for liveness and title checks; use 404checker when the failure mode is a masked not-found page.
use cases
Run 404checker on the output of a fuzzer or crawler to drop URLs that return 200 but render a not-found page, so only real endpoints reach the next stage.
ffuf and gobuster drop hard 404s by status code, but a soft 404 answers 200 and slips through. Run 404checker on the survivors to remove the masked pages those filters leave behind.
Filter masked 404s out of a candidate list so a vulnerability scanner spends its time on pages that exist instead of decoy responses.
Push a large sorted URL list through multiple browser processes to flag soft-404 redirects across a whole estate in one pass.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| input-file | FILE | --input_file | Input file with URLs, one per line. |
| threads | STRING | --threads | Number of threads (default 50). |
| processes | STRING | --processes | Number of browser processes (default number of CPUs). |
| max-urls | STRING | --max-urls | Max URLs per pass; the rest spill over (default 50000). |
| user-agent | STRING | --user-agent | User-Agent string to send with each request. |
| verbose | BOOLEAN | --verbose | Print verbose output. |
Showing key inputs. 404checker exposes 6 inputs in total.
example
# keep only URLs that resolve to a real page, dropping masked 404spython3 404checker.py --input_file urls.txt --output_file real-urls.txt --threads 50 --processes 4https://example.com/admin/loginhttps://example.com/api/v1/statushttps://example.com/dashboardhttps://example.com/uploads/https://example.com/.git/confighttps://example.com/backup/db.sqlhttps://203.0.113.24/portal/… (4,812 URLs in, 2,190 real, 2,622 masked 404s dropped)guidance
Use 404checker between content discovery and probing when a URL list may include soft 404s that answer 200. It does not discover or crawl; feed it ffuf, feroxbuster, or crawler output. Prefer httpx when you need status, title, and size rather than soft-404 detection.
Probes hosts for status, title, and size. Use for liveness and classification, not soft-404 detection.
Discovers content and can filter by size or words. 404checker is purpose-built for the masked-404 case after discovery.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A URL list feeds 404checker, which renders each one and writes the URLs that resolve to real pages as a queryable output.
Facts on this page come from the live Trickest tool library.