loading
Loading content
loading
Discovery
Search archives of URLs exposed via shortener services.
overview
urlhunter downloads daily archives of expanded short links (bit.ly, goo.gl) and matches a --keywords file against them. Scope collections with --date (single day or range). Point --archives at a folder so later runs reuse downloads.
Reach for it when private docs, internal dashboards, or one-time shares may have been shortened and later published in those archives. Every hit is a real expanded URL that once hid behind a short code.
Scope with --keywords and --date, then hand the FILE and FOLDER of expanded URLs to httpx or urlfinder. Pair with gau when you also need Wayback and Common Crawl coverage beyond shortener datasets.
source github.com/utkusen/urlhunter
use cases
Search the shortener archives for an organization's domain or project name to surface short links that expanded to sensitive internal URLs.
Use a single date or a range to choose which daily collections urlhunter downloads and searches, keeping a run fast and focused.
Point urlhunter at a persistent archives folder so later runs search already-downloaded collections instead of refetching them.
Pass the matched URLs to a prober or a parameter extractor so discovered links get classified and tested like any other endpoint.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| keywords | FILE | --keywords | Path to a file of strings to search for in the archives. |
| date | STRING | --date | Single date (2020-11-20) or range (2020-11-10:2020-11-20) of collections to search. |
| archives | FOLDER | --archives | Directory holding the archive files; downloaded to ./archives on first run. |
Showing key inputs. urlhunter exposes 3 inputs in total.
example
# search a date range of shortener archives for keywordsurlhunter --keywords keywords.txt --date 2020-11-10:2020-11-20 --archives ./archives[*] Downloading the archive: 2020-11-20.tar.gz[*] Uncompressing the archive: 2020-11-20.tar.gz[*] Matches are being searched across 1 archive(s)...https://bit.ly/2Kx9dLp|https://example.com/shared/q3-forecast.pdfhttps://bit.ly/3aQ1wZt|https://docs.example.com/internal/onboardinghttps://goo.gl/f4Rt2M|https://staging.example.com/dashboard?ref=emailhttps://bit.ly/2Yh8Nre|http://198.51.100.24/backups/db.sql[*] urlhunter is finished! Matches saved to: output.txtguidance
Use urlhunter when you need OSINT over shortener-exposed URL archives, not general web-archive harvests. Search is offline after the --archives download. Pair with urlfinder or gau for broader passive URL coverage.
Passive URL discovery from archives and OSINT sources. urlhunter searches shortener archives only.
Pulls URLs from Wayback and Common Crawl. urlhunter targets shortened-link collections.
Deep web-archive harvesting. urlhunter stays on the shortener-exposed URL dataset.
faq
related
Extract URLs and endpoints from Android APK files.
Web path scanner.
Crawl pages, harvest potential parameters, write a custom wordlist.
Recursive content discovery with smart defaults and rich response filters.
Maintained gau fork for passive archive URL collection.
Extract JavaScript file URLs from a page or URL list.
A keywords file feeds urlhunter, which searches the shortener archives over a date range and writes the matching expanded URLs as a queryable output.
Facts on this page come from the live Trickest tool library.