loading
Loading content
loading
Discovery
Search archives of URLs exposed via shortener services.
overview
urlhunter searches collections of URLs that were exposed through link shorteners like bit.ly and goo.gl. The project brute forces shortener services and publishes the expanded links as daily archives; urlhunter downloads those archives and lets you search them by keyword. People shorten links to private documents, internal dashboards, and one-time shares all the time, so these archives can hold URLs that were never meant to be public.
The model is offline search over downloaded data. You give it a date or a date range to choose which daily collections to pull, point it at a folder where the archives live (it downloads them on first run), and supply a file of keywords to match against the expanded URLs. Every result is a real, expanded link that someone once hid behind a short code.
On Trickest, urlhunter is a Discovery node that takes a date range, an archive folder, and a keywords file, then writes a file and a folder of matching URLs. Reach for it as an OSINT step to check whether an organization's domains, project names, or internal hostnames show up in the shortener archives, and wire the matches into a prober for follow-up.
source github.com/utkusen/urlhunter
use cases
Search the shortener archives for an organization's domain or project name to surface short links that expanded to sensitive internal URLs.
Use a single date or a range to choose which daily collections urlhunter downloads and searches, keeping a run fast and focused.
Point urlhunter at a persistent archives folder so later runs search already-downloaded collections instead of refetching them.
Pass the matched URLs to a prober or a parameter extractor so discovered links get classified and tested like any other endpoint.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| keywords | FILE | --keywords | Path to a file of strings to search for in the archives. |
| date | STRING | --date | Single date (2020-11-20) or range (2020-11-10:2020-11-20) of collections to search. |
| archives | FOLDER | --archives | Directory holding the archive files; downloaded to ./archives on first run. |
Showing key inputs. urlhunter exposes 3 inputs in total.
example
# search a date range of shortener archives for keywordsurlhunter --keywords keywords.txt --date 2020-11-10:2020-11-20 --archives ./archives[*] Downloading the archive: 2020-11-20.tar.gz[*] Uncompressing the archive: 2020-11-20.tar.gz[*] Matches are being searched across 1 archive(s)...https://bit.ly/2Kx9dLp|https://example.com/shared/q3-forecast.pdfhttps://bit.ly/3aQ1wZt|https://docs.example.com/internal/onboardinghttps://goo.gl/f4Rt2M|https://staging.example.com/dashboard?ref=emailhttps://bit.ly/2Yh8Nre|http://198.51.100.24/backups/db.sql[*] urlhunter is finished! Matches saved to: output.txtguidance
Use urlhunter as an OSINT step to find URLs that leaked through link shorteners, by searching published archives of expanded short links. It works offline over downloaded collections. Pair it with passive URL tools like urlfinder and gau for a broader picture of an organization's exposed links.
Passive URL discovery from archives and OSINT. urlhunter searches shortener archives specifically.
Pulls URLs from Wayback and Common Crawl. urlhunter targets shortened-link collections.
Deep web-archive harvesting. urlhunter focuses on the shortener-exposed URL dataset.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A keywords file feeds urlhunter, which searches the shortener archives over a date range and writes the matching expanded URLs as a queryable output.
Facts on this page come from the live Trickest tool library.