loading
Loading content
loading
Utilities
Pull out bits of URLs provided on stdin.
overview
unfurl is a small, fast filter that takes a list of URLs and pulls out exactly the part you ask for. Want every unique hostname from a pile of crawled links? The apex domains? The query-string keys, or the request paths? unfurl returns each cleanly, one per line, so you can reshape a messy URL list into the slice the next stage of a pipeline needs.
Beyond the built-in selectors for domains, apexes, paths, keys, values, and key=value pairs, the format mode lets you template any combination of URL components into your own output shape. Add --unique and unfurl deduplicates as it goes, which turns a raw crawl dump into a tidy wordlist of hosts or parameters.
In a Trickest workflow it reads a file of URLs and writes a file and a folder. Drop it between a crawler or archive tool and a fuzzer or prober to extract domains for scoping, parameters for fuzzing, or paths for content discovery without writing a parser.
source github.com/tomnomnom/unfurl
use cases
Run domains mode with --unique over a crawl dump to get a clean, deduplicated list of hostnames to feed a prober.
Pull query-string keys out of a URL set so a parameter fuzzer like x8 or ffuf has a real list to work from instead of guesses.
Use apexes mode to collapse sub.example.com and api.example.com down to example.com, handy for building an in-scope domain list.
Template scheme, host, and path with the format mode to emit exactly the URL shape a downstream tool expects, no scripting required.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of URLs |
| domains | BOOLEAN | domains | The hostname (e.g. sub.example.com) |
| apexes | BOOLEAN | apexes | The apex domain (e.g. example.com from sub.example.com) |
| paths | BOOLEAN | paths | The request path (e.g. /users) |
| keys | BOOLEAN | keys | Keys from the query string (one per line) |
| values | BOOLEAN | values | Values from the query string (one per line) |
| format | STRING | format | Specify a custom format |
| unique | BOOLEAN | --unique | Only output unique values |
Showing key inputs. unfurl exposes 11 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| json | BOOLEAN | json | JSON encoded url/format objects |
| keys | BOOLEAN | keys | Keys from the query string (one per line) |
| urls | FILE | · | List of URLs |
| paths | BOOLEAN | paths | The request path (e.g. /users) |
| apexes | BOOLEAN | apexes | The apex domain (e.g. example.com from sub.example.com) |
| format | STRING | format | Specify a custom format |
| unique | BOOLEAN | --unique | Only output unique values |
| values | BOOLEAN | values | Values from the query string (one per line) |
| domains | BOOLEAN | domains | The hostname (e.g. sub.example.com) |
| verbose | BOOLEAN | --verbose | Verbose mode (output URL parse errors) |
| keypairs | BOOLEAN | keypairs | Key=value pairs from the query string (one per line) |
example
# extract unique hostnames from a crawl dumpcat urls.txt | unfurl --unique domains # extract query keys for a parameter wordlistcat urls.txt | unfurl --unique keysexample.comwww.example.comapi.example.comstaging.example.comcdn.example.commail.example.comdev.example.comapp.example.comguidance
Reach for unfurl whenever you need to slice URLs into one specific part: hosts, apexes, paths, or query keys. It is a pure transform with no requests. Run it between a crawler and a fuzzer or prober. For deduplicating whole URLs rather than extracting parts, use urldedupe.
Deduplicates whole URLs by path and query shape. unfurl extracts a chosen part instead.
Flattens JSON into greppable lines. unfurl does the same idea for URL components.
Rewrites query-string values. unfurl extracts the keys and values rather than replacing them.
faq
related
Import, export, and link workflow data with Airtable.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Extract all hosted zones from AWS Route53.
Output file lines by batch size, given START_LINE and END_LINE.
Extract a batch range from a file's lines or a folder's files, with parallel processing.
A list of URLs feeds unfurl, which extracts the unique hostnames and writes them as a queryable output for the next stage to scope against.
Facts on this page come from the live Trickest tool library.