loading
Loading content
loading
Utilities
Extract chosen URL parts from stdin into clean line lists.
overview
unfurl reads a list of URLs and emits one chosen component per line: domains, apexes, paths, query keys, values, or key=value pairs. That turns a crawl dump into the slice the next stage needs.
Built-in selectors cover the common cases. format mode templates any combination of URL parts into a custom shape. --unique deduplicates as it emits so hosts or parameters become tidy wordlists.
Trickest runs unfurl as a managed utility node that takes a URL file and writes a file plus a folder. Place it between a crawler and a fuzzer or prober when you need hosts, paths, or parameters without writing a parser.
source github.com/tomnomnom/unfurl
use cases
Run domains mode with --unique over a crawl dump to get a clean, deduplicated list of hostnames to feed a prober.
Pull query-string keys out of a URL set so a parameter fuzzer like x8 or ffuf has a real list to work from instead of guesses.
Use apexes mode to collapse sub.example.com and api.example.com down to example.com, handy for building an in-scope domain list.
Template scheme, host, and path with the format mode to emit exactly the URL shape a downstream tool expects, no scripting required.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of URLs |
| domains | BOOLEAN | domains | The hostname (e.g. sub.example.com) |
| apexes | BOOLEAN | apexes | The apex domain (e.g. example.com from sub.example.com) |
| paths | BOOLEAN | paths | The request path (e.g. /users) |
| keys | BOOLEAN | keys | Keys from the query string (one per line) |
| values | BOOLEAN | values | Values from the query string (one per line) |
| format | STRING | format | Specify a custom format |
| unique | BOOLEAN | --unique | Only output unique values |
Showing key inputs. unfurl exposes 11 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| json | BOOLEAN | json | JSON encoded url/format objects |
| keys | BOOLEAN | keys | Keys from the query string (one per line) |
| urls | FILE | · | List of URLs |
| paths | BOOLEAN | paths | The request path (e.g. /users) |
| apexes | BOOLEAN | apexes | The apex domain (e.g. example.com from sub.example.com) |
| format | STRING | format | Specify a custom format |
| unique | BOOLEAN | --unique | Only output unique values |
| values | BOOLEAN | values | Values from the query string (one per line) |
| domains | BOOLEAN | domains | The hostname (e.g. sub.example.com) |
| verbose | BOOLEAN | --verbose | Verbose mode (output URL parse errors) |
| keypairs | BOOLEAN | keypairs | Key=value pairs from the query string (one per line) |
example
# extract unique hostnames from a crawl dumpcat urls.txt | unfurl --unique domains # extract query keys for a parameter wordlistcat urls.txt | unfurl --unique keysexample.comwww.example.comapi.example.comstaging.example.comcdn.example.commail.example.comdev.example.comapp.example.comguidance
Reach for unfurl when you need one URL part from a list: hosts, apexes, paths, or query keys. It makes no requests. Run it between a crawler and a fuzzer or prober. For deduplicating whole URLs instead of extracting parts, use urldedupe.
Deduplicates whole URLs by path and query shape. unfurl extracts a chosen part.
Flattens JSON into greppable lines. unfurl does the same idea for URL components.
Rewrites query-string values. unfurl extracts keys and values rather than replacing them.
faq
related
Spider a URL and return a wordlist for password crackers.
Wordlist and mined-word subdomain permutation.
CSS selectors over HTML, the jq counterpart for markup.
Import, export, and upsert workflow data against an Airtable base.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
A list of URLs feeds unfurl, which extracts the unique hostnames and writes them as a queryable output for the next stage to scope against.
Facts on this page come from the live Trickest tool library.