loading
Loading content
loading
Utilities
Extract chosen URL parts from stdin into clean line lists.
overview
unfurl reads a list of URLs and emits one chosen component per line: domains, apexes, paths, query keys, values, or key=value pairs. That turns a crawl dump into the slice the next stage needs.
Built-in selectors cover the common cases. format mode templates any combination of URL parts into a custom shape. --unique deduplicates as it emits so hosts or parameters become tidy wordlists.
Trickest runs unfurl as a managed utility node that takes a URL file and writes a file plus a folder. Place it between a crawler and a fuzzer or prober when you need hosts, paths, or parameters without writing a parser.
source github.com/tomnomnom/unfurl
use cases
Run domains mode with --unique over a crawl dump to get a clean, deduplicated list of hostnames to feed a prober.
Pull query-string keys out of a URL set so a parameter fuzzer like x8 or ffuf has a real list to work from instead of guesses.
Use apexes mode to collapse sub.example.com and api.example.com down to example.com, handy for building an in-scope domain list.
Template scheme, host, and path with the format mode to emit exactly the URL shape a downstream tool expects, no scripting required.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls | FILE | · | List of URLs |
| domains | BOOLEAN | domains | The hostname (e.g. sub.example.com) |
| apexes | BOOLEAN | apexes | The apex domain (e.g. example.com from sub.example.com) |
| paths | BOOLEAN | paths | The request path (e.g. /users) |
| keys | BOOLEAN | keys | Keys from the query string (one per line) |
| values | BOOLEAN | values | Values from the query string (one per line) |
| format | STRING | format | Specify a custom format |
| unique | BOOLEAN | --unique | Only output unique values |
Showing key inputs. unfurl exposes 11 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| json | BOOLEAN | json | JSON encoded url/format objects |
| keys | BOOLEAN | keys | Keys from the query string (one per line) |
| urls | FILE | · | List of URLs |
| paths | BOOLEAN | paths | The request path (e.g. /users) |
| apexes | BOOLEAN | apexes | The apex domain (e.g. example.com from sub.example.com) |
| format | STRING | format | Specify a custom format |
| unique | BOOLEAN | --unique | Only output unique values |
| values | BOOLEAN | values | Values from the query string (one per line) |
| domains | BOOLEAN | domains | The hostname (e.g. sub.example.com) |
| verbose | BOOLEAN | --verbose | Verbose mode (output URL parse errors) |
| keypairs | BOOLEAN | keypairs | Key=value pairs from the query string (one per line) |
example
# extract unique hostnames from a crawl dumpcat urls.txt | unfurl --unique domains # extract query keys for a parameter wordlistcat urls.txt | unfurl --unique keysexample.comwww.example.comapi.example.comstaging.example.comcdn.example.commail.example.comdev.example.comapp.example.comguidance
Reach for unfurl when you need one URL part from a list: hosts, apexes, paths, or query keys. It makes no requests. Run it between a crawler and a fuzzer or prober. For deduplicating whole URLs instead of extracting parts, use urldedupe.
Deduplicates whole URLs by path and query shape. unfurl extracts a chosen part.
Flattens JSON into greppable lines. unfurl does the same idea for URL components.
Rewrites query-string values. unfurl extracts keys and values rather than replacing them.
faq
related
Spider a URL and return a wordlist for password crackers.
Wordlist and mined-word subdomain permutation.
Import, export, and upsert workflow data against an Airtable base.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Authenticated Route53 hosted-zone inventory.
A list of URLs feeds unfurl, which extracts the unique hostnames and writes them as a queryable output for the next stage to scope against.
Facts on this page come from the live Trickest tool library.