loading
Loading content
loading
Utilities
A CLI utility to pull out bits of URLs.
overview
xurl takes a list of URLs and extracts one component from each. Choose a mode (domains, apexes, paths, query, params, values, or format) and it strips everything else away, so a messy URL dump becomes a clean column of the field you need. Format mode lets you build a custom output string from URL parts when the fixed modes do not fit.
Its value is as connective tissue in a pipeline. Between a crawler and a fuzzer you often want the parameter names, the unique apex domains, or the paths alone. xurl does that one transform cheaply and can deduplicate as it goes, which turns the output of one stage into the precise input the next stage expects without hand-editing.
Slot it between collection and testing stages to reshape URLs, extract parameters for a fuzzer, isolate apex domains for scoping, or pull paths for content discovery.
source github.com/hueristiq/xurl
use cases
Run params mode over a URL set to get a clean list of parameter names, the exact input a parameter fuzzer or value tester wants next.
Use apexes mode to reduce a pile of URLs to their root domains, deduplicated, so you can scope or group work by organization.
Extract the paths from collected URLs to seed a content-discovery wordlist tailored to how the target structures its routes.
Use format mode to assemble a custom line from URL parts when none of the fixed modes produces the exact shape the next tool expects.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| mode | STRING | · | Mode: domains, apexes, paths, query, params, values, or format. |
| input | FILE | --input | Input file of URLs. |
| format-string | STRING | · | Format string used in format mode to build custom output. |
| unique | BOOLEAN | --unique | Output unique values only. |
| verbosity | STRING | --verbosity | Log level: debug, info, warning, error, fatal, or silent (default info). |
| monochrome | BOOLEAN | --monochrome | Disable output content coloring. |
Showing key inputs. xurl exposes 6 inputs in total.
example
# xurl: unique hostnames from a crawl dumpxurl domains --input urls.txt --uniqueexample.comwww.example.comapi.example.comstaging.example.comcdn.example.commail.example.comdev.example.comapp.example.comguidance
Use xurl as glue between stages when you need one URL component extracted cleanly, parameters, apexes, paths, or a custom format. For collecting the URLs in the first place, use a crawler such as katana or a passive source such as gau.
Similar URL-component extractor. Choose by the exact modes and format syntax you prefer.
Collects URLs from web archives. Use it to produce the URL list that xurl then reshapes.
faq
related
Import, export, and link workflow data with Airtable.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Extract all hosted zones from AWS Route53.
Output file lines by batch size, given START_LINE and END_LINE.
Extract a batch range from a file's lines or a folder's files, with parallel processing.
A file of URLs feeds xurl, which extracts one component, here the parameter names, and writes the clean, deduplicated list as an output for the next stage.
Facts on this page come from the live Trickest tool library.