loading
Loading content
loading
Utilities
Deduplicate URLs by path and query-string shape.
overview
urldedupe reads a URL list and keeps one representative per path and query-string shape, not per byte-identical line. Crawl and archive dumps often repeat the same endpoint with different parameter values; this collapses that noise.
Modes stack. -s collapses similar routes that differ by integers or asset files. -qs keeps only parameterized URLs. -ne drops extensioned assets. -r switches to a regex parser when messy URLs need it.
Trickest provides urldedupe as a managed Utilities node. Feed urls-file after a crawler or archive pull; take FILE and FOLDER into a fuzzer or parameter scanner.
use cases
Collapse thousands of near-identical URLs to one per shape so a fuzzer or scanner spends its budget on distinct endpoints.
Enable similar-URL removal so /api/user/1 and /api/user/2 reduce to a single representative instead of flooding the list.
Use query-strings-only mode to drop static URLs and pass on only the endpoints that have parameters worth testing.
Turn on no-extensions to remove .png, .js, .woff, and similar files so testing focuses on dynamic endpoints.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls-file | FILE | -u | File containing URLs to deduplicate. |
| mode | STRING | -m | Comma-separated filters to enable (e.g. r, s, qs, ne). |
| remove-similar-urls | BOOLEAN | -s | Remove similar URLs based on integers and image/font files. |
| query-strings-only | BOOLEAN | -qs | Only include URLs that have query strings. |
| no-extensions | BOOLEAN | -ne | Exclude URLs that have a file extension (.png, .js, .html, and others). |
| regex-parse | BOOLEAN | -r | Use slower regex parsing for more thorough, accurate results. |
Showing key inputs. urldedupe exposes 6 inputs in total.
example
# collapse a crawl dump to unique parameterized endpointsurldedupe -u urls.txt -s -qs > unique.txthttps://example.com/search?q=loginhttps://example.com/api/v2/user?id=1https://example.com/catalog?category=books&sort=pricehttps://example.com/account/settings?tab=securityhttps://app.example.com/reports?range=30d&format=jsonhttp://198.51.100.24/index.php?page=dashboardhttps://example.com/download?doc=invoicehttps://203.0.113.10/api/orders?status=openguidance
Use urldedupe to thin a noisy URL list to unique path and query shapes before an expensive fuzz or scan stage. It collapses whole URLs, including near-duplicates. To extract a single URL part, use unfurl; to append only new lines across runs, use anew.
Appends only previously unseen lines. urldedupe collapses near-duplicate URLs within one list.
Extracts a chosen URL component. urldedupe deduplicates whole URLs by shape.
Rewrites query values to a fixed token. Pair it with urldedupe to normalize then dedupe.
faq
related
Spider a URL and return a wordlist for password crackers.
Wordlist and mined-word subdomain permutation.
CSS selectors over HTML, the jq counterpart for markup.
Extract chosen URL parts from stdin into clean line lists.
Import, export, and upsert workflow data against an Airtable base.
Decode Android APK files into smali sources and resources.
A URL list feeds urldedupe, which collapses near-duplicates to unique shapes before ffuf fuzzes the survivors and writes the findings as a queryable output.
Facts on this page come from the live Trickest tool library.