loading
Loading content
loading
Utilities
Deduplicate URLs by path and query-string shape.
overview
urldedupe reads a URL list and keeps one representative per path and query-string shape, not per byte-identical line. Crawl and archive dumps often repeat the same endpoint with different parameter values; this collapses that noise.
Modes stack. -s collapses similar routes that differ by integers or asset files. -qs keeps only parameterized URLs. -ne drops extensioned assets. -r switches to a regex parser when messy URLs need it.
Trickest provides urldedupe as a managed Utilities node. Feed urls-file after a crawler or archive pull; take FILE and FOLDER into a fuzzer or parameter scanner.
use cases
Collapse thousands of near-identical URLs to one per shape so a fuzzer or scanner spends its budget on distinct endpoints.
Enable similar-URL removal so /api/user/1 and /api/user/2 reduce to a single representative instead of flooding the list.
Use query-strings-only mode to drop static URLs and pass on only the endpoints that have parameters worth testing.
Turn on no-extensions to remove .png, .js, .woff, and similar files so testing focuses on dynamic endpoints.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls-file | FILE | -u | File containing URLs to deduplicate. |
| mode | STRING | -m | Comma-separated filters to enable (e.g. r, s, qs, ne). |
| remove-similar-urls | BOOLEAN | -s | Remove similar URLs based on integers and image/font files. |
| query-strings-only | BOOLEAN | -qs | Only include URLs that have query strings. |
| no-extensions | BOOLEAN | -ne | Exclude URLs that have a file extension (.png, .js, .html, and others). |
| regex-parse | BOOLEAN | -r | Use slower regex parsing for more thorough, accurate results. |
Showing key inputs. urldedupe exposes 6 inputs in total.
example
# collapse a crawl dump to unique parameterized endpointsurldedupe -u urls.txt -s -qs > unique.txthttps://example.com/search?q=loginhttps://example.com/api/v2/user?id=1https://example.com/catalog?category=books&sort=pricehttps://example.com/account/settings?tab=securityhttps://app.example.com/reports?range=30d&format=jsonhttp://198.51.100.24/index.php?page=dashboardhttps://example.com/download?doc=invoicehttps://203.0.113.10/api/orders?status=openguidance
Use urldedupe to thin a noisy URL list to unique path and query shapes before an expensive fuzz or scan stage. It collapses whole URLs, including near-duplicates. To extract a single URL part, use unfurl; to append only new lines across runs, use anew.
Appends only previously unseen lines. urldedupe collapses near-duplicate URLs within one list.
Extracts a chosen URL component. urldedupe deduplicates whole URLs by shape.
Rewrites query values to a fixed token. Pair it with urldedupe to normalize then dedupe.
faq
related
Spider a URL and return a wordlist for password crackers.
Wordlist and mined-word subdomain permutation.
Import, export, and upsert workflow data against an Airtable base.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Authenticated Route53 hosted-zone inventory.
A URL list feeds urldedupe, which collapses near-duplicates to unique shapes before ffuf fuzzes the survivors and writes the findings as a queryable output.
Facts on this page come from the live Trickest tool library.