Introducing Agents.·Read
OverviewAgentsWorkflowsExecutionDeveloper ToolsSecurityLibrary
Updated Jul 14, 2026

Utilities

Collapse a URL list to unique path and query shapes

Deduplicate URLs by path and query-string shape.

C++ · MIT license

Agent
Shrink a crawl dump before fuzzingCollapse parameterized routesKeep only URLs with parameters

overview

What urldedupe does

urldedupe reads a URL list and keeps one representative per path and query-string shape, not per byte-identical line. Crawl and archive dumps often repeat the same endpoint with different parameter values; this collapses that noise.

Modes stack. -s collapses similar routes that differ by integers or asset files. -qs keeps only parameterized URLs. -ne drops extensioned assets. -r switches to a regex parser when messy URLs need it.

Trickest provides urldedupe as a managed Utilities node. Feed urls-file after a crawler or archive pull; take FILE and FOLDER into a fuzzer or parameter scanner.

source github.com/ameenmaali/urldedupe

use cases

Where urldedupe fits

Shrink a crawl dump before fuzzing

Collapse thousands of near-identical URLs to one per shape so a fuzzer or scanner spends its budget on distinct endpoints.

Collapse parameterized routes

Enable similar-URL removal so /api/user/1 and /api/user/2 reduce to a single representative instead of flooding the list.

Keep only URLs with parameters

Use query-strings-only mode to drop static URLs and pass on only the endpoints that have parameters worth testing.

Drop static asset URLs

Turn on no-extensions to remove .png, .js, .woff, and similar files so testing focuses on dynamic endpoints.

reference

urldedupe inputs and flags

6 inputs
NameTypeFlagDescription
urls-fileFILE-uFile containing URLs to deduplicate.
modeSTRING-mComma-separated filters to enable (e.g. r, s, qs, ne).
remove-similar-urlsBOOLEAN-sRemove similar URLs based on integers and image/font files.
query-strings-onlyBOOLEAN-qsOnly include URLs that have query strings.
no-extensionsBOOLEAN-neExclude URLs that have a file extension (.png, .js, .html, and others).
regex-parseBOOLEAN-rUse slower regex parsing for more thorough, accurate results.

Showing key inputs. urldedupe exposes 6 inputs in total.

example

Run urldedupe

urldedupe · command
# collapse a crawl dump to unique parameterized endpoints
urldedupe -u urls.txt -s -qs > unique.txt
sample output
https://example.com/search?q=login
https://example.com/api/v2/user?id=1
https://example.com/catalog?category=books&sort=price
https://example.com/account/settings?tab=security
https://app.example.com/reports?range=30d&format=json
http://198.51.100.24/index.php?page=dashboard
https://example.com/download?doc=invoice
https://203.0.113.10/api/orders?status=open

guidance

Choosing urldedupe

Use urldedupe to thin a noisy URL list to unique path and query shapes before an expensive fuzz or scan stage. It collapses whole URLs, including near-duplicates. To extract a single URL part, use unfurl; to append only new lines across runs, use anew.

anew

Appends only previously unseen lines. urldedupe collapses near-duplicate URLs within one list.

unfurl

Extracts a chosen URL component. urldedupe deduplicates whole URLs by shape.

qsreplace

Rewrites query values to a fixed token. Pair it with urldedupe to normalize then dedupe.

workflows

Workflows using urldedupe

2 workflows

  1. Find Open Redirects on a Domain

    Mine a domain's archived URLs for redirect parameters, then confirm which ones send a browser to an attacker-controlled host.

  2. Find Reflected XSS on a Domain

    Collect a domain's archived URLs, keep parameters that reflect input, and fuzz each one for reflected and DOM cross-site scripting.

faq

urldedupe questions

By default urldedupe treats two URLs as duplicates when they share the same path and the same set of query-string keys, so it keeps one representative per endpoint shape. Add -s to also collapse URLs that differ only by integers or asset files, so /api/user/1 and /api/user/2 reduce to a single entry.

sort | uniq only removes byte-identical lines, so /product?id=1 and /product?id=2 both survive. urldedupe compares path and query-string shape, and with -s it recognizes those two as the same endpoint and keeps one.

Yes. Set -qs (query-strings-only) to drop every static URL and pass on only endpoints that carry a query string. Combine it with -s to also collapse near-duplicate parameterized routes.

Turn on -r when accuracy matters more than throughput. The regex parser is slower than the default, but it can recognize duplicate shapes in messy or unusual URLs more thoroughly. For large lists where speed wins, leave it off.

urldedupe writes unique URLs to stdout or a file; pass that file to ffuf as its input list. Deduplicating first means the fuzzer spends its budget on distinct endpoints instead of hundreds of copies of the same route.

After a crawler or archive tool and before a fuzzer or parameter scanner, so the expensive stage runs against a tight, deduplicated set.

related

More Utilities tools

  1. browser-fetch

    Headless Chromium render: requested URL, final URL, status, and title next to the page.

  2. cewl

    Spider a URL and return a wordlist for password crackers.

  3. csvkit

    SQL on one CSV. The table name is the file stem.

  4. dnsgen

    Wordlist and mined-word subdomain permutation.

  5. duckdb

    In-process SQL on CSV, JSON, and Parquet. No server.

  6. edge-tts

    Microsoft Edge neural speech from text or a script file. No API key.

Run urldedupe yourself

A URL list feeds urldedupe, which collapses near-duplicates to unique shapes before ffuf fuzzes the survivors and writes the findings as a queryable output.

Open urldedupe in the libraryView the workflow

Facts on this page come from the live Trickest tool library.