Introducing Agents.·Read
OverviewAgentsWorkflowsExecutionDeveloper ToolsSecurityLibrary
Updated Jul 14, 2026

Utilities

Collapse a URL list to unique path and query shapes

Deduplicate URLs by path and query-string shape.

Agent
Shrink a crawl dump before fuzzingCollapse parameterized routesKeep only URLs with parameters

overview

What urldedupe does

urldedupe reads a URL list and keeps one representative per path and query-string shape, not per byte-identical line. Crawl and archive dumps often repeat the same endpoint with different parameter values; this collapses that noise.

Modes stack. -s collapses similar routes that differ by integers or asset files. -qs keeps only parameterized URLs. -ne drops extensioned assets. -r switches to a regex parser when messy URLs need it.

Trickest provides urldedupe as a managed Utilities node. Feed urls-file after a crawler or archive pull; take FILE and FOLDER into a fuzzer or parameter scanner.

source github.com/ameenmaali/urldedupe

use cases

Where urldedupe fits

Shrink a crawl dump before fuzzing

Collapse thousands of near-identical URLs to one per shape so a fuzzer or scanner spends its budget on distinct endpoints.

Collapse parameterized routes

Enable similar-URL removal so /api/user/1 and /api/user/2 reduce to a single representative instead of flooding the list.

Keep only URLs with parameters

Use query-strings-only mode to drop static URLs and pass on only the endpoints that have parameters worth testing.

Drop static asset URLs

Turn on no-extensions to remove .png, .js, .woff, and similar files so testing focuses on dynamic endpoints.

reference

urldedupe inputs and flags

6 inputs
NameTypeFlagDescription
urls-fileFILE-uFile containing URLs to deduplicate.
modeSTRING-mComma-separated filters to enable (e.g. r, s, qs, ne).
remove-similar-urlsBOOLEAN-sRemove similar URLs based on integers and image/font files.
query-strings-onlyBOOLEAN-qsOnly include URLs that have query strings.
no-extensionsBOOLEAN-neExclude URLs that have a file extension (.png, .js, .html, and others).
regex-parseBOOLEAN-rUse slower regex parsing for more thorough, accurate results.

Showing key inputs. urldedupe exposes 6 inputs in total.

example

Run urldedupe

urldedupe · command
# collapse a crawl dump to unique parameterized endpointsurldedupe -u urls.txt -s -qs > unique.txt
sample output
https://example.com/search?q=loginhttps://example.com/api/v2/user?id=1https://example.com/catalog?category=books&sort=pricehttps://example.com/account/settings?tab=securityhttps://app.example.com/reports?range=30d&format=jsonhttp://198.51.100.24/index.php?page=dashboardhttps://example.com/download?doc=invoicehttps://203.0.113.10/api/orders?status=open

guidance

Choosing urldedupe

Use urldedupe to thin a noisy URL list to unique path and query shapes before an expensive fuzz or scan stage. It collapses whole URLs, including near-duplicates. To extract a single URL part, use unfurl; to append only new lines across runs, use anew.

anew

Appends only previously unseen lines. urldedupe collapses near-duplicate URLs within one list.

unfurl

Extracts a chosen URL component. urldedupe deduplicates whole URLs by shape.

qsreplace

Rewrites query values to a fixed token. Pair it with urldedupe to normalize then dedupe.

workflows

Workflows using urldedupe

2 workflows

  1. Find Open Redirects on a Domain

    Mine a domain's archived URLs for redirect parameters, then confirm which ones send a browser to an attacker-controlled host.

  2. Find Reflected XSS on a Domain

    Collect a domain's archived URLs, keep parameters that reflect input, and fuzz each one for reflected and DOM cross-site scripting.

faq

urldedupe questions

By default urldedupe treats two URLs as duplicates when they share the same path and the same set of query-string keys, so it keeps one representative per endpoint shape. Add -s to also collapse URLs that differ only by integers or asset files, so /api/user/1 and /api/user/2 reduce to a single entry.

related

More Utilities tools

  1. browser-fetch

    Headless Chromium render: requested URL, final URL, status, and title next to the page.

  2. cewl

    Spider a URL and return a wordlist for password crackers.

  3. csvkit

    SQL on one CSV. The table name is the file stem.

  4. dnsgen

    Wordlist and mined-word subdomain permutation.

  5. duckdb

    In-process SQL on CSV, JSON, and Parquet. No server.

  6. edge-tts

    Microsoft Edge neural speech from text or a script file. No API key.

Run urldedupe yourself

A URL list feeds urldedupe, which collapses near-duplicates to unique shapes before ffuf fuzzes the survivors and writes the findings as a queryable output.

Open urldedupe in the libraryView the workflow

Facts on this page come from the live Trickest tool library.