Introducing Agents.·Read
OverviewAgentsWorkflowsExecutionDeveloper ToolsSecurityLibrary
Updated Jul 14, 2026

Utilities

Collapse a URL list to unique path and query shapes

Deduplicate URLs by path and query-string shape.

Agent
Shrink a crawl dump before fuzzingCollapse parameterized routesKeep only URLs with parameters

overview

What urldedupe does

urldedupe reads a URL list and keeps one representative per path and query-string shape, not per byte-identical line. Crawl and archive dumps often repeat the same endpoint with different parameter values; this collapses that noise.

Modes stack. -s collapses similar routes that differ by integers or asset files. -qs keeps only parameterized URLs. -ne drops extensioned assets. -r switches to a regex parser when messy URLs need it.

Trickest provides urldedupe as a managed Utilities node. Feed urls-file after a crawler or archive pull; take FILE and FOLDER into a fuzzer or parameter scanner.

source github.com/ameenmaali/urldedupe

use cases

Where urldedupe fits

Shrink a crawl dump before fuzzing

Collapse thousands of near-identical URLs to one per shape so a fuzzer or scanner spends its budget on distinct endpoints.

Collapse parameterized routes

Enable similar-URL removal so /api/user/1 and /api/user/2 reduce to a single representative instead of flooding the list.

Keep only URLs with parameters

Use query-strings-only mode to drop static URLs and pass on only the endpoints that have parameters worth testing.

Drop static asset URLs

Turn on no-extensions to remove .png, .js, .woff, and similar files so testing focuses on dynamic endpoints.

reference

urldedupe inputs and flags

6 inputs
NameTypeFlagDescription
urls-fileFILE-uFile containing URLs to deduplicate.
modeSTRING-mComma-separated filters to enable (e.g. r, s, qs, ne).
remove-similar-urlsBOOLEAN-sRemove similar URLs based on integers and image/font files.
query-strings-onlyBOOLEAN-qsOnly include URLs that have query strings.
no-extensionsBOOLEAN-neExclude URLs that have a file extension (.png, .js, .html, and others).
regex-parseBOOLEAN-rUse slower regex parsing for more thorough, accurate results.

Showing key inputs. urldedupe exposes 6 inputs in total.

example

Run urldedupe

urldedupe · command
# collapse a crawl dump to unique parameterized endpointsurldedupe -u urls.txt -s -qs > unique.txt
sample output
https://example.com/search?q=loginhttps://example.com/api/v2/user?id=1https://example.com/catalog?category=books&sort=pricehttps://example.com/account/settings?tab=securityhttps://app.example.com/reports?range=30d&format=jsonhttp://198.51.100.24/index.php?page=dashboardhttps://example.com/download?doc=invoicehttps://203.0.113.10/api/orders?status=open

guidance

Choosing urldedupe

Use urldedupe to thin a noisy URL list to unique path and query shapes before an expensive fuzz or scan stage. It collapses whole URLs, including near-duplicates. To extract a single URL part, use unfurl; to append only new lines across runs, use anew.

anew

Appends only previously unseen lines. urldedupe collapses near-duplicate URLs within one list.

unfurl

Extracts a chosen URL component. urldedupe deduplicates whole URLs by shape.

qsreplace

Rewrites query values to a fixed token. Pair it with urldedupe to normalize then dedupe.

workflows

Workflows using urldedupe

2 workflows

  1. Find Open Redirects on a Domain

    Mine a domain's archived URLs for redirect parameters, then confirm which ones send a browser to an attacker-controlled host.

  2. Find Reflected XSS on a Domain

    Collect a domain's archived URLs, keep parameters that reflect input, and fuzz each one for reflected and DOM cross-site scripting.

faq

urldedupe questions

By default urldedupe treats two URLs as duplicates when they share the same path and the same set of query-string keys, so it keeps one representative per endpoint shape. Add -s to also collapse URLs that differ only by integers or asset files, so /api/user/1 and /api/user/2 reduce to a single entry.

related

More Utilities tools

  1. cewl

    Spider a URL and return a wordlist for password crackers.

  2. dnsgen

    Wordlist and mined-word subdomain permutation.

  3. pup

    CSS selectors over HTML, the jq counterpart for markup.

  4. unfurl

    Extract chosen URL parts from stdin into clean line lists.

  5. whisper

    Speech to txt, vtt, srt, and json from an audio file or folder.

  6. youtube-transcript

    Public caption tracks to transcript.txt and results.jsonl.

Run urldedupe yourself

A URL list feeds urldedupe, which collapses near-duplicates to unique shapes before ffuf fuzzes the survivors and writes the findings as a queryable output.

Open urldedupe in the libraryView the workflow

Facts on this page come from the live Trickest tool library.