How is urldedupe different from sort and uniq?

sort | uniq only removes byte-identical lines, so /product?id=1 and /product?id=2 both survive. urldedupe compares the path and query-string shape, and with -s it recognizes those two as the same endpoint and keeps one. The result is a list of distinct endpoints instead of distinct strings.

Can I keep only parameterized URLs?

Yes. Set -qs (query-strings-only) to drop every static URL and pass on only endpoints that carry a query string, which are usually the ones worth fuzzing. Combine it with -s to also collapse near-duplicate parameterized routes.

How do I feed a crawler's output into urldedupe?

Point urldedupe at the file a crawler or archive tool wrote with -u, or pipe it in on stdin, for example `gau example.com | urldedupe -s`. In a Trickest workflow, wire a katana, gau, or waybackurls node into the urldedupe node's URLs input.

When should I use regex parsing?

Turn on -r when accuracy matters more than speed. The regex parser is slower than the default, but it can be more thorough at recognizing duplicate shapes in messy or unusual URLs. For large lists where speed wins, leave it off.

How do I pass urldedupe output to a fuzzer like ffuf?

urldedupe writes the unique URLs to stdout or a file, so hand that file to ffuf or another scanner as its input list. Deduplicating first means the fuzzer spends its request budget on distinct endpoints instead of hundreds of copies of the same route with different parameter values.

Where does urldedupe fit in a workflow?

After a crawler or archive tool (katana, gau, waybackurls) and before a fuzzer or parameter scanner, so the expensive testing stage runs against a tight, deduplicated set instead of a raw crawl full of repeats.

Updated Jul 14, 2026

Utilities

Deduplicate a URL list to unique path and query shapes

Quickly deduplicate a list of URLs by path and query-string shape.

Shrink a crawl dump before fuzzing Collapse parameterized routes Keep only URLs with parameters

overview

What urldedupe does

urldedupe reads a list of URLs and returns the unique ones, where unique means a distinct path and query-string combination rather than a byte-identical string. A crawl or archive pull typically produces the same endpoint hundreds of times with different parameter values; urldedupe collapses those down to one representative per shape so the next stage works on real variety, not noise.

Its smarter modes are what set it apart from a plain sort and uniq. With similar-URL removal, /api/user/1 and /api/user/2 are recognized as the same shape and reduced to one. You can keep only URLs that carry query strings, drop URLs with file extensions like .png or .js, and switch on a slower regex parser when accuracy matters more than speed. The modes combine, so you tune the dedup to the pipeline.

On Trickest, urldedupe is a Utilities node that reads a URLs file and writes a file and a folder. Slot it after a crawler or archive tool and before a fuzzer or parameter scanner so those expensive stages run against a tight, deduplicated set instead of a list full of repeats.

source github.com/ameenmaali/urldedupe

use cases

Where urldedupe fits

Shrink a crawl dump before fuzzing

Collapse thousands of near-identical URLs to one per shape so a fuzzer or scanner spends its budget on distinct endpoints.

Collapse parameterized routes

Enable similar-URL removal so /api/user/1 and /api/user/2 reduce to a single representative instead of flooding the list.

Keep only URLs with parameters

Use query-strings-only mode to drop static URLs and pass on only the endpoints that have parameters worth testing.

Drop static asset URLs

Turn on no-extensions to remove .png, .js, .woff, and similar files so testing focuses on dynamic endpoints.

reference

urldedupe inputs and flags

6 inputs

Name	Type	Flag	Description
urls-file	FILE	-u	File containing URLs to deduplicate.
mode	STRING	-m	Comma-separated filters to enable (e.g. r, s, qs, ne).
remove-similar-urls	BOOLEAN	-s	Remove similar URLs based on integers and image/font files.
query-strings-only	BOOLEAN	-qs	Only include URLs that have query strings.
no-extensions	BOOLEAN	-ne	Exclude URLs that have a file extension (.png, .js, .html, and others).
regex-parse	BOOLEAN	-r	Use slower regex parsing for more thorough, accurate results.

Showing key inputs. urldedupe exposes 6 inputs in total.

example

Run urldedupe

urldedupe · command

# collapse a crawl dump to unique parameterized endpointsurldedupe -u urls.txt -s -qs > unique.txt

sample output

https://example.com/search?q=loginhttps://example.com/api/v2/user?id=1https://example.com/catalog?category=books&sort=pricehttps://example.com/account/settings?tab=securityhttps://app.example.com/reports?range=30d&format=jsonhttp://198.51.100.24/index.php?page=dashboardhttps://example.com/download?doc=invoicehttps://203.0.113.10/api/orders?status=open

guidance

Choosing urldedupe

Use urldedupe to thin a noisy URL list down to unique path and query shapes before an expensive fuzzing or scanning stage. It deduplicates whole URLs, including near-duplicates. To extract a single part of a URL instead, use unfurl; to append only new lines across runs, use anew.

anew

Appends only previously unseen lines. urldedupe collapses near-duplicate URLs within one list.

unfurl

Extracts a chosen URL component. urldedupe deduplicates whole URLs by shape.

qsreplace

Rewrites query values to a fixed token. Pair it with urldedupe to normalize then dedupe.

faq

urldedupe questions

By default urldedupe treats two URLs as duplicates when they share the same path and the same set of query-string keys, so it keeps one representative per endpoint shape rather than one per literal string. Add -s to also collapse URLs that differ only by integers or asset files, so /api/user/1 and /api/user/2 reduce to a single entry.

Run urldedupe yourself

A URL list feeds urldedupe, which collapses near-duplicates to unique shapes before ffuf fuzzes the survivors and writes the findings as a queryable output.

Run urldedupe in Trickest View the workflow

Facts on this page come from the live Trickest tool library.

Name

Type

Flag

Description

urls-file

FILE

-u

File containing URLs to deduplicate.

mode

STRING

-m

Comma-separated filters to enable (e.g. r, s, qs, ne).

remove-similar-urls

BOOLEAN

-s

Remove similar URLs based on integers and image/font files.

query-strings-only

BOOLEAN

-qs

Only include URLs that have query strings.

no-extensions

BOOLEAN

-ne

Exclude URLs that have a file extension (.png, .js, .html, and others).

regex-parse

BOOLEAN

-r

Use slower regex parsing for more thorough, accurate results.

https://example.com/search?q=loginhttps://example.com/api/v2/user?id=1https://example.com/catalog?category=books&sort=pricehttps://example.com/account/settings?tab=securityhttps://app.example.com/reports?range=30d&format=jsonhttp://198.51.100.24/index.php?page=dashboardhttps://example.com/download?doc=invoicehttps://203.0.113.10/api/orders?status=open

Deduplicate a URL list to unique path and query shapes

What urldedupe does

Where urldedupe fits

Shrink a crawl dump before fuzzing

Collapse parameterized routes

Keep only URLs with parameters

Drop static asset URLs

urldedupe inputs and flags

Run urldedupe

Choosing urldedupe

anew

unfurl

qsreplace

urldedupe questions

More Utilities tools

airtable-integration

apktool-decode

assert-tool

aws-route53-zones

batch-output

batch-output-v2

Run urldedupe yourself

Deduplicate a URL list to unique path and query shapes

What urldedupe does

Where urldedupe fits

Shrink a crawl dump before fuzzing

Collapse parameterized routes

Keep only URLs with parameters

Drop static asset URLs

urldedupe inputs and flags

Run urldedupe

Choosing urldedupe

anew

unfurl

qsreplace

urldedupe questions

More Utilities tools

airtable-integration

apktool-decode

assert-tool

aws-route53-zones

batch-output

batch-output-v2

Run urldedupe yourself