urldedupe
urldedupe is a tool to quickly pass in a list of URLs, and get back a list of deduplicated (unique) URL and query string combination. This is useful to ensure you don’t have a URL list will hundreds of duplicated parameters with differing qs values.
Name:urldedupe
Category:Utilities
Publisher:trickest
Created:6/23/2021
Container:
quay.io/trickest/urldedupe:cc9b25aOutput Type:
License:Unknown
Source:View Source
Parameters
-mThe mode/filters to be enabled (can be 1 or more, comma separated). Default is none, available options are the other flags (--mode r,s,qs,ne)-uFile containing urls-rThis is significantly slower than normal parsing, but may be more thorough or accurate-neDo not include URLs if they have an extension (i.e. .png, .jpg, .woff, .js, .html)-qsOnly include URLs if they have query strings-sRemove similar URLs (based on integers and image/font files) - i.e. /api/user/1 & /api/user/2 deduplicated