Details

Category: Utilities

Publisher: trickest

Created Date: 6/23/2021

Container: quay.io/trickest/urldedupe:cc9b25a

Source URL: https://github.com/ameenmaali/urldedupe

Parameters

​
mode
string
Command: -m - The mode/filters to be enabled (can be 1 or more, comma separated). Default is none, available options are the other flags (--mode r,s,qs,ne)
​
urls-file
file
required
Command: -u - File containing urls
​
regex-parse
boolean
Command: -r - This is significantly slower than normal parsing, but may be more thorough or accurate
​
no-extensions
boolean
Command: -ne - Do not include URLs if they have an extension (i.e. .png, .jpg, .woff, .js, .html)
​
query-strings-only
boolean
Command: -qs - Only include URLs if they have query strings
​
remove-similar-urls
boolean
Command: -s - Remove similar URLs (based on integers and image/font files) - i.e. /api/user/1 & /api/user/2 deduplicated