Introducing Agents.·Read
OverviewAgentsWorkflowsExecutionDeveloper ToolsSecurityLibrary
Updated Jul 15, 2026

Utilities

Fetch and mirror files over HTTP and FTP

Non-interactive downloader for HTTP, HTTPS, FTP, and FTPS transfers.

Agent
Fetch inputs into a workflowMirror a site for offline analysisArchive exactly what was transferred

overview

What wget does

wget is the non-interactive downloader operators drop into scripts when a transfer must run unattended. It fetches over HTTP, HTTPS, FTP, and FTPS, follows redirects, resumes partial files, and can recurse into a site tree for offline copies.

Controls that matter in pipelines: --mirror or --recursive with --level, --input-file for URL lists, -U for a custom user-agent, --limit-rate to throttle, and --warc-file to archive request and response bytes.

Trickest runs wget as a managed utilities step. Pass a URL or --input-file list; the node writes a folder and a file. Use it to pull wordlists and configs, or to mirror a site before a parser or scanner.

source github.com/jay/wget

use cases

Where wget fits

Fetch inputs into a workflow

Pull a wordlist, config, or target file from a URL so later nodes have the data they need without a manual upload.

Mirror a site for offline analysis

Use recursive mirroring with link conversion to capture a whole site tree that downstream tools can grep and parse.

Archive exactly what was transferred

Write a WARC file to preserve the full request and response of every download for evidence or replay.

Throttle and retry on flaky hosts

Set a rate limit, wait, and retry count so large or unreliable transfers complete without hammering the server.

reference

wget inputs and flags

144 inputs
NameTypeFlagDescription
urlSTRING·Specify the URL.
urlsFILE--input-fileDownload URLs found in a local or external file.
recursiveBOOLEAN--recursiveSpecify recursive download.
mirrorBOOLEAN--mirrorShortcut for -N -r -l inf --no-remove-listing.
levelSTRING--levelMaximum recursion depth (inf or 0 for infinite).
limit-rateSTRING--limit-rateLimit download rate to RATE.
user-agentSTRING-UCustom user-agent.
warc-fileFILE--warc-fileSave request/response data to a .warc.gz file.

Showing key inputs. wget exposes 144 inputs in total.

Full flag reference (144 inputs)
NameTypeFlagDescription
urlSTRING·Specify the url.
urlsFILE--input-fileDownload URLs found in local or external FILE
userSTRING--userFTP, HTTP user
waitSTRING--waitWait SECONDS between retrievals(applies if more then 1 URL is to be retrieved)
debugBOOLEAN--debugPrint lots of debugging information
levelSTRING--levelMaximum recursion depth (inf or 0 for infinite)
quietBOOLEAN--quietQuiet mode (no output)
quotaSTRING--quotaSet retrieval quota to NUMBER
xattrBOOLEAN--xattrTurn on storage of metadata in extended file attributes
headerSTRING--headerInsert STRING among the headers
mirrorBOOLEAN--mirrorShortcut for -N -r -l inf --no-remove-listing
no-iriBOOLEAN--no-iriTurn off iri support
spiderBOOLEAN--spiderDon't download anything
unlinkBOOLEAN--unlinkRemove file before clobber
ciphersSTRING--ciphersSet the priority string (GnuTLS) or cipher list string (OpenSSL) directly.
no-globBOOLEAN--no-globTurn off FTP file name globbing
no-hstsBOOLEAN--no-hstsDisable HSTS
refererSTRING--refererInclude 'Referer: URL' header in HTTP request
retriesSTRING--triesSet number of retries to NUMBER (0 unlimits)
timeoutSTRING--timeoutSet all timeout values to SECONDS
verboseBOOLEAN--verboseBe verbose (this is the default)
crl-fileFILE--crl-fileFile with bundle of CRLs
cut-dirsSTRING--cut-dirsIgnore NUMBER remote directory components
ftp-userSTRING--ftp-userFTP Username
no-cacheBOOLEAN--no-cacheDisallow server-cached data
no-netrcBOOLEAN--no-netrcDon't try to obtain credentials from .netrc
no-proxyBOOLEAN--no-proxyExplicitly turn off proxy
passwordSTRING--passwordFTP, HTTP password
progressSTRING--progressSelect progress gauge type
relativeBOOLEAN--relativeFollow relative links only
warc-cdxBOOLEAN--warc-cdxWrite CDX index files
body-dataSTRING--body-dataSend STRING as data. --method MUST be set
body-fileFILE--body-fileSend contents of FILE. --method MUST be set
hsts-fileFILE--hsts-filePath of HSTS database (will override default)
http-userSTRING--http-userHTTP User
no-configBOOLEAN--no-configDo not read any config file
no-parentBOOLEAN--no-parentDon't ascend to the parent directory
post-dataSTRING--post-dataPost data
post-fileSTRING--post-fileUse the POST method; send contents of FILE
recursiveBOOLEAN--recursiveSpecify recursive download
start-posSTRING--start-posStart downloading from zero-based position OFFSET
waitretrySTRING--waitretrywait 1..SECONDS between retries of a retrieval
warc-fileFILE--warc-fileSave request/response data to a .warc.gz file
follow-ftpBOOLEAN--follow-ftpFollow FTP links from HTML documents
https-onlyBOOLEAN--https-onlyOnly follow secure HTTPS links
inet4-onlyBOOLEAN--inet4-onlyConnect only to IPv4 addresses
inet6-onlyBOOLEAN--inet6-onlyConnect only to IPv6 addresses
limit-rateSTRING--limit-rateLimit download rate to RATE
no-cookiesBOOLEAN--no-cookiesDon't use cookies
no-verboseBOOLEAN--no-verboseTurn off verboseness, without being quiet
proxy-userSTRING--proxy-userProxy user
regex-typeSTRING--regex-typeRegex type
span-hostsBOOLEAN--span-hostsGo to foreign hosts when recursive
user-agentSTRING-UCustom user-agent
warc-dedupFILE--warc-dedupDo not store records listed in this CDX file
certificateFILE--certificateClient certificate file
compressionSTRING--compressionChoose compression, one of auto, gzip and none. (default: none)
config-fileFILE--configSpecify config file
dns-timeoutSTRING--dns-timeoutSet the DNS lookup timeout to SECS
execute-CMDSTRING--executeExecute a `.wgetrc'-style command
http-methodSTRING--methodHTTP Method
ignore-caseSTRING--ignore-caseIgnore case when matching files/directories
ignore-tagsSTRING--ignore-tagsComma-separated list of ignored HTML tags
private-keyFILE--private-keyPrivate key file
random-waitBOOLEAN--random-waitwait from 0.5*WAIT...1.5*WAIT secs between retrievals
reject-listFILE--rejectComma-separated list of rejected extensions
use-askpassSTRING--use-askpassSpecify credential handler for requesting username and password
warc-headerSTRING--warc-headerInsert STRING into the warcinfo record
accept-regexSTRING--accept-regexRegex accepted urls
bind-addressSTRING--bind-addressBind to ADDRESS (hostname or IP) on local host
ca-directoryFOLDER--ca-directoryDirectory where hash list of CAs is stored
default-pageSTRING--default-pagehange the default page name (normally this is index.html)
ftp-passwordSTRING--ftp-passwordFTP Password
max-redirectSTRING--max-redirectMaximum redirections allowed per page
no-dns-cacheBOOLEAN--no-dns-cacheDisable caching DNS lookups
pinnedpubkeyFILE--pinnedpubkeyPublic key (PEM/DER) file
read-timeoutSTRING--read-timeoutset the read timeout to SECS
reject-regexSTRING--reject-regexRegex rejected urls
rejected-logBOOLEAN--rejected-logLog reasons for URL rejection to FILE
report-speedSTRING--report-speedOutput bandwidth as TYPE. TYPE can be bits
save-cookiesBOOLEAN--save-cookiesSave cookies to FILE after session
save-headersBOOLEAN--save-headersSve the HTTP headers to file
timestampingBOOLEAN--timestampingDon't re-retrieve files unless newer than local
warc-tempdirFOLDER--warc-tempdirLocation for temporary files created by the WARC writer
append-outputFILE--append-outputAppend messages to FILE
convert-linksBOOLEAN--convert-linksmake links in downloaded HTML or CSS point to local files
ftps-implicitSTRING--ftps-implicitUse implicit FTPS (default port is 990)
http-passwordSTRING--http-passwordHTTP Password
ignore-lengthBOOLEAN--ignore-lengthIgnore 'Content-Length' header field
prefer-familySTRING--prefer-familyConnect first to addresses of specified family, one of IPv6, IPv4, or none
retr-symlinksBOOLEAN--retr-symlinksWhen recursing, get linked-to files (not dir)
show-progressBOOLEAN--show-progressDisplay the progress bar in any verbosity mode
skip-existingBOOLEAN--no-clobberSkip downloads that would download to
warc-max-sizeSTRING--warc-max-sizeSet maximum size of WARC files to NUMBER
ca-certificateFILE--ca-certificateFile with the bundle of CAs
local-encodingSTRING--local-encodingUse ENC as the local encoding for IRIs
no-directoriesBOOLEAN--no-directoriesDon't create directories
no-passive-ftpBOOLEAN--no-passive-ftpDisable the "passive" transfer mode
proxy-passwordSTRING--proxy-passwordProxy pass
connect-timeoutSTRING--connect-timeoutSet the connect timeout to SECS
exclude-domainsFILE--exclude-domainsComma-separated list of rejected domains
force-html-urlsBOOLEAN--force-htmlTreat input file as HTML
ftps-resume-sslBOOLEAN--ftps-resume-sslResume the SSL/TLS session started in the control connection when opening a data connection
no-warc-digestsBOOLEAN--no-warc-digestsDo not calculate SHA1 digests
page-requisitesBOOLEAN--page-requisitesGet all images, etc. needed to display HTML page
remote-encodingSTRING--remote-encodingUse ENC as the default remote encoding
secure-protocolSTRING--secure-protocolChoose secure protocol, one of auto, SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS
server-responseBOOLEAN--server-responsePrint server response
strict-commentsBOOLEAN--strict-commentsTurn on strict (SGML) handling of HTML comments
adjust-extensionBOOLEAN--adjust-extensionSave HTML/CSS documents with proper extensions
certificate-typeSTRING--certificate-typeClient certificate type, PEM or DER
content-on-errorBOOLEAN--content-on-errorOutput the received content on server errors
directory-prefixSTRING--directory-prefixSave files to PREFIX/..
domains-acceptedFILE--domainsComma-separated list of accepted domains
follow-tags-listSTRING--follow-tagsComma-separated list of followed HTML tags
no-warc-keep-logBOOLEAN--no-warc-keep-logDo not store the log file in a WARC record
private-key-typeSTRING--private-key-typePrivate key type, PEM or DER
random-file-dataFILE--random-fileFile with random data for seeding the SSL PRNG
auth-no-challengeBOOLEAN--auth-no-challengeSend Basic HTTP authentication information without first waiting for the server's challenge
convert-file-onlyBOOLEAN--convert-file-onlyConvert the file part of the URLs only (usually known as the basename)
force-directoriesBOOLEAN--force-directoriesForce creation of directories
load-cookies-fileFILE--load-cookiesLoad cookies from FILE before session
no-remove-listingBOOLEAN--no-remove-listingDon't remove '.listing' files
retry-connrefusedBOOLEAN--retry-connrefusedRetry even if connection is refused
no-http-keep-aliveBOOLEAN--no-http-keep-aliveDisable HTTP keep-alive (persistent connections)
resolve-html-inputSTRING--baseResolves HTML input-file links (-i -F) relative to URL
trust-server-namesBOOLEAN--trust-server-namesUse the name specified by the redirection URL's last component
content-dispositionBOOLEAN--content-dispositionHonor the Content-Disposition header when choosing local file names
exclude-directoriesFILE--exclude-directoriesList of excluded directories
include-directoriesFILE--include-directoriesList of allowed directories
no-host-directoriesBOOLEAN--no-host-directoriesDon't create host directories
no-warc-compressionBOOLEAN--no-warc-compressionDo not compress WARC files with GZIP
restrict-file-namesSTRING--restrict-file-namesRestrict chars in file names to ones OS allows
retry-on-http-errorSTRING--retry-on-http-errorComma-separated list of HTTP errors to retry
ftps-fallback-to-ftpBOOLEAN--ftps-fallback-to-ftpFall back to FTP if FTPS is not supported in the target server
keep-session-cookiesBOOLEAN--keep-session-cookiesLoad and save session (non-permanent) cookies
no-check-certificateBOOLEAN--no-check-certificateDon't validate the server's certificate
no-if-modified-sinceBOOLEAN--no-if-modified-sinceDon't use conditional if-modified-since get requests in timestamping mode
preserve-permissionsBOOLEAN--preserve-permissionsPreserve remote file permissions
protocol-directoriesBOOLEAN--protocol-directoriesUse protocol name in directories
accept-list-extensionsFILE--acceptComma-separated list of accepted extensions
no-use-server-timestampsBOOLEAN--no-use-server-timestampsDo-no-use-server-timestamps the one on the server
ftps-clear-data-connectionBOOLEAN--ftps-clear-data-connectionCipher the control channel only; all the data will be in plaintext
continue-partially-downloaded-fileBOOLEAN--continueResume getting a partially-downloaded file

example

Run wget

wget · command
# mirror example.com with a custom UA and rate limit; write a WARC archivewget --mirror --level=2 -U "Trickest-wget" --limit-rate=500k --warc-file=example-mirror https://www.example.com/
sample output
--2026-06-22 12:01:04--  https://www.example.com/Resolving www.example.com... 198.51.100.10Connecting to www.example.com|198.51.100.10|:443... connected.HTTP request sent, awaiting response... 200 OKLength: 1256 (1.2K) [text/html]Saving to: 'www.example.com/index.html'--2026-06-22 12:01:05--  https://www.example.com/aboutSaving to: 'www.example.com/about'FINISHED --2026-06-22 12:01:08--Downloaded: 2 files, 3.1K in 0.4s

guidance

Choosing wget

Use wget when you need to download or mirror content into a workflow over HTTP or FTP. It retrieves files; it does not test them. Pair with a parser or scanner downstream. For a single request with precise header control, curl is the lighter sibling.

curl

Single-request transfer tool with precise header control. wget adds recursion and site mirroring.

httrack

Dedicated website copier with a richer mirroring UI. wget is the lighter, scriptable option.

aria2

Multi-connection, multi-protocol downloader tuned for large-file throughput.

faq

wget questions

HTTP, HTTPS, FTP, and FTPS, with redirect following, resume, and authentication for each.

related

More Utilities tools

  1. cewl

    Spider a URL and return a wordlist for password crackers.

  2. dnsgen

    Wordlist and mined-word subdomain permutation.

  3. pup

    CSS selectors over HTML, the jq counterpart for markup.

  4. unfurl

    Extract chosen URL parts from stdin into clean line lists.

  5. whisper

    Speech to txt, vtt, srt, and json from an audio file or folder.

  6. youtube-transcript

    Public caption tracks to transcript.txt and results.jsonl.

Run wget yourself

A URL feeds Wget, which retrieves the content and writes the downloaded files as a queryable output for later nodes.

Run wget in TrickestBrowse the library

Facts on this page come from the live Trickest tool library.