loading
loading
Utilities
Retrieve files over HTTP, HTTPS, FTP, and FTPS, with recursive mirroring.
overview
wget is the standard non-interactive downloader. It fetches files over HTTP, HTTPS, FTP, and FTPS, follows redirects, resumes partial transfers, and runs unattended in scripts and pipelines. Where it goes beyond a single GET is recursion: point it at a site and it can mirror the whole tree, rewriting links to work offline.
Its breadth shows in the controls it exposes: rate limits, retries, custom headers and user agents, cookies, authentication, accept and reject patterns, and WARC capture for archiving exactly what was sent and received. That makes it as useful for pulling a single artifact as for capturing a site for later analysis.
On Trickest, wget is a Utilities node that takes a URL or a file of URLs and writes a folder and a file. Use it to fetch wordlists, configs, or target content into a workflow, or to mirror and archive a site for downstream processing.
source github.com/jay/wget
use cases
Pull a wordlist, config, or target file from a URL so later nodes have the data they need without a manual upload.
Use recursive mirroring with link conversion to capture a whole site tree that downstream tools can grep and parse.
Write a WARC file to preserve the full request and response of every download for evidence or replay.
Set a rate limit, wait, and retry count so large or unreliable transfers complete without hammering the server.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | · | Specify the URL. |
| urls | FILE | --input-file | Download URLs found in a local or external file. |
| recursive | BOOLEAN | --recursive | Specify recursive download. |
| mirror | BOOLEAN | --mirror | Shortcut for -N -r -l inf --no-remove-listing. |
| level | STRING | --level | Maximum recursion depth (inf or 0 for infinite). |
| limit-rate | STRING | --limit-rate | Limit download rate to RATE. |
| user-agent | STRING | -U | Custom user-agent. |
| warc-file | FILE | --warc-file | Save request/response data to a .warc.gz file. |
Showing key inputs. wget exposes 144 inputs in total.
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | · | Specify the url. |
| urls | FILE | --input-file | Download URLs found in local or external FILE |
| user | STRING | --user | FTP, HTTP user |
| wait | STRING | --wait | Wait SECONDS between retrievals(applies if more then 1 URL is to be retrieved) |
| debug | BOOLEAN | --debug | Print lots of debugging information |
| level | STRING | --level | Maximum recursion depth (inf or 0 for infinite) |
| quiet | BOOLEAN | --quiet | Quiet mode (no output) |
| quota | STRING | --quota | Set retrieval quota to NUMBER |
| xattr | BOOLEAN | --xattr | Turn on storage of metadata in extended file attributes |
| header | STRING | --header | Insert STRING among the headers |
| mirror | BOOLEAN | --mirror | Shortcut for -N -r -l inf --no-remove-listing |
| no-iri | BOOLEAN | --no-iri | Turn off iri support |
| spider | BOOLEAN | --spider | Don't download anything |
| unlink | BOOLEAN | --unlink | Remove file before clobber |
| ciphers | STRING | --ciphers | Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly. |
| no-glob | BOOLEAN | --no-glob | Turn off FTP file name globbing |
| no-hsts | BOOLEAN | --no-hsts | Disable HSTS |
| referer | STRING | --referer | Include 'Referer: URL' header in HTTP request |
| retries | STRING | --tries | Set number of retries to NUMBER (0 unlimits) |
| timeout | STRING | --timeout | Set all timeout values to SECONDS |
| verbose | BOOLEAN | --verbose | Be verbose (this is the default) |
| crl-file | FILE | --crl-file | File with bundle of CRLs |
| cut-dirs | STRING | --cut-dirs | Ignore NUMBER remote directory components |
| ftp-user | STRING | --ftp-user | FTP Username |
| no-cache | BOOLEAN | --no-cache | Disallow server-cached data |
| no-netrc | BOOLEAN | --no-netrc | Don't try to obtain credentials from .netrc |
| no-proxy | BOOLEAN | --no-proxy | Explicitly turn off proxy |
| password | STRING | --password | FTP, HTTP password |
| progress | STRING | --progress | Select progress gauge type |
| relative | BOOLEAN | --relative | Follow relative links only |
| warc-cdx | BOOLEAN | --warc-cdx | Write CDX index files |
| body-data | STRING | --body-data | Send STRING as data. --method MUST be set |
| body-file | FILE | --body-file | Send contents of FILE. --method MUST be set |
| hsts-file | FILE | --hsts-file | Path of HSTS database (will override default) |
| http-user | STRING | --http-user | HTTP User |
| no-config | BOOLEAN | --no-config | Do not read any config file |
| no-parent | BOOLEAN | --no-parent | Don't ascend to the parent directory |
| post-data | STRING | --post-data | Post data |
| post-file | STRING | --post-file | Use the POST method; send contents of FILE |
| recursive | BOOLEAN | --recursive | Specify recursive download |
| start-pos | STRING | --start-pos | Start downloading from zero-based position OFFSET |
| waitretry | STRING | --waitretry | wait 1..SECONDS between retries of a retrieval |
| warc-file | FILE | --warc-file | Save request/response data to a .warc.gz file |
| follow-ftp | BOOLEAN | --follow-ftp | Follow FTP links from HTML documents |
| https-only | BOOLEAN | --https-only | Only follow secure HTTPS links |
| inet4-only | BOOLEAN | --inet4-only | Connect only to IPv4 addresses |
| inet6-only | BOOLEAN | --inet6-only | Connect only to IPv6 addresses |
| limit-rate | STRING | --limit-rate | Limit download rate to RATE |
| no-cookies | BOOLEAN | --no-cookies | Don't use cookies |
| no-verbose | BOOLEAN | --no-verbose | Turn off verboseness, without being quiet |
| proxy-user | STRING | --proxy-user | Proxy user |
| regex-type | STRING | --regex-type | Regex type |
| span-hosts | BOOLEAN | --span-hosts | Go to foreign hosts when recursive |
| user-agent | STRING | -U | Custom user-agent |
| warc-dedup | FILE | --warc-dedup | Do not store records listed in this CDX file |
| certificate | FILE | --certificate | Client certificate file |
| compression | STRING | --compression | Choose compression, one of auto, gzip and none. (default: none) |
| config-file | FILE | --config | Specify config file |
| dns-timeout | STRING | --dns-timeout | Set the DNS lookup timeout to SECS |
| execute-CMD | STRING | --execute | Execute a `.wgetrc'-style command |
| http-method | STRING | --method | HTTP Method |
| ignore-case | STRING | --ignore-case | Ignore case when matching files/directories |
| ignore-tags | STRING | --ignore-tags | Comma-separated list of ignored HTML tags |
| private-key | FILE | --private-key | Private key file |
| random-wait | BOOLEAN | --random-wait | wait from 0.5*WAIT...1.5*WAIT secs between retrievals |
| reject-list | FILE | --reject | Comma-separated list of rejected extensions |
| use-askpass | STRING | --use-askpass | Specify credential handler for requesting username and password |
| warc-header | STRING | --warc-header | Insert STRING into the warcinfo record |
| accept-regex | STRING | --accept-regex | Regex accepted urls |
| bind-address | STRING | --bind-address | Bind to ADDRESS (hostname or IP) on local host |
| ca-directory | FOLDER | --ca-directory | Directory where hash list of CAs is stored |
| default-page | STRING | --default-page | hange the default page name (normally this is index.html) |
| ftp-password | STRING | --ftp-password | FTP Password |
| max-redirect | STRING | --max-redirect | Maximum redirections allowed per page |
| no-dns-cache | BOOLEAN | --no-dns-cache | Disable caching DNS lookups |
| pinnedpubkey | FILE | --pinnedpubkey | Public key (PEM/DER) file |
| read-timeout | STRING | --read-timeout | set the read timeout to SECS |
| reject-regex | STRING | --reject-regex | Regex rejected urls |
| rejected-log | BOOLEAN | --rejected-log | Log reasons for URL rejection to FILE |
| report-speed | STRING | --report-speed | Output bandwidth as TYPE. TYPE can be bits |
| save-cookies | BOOLEAN | --save-cookies | Save cookies to FILE after session |
| save-headers | BOOLEAN | --save-headers | Sve the HTTP headers to file |
| timestamping | BOOLEAN | --timestamping | Don't re-retrieve files unless newer than local |
| warc-tempdir | FOLDER | --warc-tempdir | Location for temporary files created by the WARC writer |
| append-output | FILE | --append-output | Append messages to FILE |
| convert-links | BOOLEAN | --convert-links | make links in downloaded HTML or CSS point to local files |
| ftps-implicit | STRING | --ftps-implicit | Use implicit FTPS (default port is 990) |
| http-password | STRING | --http-password | HTTP Password |
| ignore-length | BOOLEAN | --ignore-length | Ignore 'Content-Length' header field |
| prefer-family | STRING | --prefer-family | Connect first to addresses of specified family, one of IPv6, IPv4, or none |
| retr-symlinks | BOOLEAN | --retr-symlinks | When recursing, get linked-to files (not dir) |
| show-progress | BOOLEAN | --show-progress | Display the progress bar in any verbosity mode |
| skip-existing | BOOLEAN | --no-clobber | Skip downloads that would download to |
| warc-max-size | STRING | --warc-max-size | Set maximum size of WARC files to NUMBER |
| ca-certificate | FILE | --ca-certificate | File with the bundle of CAs |
| local-encoding | STRING | --local-encoding | Use ENC as the local encoding for IRIs |
| no-directories | BOOLEAN | --no-directories | Don't create directories |
| no-passive-ftp | BOOLEAN | --no-passive-ftp | Disable the "passive" transfer mode |
| proxy-password | STRING | --proxy-password | Proxy pass |
| connect-timeout | STRING | --connect-timeout | Set the connect timeout to SECS |
| exclude-domains | FILE | --exclude-domains | Comma-separated list of rejected domains |
| force-html-urls | BOOLEAN | --force-html | Treat input file as HTML |
| ftps-resume-ssl | BOOLEAN | --ftps-resume-ssl | Resume the SSL/TLS session started in the control connection when opening a data connection |
| no-warc-digests | BOOLEAN | --no-warc-digests | Do not calculate SHA1 digests |
| page-requisites | BOOLEAN | --page-requisites | Get all images, etc. needed to display HTML page |
| remote-encoding | STRING | --remote-encoding | Use ENC as the default remote encoding |
| secure-protocol | STRING | --secure-protocol | Choose secure protocol, one of auto, SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS |
| server-response | BOOLEAN | --server-response | Print server response |
| strict-comments | BOOLEAN | --strict-comments | Turn on strict (SGML) handling of HTML comments |
| adjust-extension | BOOLEAN | --adjust-extension | Save HTML/CSS documents with proper extensions |
| certificate-type | STRING | --certificate-type | Client certificate type, PEM or DER |
| content-on-error | BOOLEAN | --content-on-error | Output the received content on server errors |
| directory-prefix | STRING | --directory-prefix | Save files to PREFIX/.. |
| domains-accepted | FILE | --domains | Comma-separated list of accepted domains |
| follow-tags-list | STRING | --follow-tags | Comma-separated list of followed HTML tags |
| no-warc-keep-log | BOOLEAN | --no-warc-keep-log | Do not store the log file in a WARC record |
| private-key-type | STRING | --private-key-type | Private key type, PEM or DER |
| random-file-data | FILE | --random-file | File with random data for seeding the SSL PRNG |
| auth-no-challenge | BOOLEAN | --auth-no-challenge | Send Basic HTTP authentication information without first waiting for the server's challenge |
| convert-file-only | BOOLEAN | --convert-file-only | Convert the file part of the URLs only (usually known as the basename) |
| force-directories | BOOLEAN | --force-directories | Force creation of directories |
| load-cookies-file | FILE | --load-cookies | Load cookies from FILE before session |
| no-remove-listing | BOOLEAN | --no-remove-listing | Don't remove '.listing' files |
| retry-connrefused | BOOLEAN | --retry-connrefused | Retry even if connection is refused |
| no-http-keep-alive | BOOLEAN | --no-http-keep-alive | Disable HTTP keep-alive (persistent connections) |
| resolve-html-input | STRING | --base | Resolves HTML input-file links (-i -F) relative to URL |
| trust-server-names | BOOLEAN | --trust-server-names | Use the name specified by the redirection URL's last component |
| content-disposition | BOOLEAN | --content-disposition | Honor the Content-Disposition header when choosing local file names |
| exclude-directories | FILE | --exclude-directories | List of excluded directories |
| include-directories | FILE | --include-directories | List of allowed directories |
| no-host-directories | BOOLEAN | --no-host-directories | Don't create host directories |
| no-warc-compression | BOOLEAN | --no-warc-compression | Do not compress WARC files with GZIP |
| restrict-file-names | STRING | --restrict-file-names | Restrict chars in file names to ones OS allows |
| retry-on-http-error | STRING | --retry-on-http-error | Comma-separated list of HTTP errors to retry |
| ftps-fallback-to-ftp | BOOLEAN | --ftps-fallback-to-ftp | Fall back to FTP if FTPS is not supported in the target server |
| keep-session-cookies | BOOLEAN | --keep-session-cookies | Load and save session (non-permanent) cookies |
| no-check-certificate | BOOLEAN | --no-check-certificate | Don't validate the server's certificate |
| no-if-modified-since | BOOLEAN | --no-if-modified-since | Don't use conditional if-modified-since get requests in timestamping mode |
| preserve-permissions | BOOLEAN | --preserve-permissions | Preserve remote file permissions |
| protocol-directories | BOOLEAN | --protocol-directories | Use protocol name in directories |
| accept-list-extensions | FILE | --accept | Comma-separated list of accepted extensions |
| no-use-server-timestamps | BOOLEAN | --no-use-server-timestamps | Do-no-use-server-timestamps the one on the server |
| ftps-clear-data-connection | BOOLEAN | --ftps-clear-data-connection | Cipher the control channel only; all the data will be in plaintext |
| continue-partially-downloaded-file | BOOLEAN | --continue | Resume getting a partially-downloaded file |
example
# mirror example.com with a custom UA and rate limit; write a WARC archivewget --mirror --level=2 -U "Trickest-wget" --limit-rate=500k --warc-file=example-mirror https://www.example.com/--2026-06-22 12:01:04-- https://www.example.com/Resolving www.example.com... 198.51.100.10Connecting to www.example.com|198.51.100.10|:443... connected.HTTP request sent, awaiting response... 200 OKLength: 1256 (1.2K) [text/html]Saving to: 'www.example.com/index.html'--2026-06-22 12:01:05-- https://www.example.com/aboutSaving to: 'www.example.com/about'FINISHED --2026-06-22 12:01:08--Downloaded: 2 files, 3.1K in 0.4sguidance
Use Wget to download or mirror content into a workflow over HTTP or FTP. It retrieves files, it does not test them, so pair it with a parser or scanner downstream. For scriptable transfers with fine header control on a single request, curl is the lighter sibling.
Single-request transfer tool with precise control. Wget adds recursion and site mirroring.
Dedicated website copier with a richer mirroring UI. Wget is the lighter, scriptable option.
Multi-connection, multi-protocol downloader tuned for speed on large files.
faq
related
Import, export, and link workflow data with Airtable.
Decode Android APK files into smali sources and resources.
Check a file's values against conditions and exit with a matching code.
Extract all hosted zones from AWS Route53.
Output file lines by batch size, given START_LINE and END_LINE.
Extract a batch range from a file's lines or a folder's files, with parallel processing.
A URL feeds Wget, which retrieves the content and writes the downloaded files as a queryable output for later nodes.
Facts on this page come from the live Trickest tool library.