Utilities
Render URLs in headless Chromium and save the page
Headless Chromium render: requested URL, final URL, status, and title next to the page.
overview
What browser-fetch does
You have a URL that is empty without a browser, and the next node needs the rendered HTML. Set --url for one page or --urls for a file. The wrapper launches Chromium, waits for --wait-until (load, domcontentloaded, networkidle, or commit), and writes each page plus requested URL, final URL, status, and title.
--timeout is milliseconds (default 30000). --settle waits extra milliseconds after load. --concurrency bounds parallel pages. --max-urls caps the batch. --locale and --user-agent override the browser. An upstream folder on --indir is scanned for URL lists.
browser-fetch renders what Chromium can open from the fleet. It does not claim to bypass a bot wall, a login, or a paywall. A challenge page is a failed fetch. Use trafilatura when static HTML is enough. Use playwright when you need a screenshot or a trace, not just the HTML.
use cases
Where browser-fetch fits
Render one URL
Set --url https://example.com. The folder holds the page plus title and status.
Render a file of URLs
Pass --urls. --max-urls caps the batch. --concurrency sets how many pages run at once.
Wait for network idle
Set --wait-until networkidle when the body arrives after XHR.
Hand HTML to trafilatura
When you already have rendered HTML on disk, a later parser can read it. For a live static page, call trafilatura directly.
reference
browser-fetch inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | --url | Single URL to render. |
| urls | FILE | --urls | File of URLs to render. |
| wait-until | STRING | --wait-until | load, domcontentloaded, networkidle, or commit. |
| timeout | STRING | --timeout | Navigation timeout in milliseconds. Default 30000. |
Showing key inputs. browser-fetch exposes 10 inputs in total.
Full flag reference (10 inputs)
| Name | Type | Flag | Description |
|---|---|---|---|
| in | FOLDER | --indir | Upstream folder with URLs (one per line, JSON array, or JSONL with a url field) |
| urls | FILE | --urls | File of URLs to render |
| url | STRING | --url | Single URL to render |
| wait-until | STRING | --wait-until | Navigation wait: load | domcontentloaded | networkidle | commit |
| timeout | STRING | --timeout | Navigation timeout in milliseconds (default: 30000) |
| concurrency | STRING | --concurrency | Pages rendered at once (default: 4) |
| settle | STRING | --settle | Extra wait after load in milliseconds (default: 800) |
| locale | STRING | --locale | Browser locale (default: en-US) |
| user-agent | STRING | --user-agent | Override the browser User-Agent |
| max-urls | STRING | --max-urls | Cap on URLs fetched per run (default: 500) |
example
Run browser-fetch
browser-fetch --url https://example.com --wait-until load --outdir /hive/outfetched=1 ok=1ok 200 559 https://example.com{"index": 0, "url": "https://example.com", "final_url": "https://example.com/", "status": 200, "title": "Example Domain", "html_file": "pages/0000-example-com.html", "html_chars": 559, "state": "ok"}guidance
Choosing browser-fetch
Use browser-fetch when the HTML is empty without Chromium. Use trafilatura when a static fetch is enough. Use playwright when you need a screenshot or a trace.
trafilatura
Extracts article text from static HTML. Faster when JavaScript is not required.
playwright
Screenshot, PDF, or trace for one URL. browser-fetch keeps HTML and metadata.
wget
Downloads the first response. It does not run Chromium.
faq
browser-fetch questions
related
More Utilities tools
cewl
Spider a URL and return a wordlist for password crackers.
csvkit
SQL on one CSV. The table name is the file stem.
dnsgen
Wordlist and mined-word subdomain permutation.
duckdb
In-process SQL on CSV, JSON, and Parquet. No server.
edge-tts
Microsoft Edge neural speech from text or a script file. No API key.
exiftool
Read embedded metadata from hundreds of image, media, and document formats.
Run browser-fetch yourself
https://example.com feeds browser-fetch. pages.jsonl is from a completed run.
Facts on this page come from the live Trickest tool library.