Parse sitemap.xml (and nested sitemaps) into a deduped URL inventory.
3 · Fetch & extract (distributed)
Fetch each page and pull title, meta, OG tags, H1 and word count — fanned across the fleet.
1 · Seed
Target sitemap sources (domains / sitemap URLs).
4 · Dedupe, classify & roll up
Collect all shards, dedupe, classify, and roll the pages into one inventory.
Overview
A site's sitemap is the list of pages it wants found, which makes it the fastest
way to inventory the whole public surface without a crawler guessing at links.
This workflow reads a domain's sitemap.xml, follows a sitemap index into its
child sitemaps, and fetches every URL it lists. For each page it pulls the title,
meta description, Open Graph tags, first heading and word count, then classifies
the page by type and flags where the metadata is thin or missing. What you get
back is one table: every URL a site publishes, what each page is, and which ones
have title or description problems. It turns a raw sitemap into a content map you
can read, which is the groundwork for
exposure management or a content audit.
Pipeline
Seed the target list: the domains to inventory, one per line.
Read each domain's sitemap.xml. A sitemap index is followed into its child
sitemaps; every listed URL is collected and deduplicated.
Fetch each page and extract its title, meta description, Open Graph title and
description, first h1, HTTP status and body word count.
Deduplicate by URL, classify each page by type, flag thin or missing
metadata, and roll the results up into per-type counts, word totals and a
status-code breakdown.
Inputs
Target list. The domains to inventory, seeded in the first node and
extended by pasting more or feeding an upstream source.
Outputs
Page table. One row per URL, carrying the extracted title, description,
Open Graph tags, heading and word count, plus the page type and metadata
flags the run assigned.
Metadata flags. Per page, whether the title and description fall in their
target lengths and whether Open Graph tags and an h1 are present, so thin
pages stand out.
Rollup. Totals across the site: pages, words, page types, HTTP status
spread, and the share of pages with a compliant title and description.
Sample output
Illustrative, in the record shape the workflow emits. Real runs name live URLs,
so the site here is a documentation domain.
Each page comes back with its extracted metadata, a type, and the hygiene flags,
so the thin pages stand out at a glance:
The title, meta description, Open Graph title and description, the first h1,
the HTTP status, and the body word count with navigation and boilerplate
stripped. It prefers the Open Graph title and description when a page sets them,
which is what a link preview actually shows.
How does it decide a page's type?
It reads keyword signals in the URL, title, heading and description and sorts
each page into a type such as pricing, blog, documentation, product, about,
support, careers or legal. Anything without a clear signal is left as general,
so the classification never invents a category a page did not earn.
How much of a large site does it cover?
A flat sitemap is read in full. A sitemap index, which large sites use to split
their URLs across many files, is followed into its child sitemaps, and by
default the run samples the first of them as a scale guard. Widen that in the
crawl node when you want the whole index.
Does it need an API key or a login?
No. It reads the public sitemap.xml and fetches public pages over HTTP, the
same way a search crawler would, with no credentials and no account.
Related workflows
Directory and Content Discovery.
Reach for this instead when a site has no useful sitemap and you need to brute
force paths to find the pages it did not publish.
Domain OSINT and Footprinting. Reach
for this first when you have a company rather than a domain and need to find
the sites worth inventorying in the first place.