Sitemap URL & Metadata Extractor

Overview

A site's sitemap is the list of pages it wants found, which makes it the fastest way to inventory the whole public surface without a crawler guessing at links. This workflow reads a domain's sitemap.xml, follows a sitemap index into its child sitemaps, and fetches every URL it lists. For each page it pulls the title, meta description, Open Graph tags, first heading and word count, then classifies the page by type and flags where the metadata is thin or missing. What you get back is one table: every URL a site publishes, what each page is, and which ones have title or description problems. It turns a raw sitemap into a content map you can read, which is the groundwork for exposure management or a content audit.

Pipeline

1. Seed the target list: the domains to inventory, one per line.
2. Read each domain's sitemap.xml. A sitemap index is followed into its child sitemaps; every listed URL is collected and deduplicated.
3. Fetch each page and extract its title, meta description, Open Graph title and description, first h1, HTTP status and body word count.
4. Deduplicate by URL, classify each page by type, flag thin or missing metadata, and roll the results up into per-type counts, word totals and a status-code breakdown.

Inputs

• Target list. The domains to inventory, seeded in the first node and extended by pasting more or feeding an upstream source.

Outputs

• Page table. One row per URL, carrying the extracted title, description, Open Graph tags, heading and word count, plus the page type and metadata flags the run assigned.
• Metadata flags. Per page, whether the title and description fall in their target lengths and whether Open Graph tags and an h1 are present, so thin pages stand out.
• Rollup. Totals across the site: pages, words, page types, HTTP status spread, and the share of pages with a compliant title and description.

Sample output

Illustrative, in the record shape the workflow emits. Real runs name live URLs, so the site here is a documentation domain.

Each page comes back with its extracted metadata, a type, and the hygiene flags, so the thin pages stand out at a glance:

The rollup is the whole site in one record:

{"total_pages": 214, "total_words": 168430, "avg_word_count": 787.1, "categories": {"blog": 96, "documentation": 61, "product": 22, "landing": 14, "pricing": 1, "general": 20}, "http_status_dist": {"200": 209, "301": 4, "404": 1}, "title_ok_pct": 41.6, "desc_ok_pct": 33.2}

FAQ

What does it pull from each page?

The title, meta description, Open Graph title and description, the first h1, the HTTP status, and the body word count with navigation and boilerplate stripped. It prefers the Open Graph title and description when a page sets them, which is what a link preview actually shows.

How does it decide a page's type?

It reads keyword signals in the URL, title, heading and description and sorts each page into a type such as pricing, blog, documentation, product, about, support, careers or legal. Anything without a clear signal is left as general, so the classification never invents a category a page did not earn.

How much of a large site does it cover?

A flat sitemap is read in full. A sitemap index, which large sites use to split their URLs across many files, is followed into its child sitemaps, and by default the run samples the first of them as a scale guard. Widen that in the crawl node when you want the whole index.

Does it need an API key or a login?

No. It reads the public sitemap.xml and fetches public pages over HTTP, the same way a search crawler would, with no credentials and no account.

Related workflows

• Directory and Content Discovery. Reach for this instead when a site has no useful sitemap and you need to brute force paths to find the pages it did not publish.
• Domain OSINT and Footprinting. Reach for this first when you have a company rather than a domain and need to find the sites worth inventorying in the first place.