---
title: "Sitemap URL & Metadata Extractor"
canonical: https://trickest.com/library/sitemap-url-extractor
description: "Read a site's sitemap, fetch every page it lists, and inventory each URL's title, description, page type and metadata gaps in one table."
---

# Sitemap URL & Metadata Extractor

## Overview

A site's sitemap is the list of pages it wants found, which makes it the fastest
way to inventory the whole public surface without a crawler guessing at links.
This workflow reads a domain's `sitemap.xml`, follows a sitemap index into its
child sitemaps, and fetches every URL it lists. For each page it pulls the title,
meta description, Open Graph tags, first heading and word count, then classifies
the page by type and flags where the metadata is thin or missing. What you get
back is one table: every URL a site publishes, what each page is, and which ones
have title or description problems. It turns a raw sitemap into a content map you
can read, which is the groundwork for
[exposure management](/use-cases/exposure-management) or a content audit.

## Pipeline

1. Seed the target list: the domains to inventory, one per line.
2. Read each domain's `sitemap.xml`. A sitemap index is followed into its child
   sitemaps; every listed URL is collected and deduplicated.
3. Fetch each page and extract its title, meta description, Open Graph title and
   description, first `h1`, HTTP status and body word count.
4. Deduplicate by URL, classify each page by type, flag thin or missing
   metadata, and roll the results up into per-type counts, word totals and a
   status-code breakdown.

## Inputs

- **Target list.** The domains to inventory, seeded in the first node and
  extended by pasting more or feeding an upstream source.

## Outputs

- **Page table.** One row per URL, carrying the extracted title, description,
  Open Graph tags, heading and word count, plus the page type and metadata
  flags the run assigned.
- **Metadata flags.** Per page, whether the title and description fall in their
  target lengths and whether Open Graph tags and an `h1` are present, so thin
  pages stand out.
- **Rollup.** Totals across the site: pages, words, page types, HTTP status
  spread, and the share of pages with a compliant title and description.

## Sample output

Illustrative, in the record shape the workflow emits. Real runs name live URLs,
so the site here is a documentation domain.

Each page comes back with its extracted metadata, a type, and the hygiene flags,
so the thin pages stand out at a glance:

```jsonl
{"url": "https://www.example.com/pricing", "http_status": 200, "title": "Pricing", "word_count": 812, "category": "pricing", "title_ok": false, "has_og_title": true, "has_h1": true}
{"url": "https://www.example.com/blog/how-we-scale", "http_status": 200, "title": "How we scale ingestion to a billion rows a day", "word_count": 2140, "category": "blog", "title_ok": true, "has_og_title": true, "has_h1": true}
{"url": "https://www.example.com/docs/api", "http_status": 200, "title": "API reference", "word_count": 340, "category": "documentation", "title_ok": false, "has_og_title": false, "has_h1": true}
{"url": "https://www.example.com/old-landing", "http_status": 301, "title": "", "word_count": 0, "category": "landing", "title_ok": false, "has_og_title": false, "has_h1": false}
```

The rollup is the whole site in one record:

```json
{"total_pages": 214, "total_words": 168430, "avg_word_count": 787.1, "categories": {"blog": 96, "documentation": 61, "product": 22, "landing": 14, "pricing": 1, "general": 20}, "http_status_dist": {"200": 209, "301": 4, "404": 1}, "title_ok_pct": 41.6, "desc_ok_pct": 33.2}
```

## FAQ

### What does it pull from each page?

The title, meta description, Open Graph title and description, the first `h1`,
the HTTP status, and the body word count with navigation and boilerplate
stripped. It prefers the Open Graph title and description when a page sets them,
which is what a link preview actually shows.

### How does it decide a page's type?

It reads keyword signals in the URL, title, heading and description and sorts
each page into a type such as pricing, blog, documentation, product, about,
support, careers or legal. Anything without a clear signal is left as general,
so the classification never invents a category a page did not earn.

### How much of a large site does it cover?

A flat sitemap is read in full. A sitemap index, which large sites use to split
their URLs across many files, is followed into its child sitemaps, and by
default the run samples the first of them as a scale guard. Widen that in the
crawl node when you want the whole index.

### Does it need an API key or a login?

No. It reads the public `sitemap.xml` and fetches public pages over HTTP, the
same way a search crawler would, with no credentials and no account.

## Related workflows

- [Directory and Content Discovery](/library/directory-and-content-discovery).
  Reach for this instead when a site has no useful sitemap and you need to brute
  force paths to find the pages it did not publish.
- [Domain OSINT and Footprinting](/library/domain-osint-and-footprinting). Reach
  for this first when you have a company rather than a domain and need to find
  the sites worth inventorying in the first place.

---
_Markdown source of https://trickest.com/library/sitemap-url-extractor._