GitHub Trending Scraper

Overview

The GitHub Trending Scraper reads the GitHub Trending page and turns it into a clean, queryable table instead of a list you have to scroll. It fetches the trending HTML, extracts each repo row, and parses the name, primary language, description, and URL into JSONL. Run it daily to spot repos that are gaining attention, feed the table into a dashboard, or hand it to a downstream step that ranks and filters what is worth a closer look. You can inspect the fetch and the row parse in visual workflows before you schedule it.

Pipeline

1. Fetch GitHub Trending (fetch-trending, tool) retrieves the trending HTML for the chosen language and window.
2. Extract Repo Rows (extract-rows, tool) isolates each repository row from that page.
3. Parse Repo Rows (parse-rows, script) writes name, language, description, and URL as JSONL.

Inputs

• A GitHub Trending URL, optionally filtered by language or time window (for example, the past week).
• Optional: a maximum number of rows to keep.

Outputs

• A JSONL file where each line is one trending repo with its name, language, description, and URL.

Integrations

• Fetch Trending. Reads the GitHub Trending HTML.
• Extract Rows. Isolates each repository row from that page.

Sample output

The artifacts below are illustrative. They show the JSONL shape, not a completed run.

trending.jsonl

name	language	description	url
example/sample-repo	Go	An illustrative CLI used to show the row shape.	https://github.com/example/sample-repo
example/another-repo	Python	A second illustrative row, not captured from a live trending page.	https://github.com/example/another-repo

FAQ

Can I filter by programming language?

Yes. Pass a language-specific trending URL (for example, https://github.com/trending/python) and the scraper extracts only the repos on that page.

How often should I run it?

GitHub Trending shifts daily, so a daily run captures the full churn. Running more often mostly duplicates the previous output.

Does it capture star counts?

The scraper pulls the fields GitHub shows on the trending page, which include the repo name, language, description, and URL but not the exact star count.

Will GitHub block the scraper?

A single fetch per run is well under any rate limit. For frequent runs, space them out and avoid hammering the page from one IP.

Related workflows

• arXiv Paper Summarizer - turn a trending research paper into a structured summary.
• HIBP Breach Monitor - watch for new data breaches and rank their severity.
• Scan GitHub for Leaked Secrets - scan an org's commit history for leaked credentials.