Is gh-downloader a GitHub OSINT or recon tool?

Yes. It sits at the harvesting stage of GitHub OSINT: instead of querying the live GitHub API, it mines the GitHub Archive event stream to build broad, deduplicated lists of repositories and users active in a time window. Those lists become the seed set for the rest of a recon workflow.

What does gh-downloader produce?

Two deduplicated CSV files, exposed as a folder and a file output: every unique repository and every unique user found across the supplied GitHub Archive URLs. Repeated entries collapse to one row each, so the downstream data stays small enough to query or chain.

Where do the input URLs come from, and how do I supply them?

From GitHub Archive's hourly index, files named like 2024-01-15-9.json.gz. You put the URLs you want in a text file and pass it with --urls-file. On Trickest, wire a file input into the node's urls-file slot instead of setting the flag by hand.

How do I speed up a large run?

Raise --threads so the tool fetches and parses many archive URLs at once instead of one at a time. A list that spans days or weeks of hourly snapshots benefits most, since each hour is an independent download.

What does the --one flag do?

--one collapses the results into a single output file under the name you give it, rather than keeping repositories and users split across separate files. Use it when the next stage wants one combined CSV.

What runs after gh-downloader in a workflow?

gh-investigator. It reads the repository and user CSVs and pulls out the accounts and projects worth a closer look. For hunting leaked credentials across live GitHub search instead of archive snapshots, reach for git-wild-hunt.

Updated Jul 14, 2026

Recon

Turn GitHub Archive URLs into unique repository and user CSVs

Process GitHub Archive URLs into unique repository and user CSV files.

Harvest active repositories from GitHub Archive Build a target list of GitHub users Process large archive sets in parallel

overview

What gh-downloader does

gh-downloader reads a file of GitHub Archive URLs, fetches each one, and parses the event stream into two deduplicated CSV files: every unique repository and every unique user that appears in the data. GitHub Archive publishes hourly snapshots of public GitHub activity, so a list of those URLs gives you a wide, time-bounded view of who is pushing code and which repositories are active.

The tool is built for volume. It works through the URL list with a configurable thread pool, so a long list of hourly archives processes in parallel instead of one request at a time. The output stays clean because it collapses repeated repositories and users down to a single row each, which keeps the downstream CSV small enough to query or feed into the next stage.

On Trickest, gh-downloader is a Recon node that takes a URLs file and writes a folder plus a file of results. It opens a GitHub Archive scraping pipeline: run it first to harvest the raw repository and user lists, then pass those CSV files to gh-investigator to pull out the interesting accounts and projects.

source github.com/carlospolop/github_archive_scraper

use cases

Where gh-downloader fits

Harvest active repositories from GitHub Archive

Feed a list of hourly GitHub Archive URLs and collect every unique repository that appears, building a deduplicated CSV of projects active in that time window.

Build a target list of GitHub users

Pull every unique user out of the same archive data so you have a clean account list to profile, cross-reference, or enrich in later OSINT stages.

Process large archive sets in parallel

Raise the thread count to work through a long list of hourly archives at once, turning days or weeks of GitHub activity into CSV files in a single run.

Seed a GitHub investigation pipeline

Generate the repository and user CSV files that gh-investigator consumes, so the discovery and analysis stages chain cleanly inside one workflow.

reference

gh-downloader inputs and flags

3 inputs

Name	Type	Flag	Description
urls-file	FILE	--urls-file	File containing the GitHub Archive URLs to process.
threads	STRING	--threads	Number of threads to use for processing the URLs.
one	STRING	--one	Collapse the results into one file under the given file name.

Showing key inputs. gh-downloader exposes 3 inputs in total.

example

Run gh-downloader

gh-downloader · command

# harvest hourly GitHub Archive URLs into deduplicated repo and user CSVsgh-downloader --urls-file archive-urls.txt --threads 20 --one github-activity.csv

sample output

[+] Queued 24 GitHub Archive URLs across 20 threads[+] Parsed 41,882 events, wrote deduplicated repos and usersrepositories.csv  (repo_name, repo_id)example-labs/payments-api, 208446120demo-sandbox/edge-proxy, 331902547octo-demo/terraform-modules, 417563980users.csv  (user_login, user_id)example-labs, 10294857demo-sandbox, 20481923

guidance

Choosing gh-downloader

Reach for gh-downloader when you have a list of GitHub Archive URLs and need clean, deduplicated repository and user CSV files out of them. It is the harvesting front end of a GitHub Archive pipeline. For turning those CSV files into findings, run gh-investigator next. To target the live GitHub search instead of archive snapshots, use git-wild-hunt.

gh-investigator

Consumes the repository and user CSV files gh-downloader produces and pulls out the interesting accounts and projects. Run it after, not instead.

gh-enhancer

Another GitHub Archive processing node in the same family. Use it alongside gh-downloader when enriching the harvested archive data.

git-wild-hunt

Hunts for leaked credentials across live GitHub search rather than archive snapshots. Different source, complementary coverage.

faq

gh-downloader questions

GitHub Archive is a public dataset that records every public event on GitHub and republishes it as downloadable hourly JSON snapshots at gharchive.org. Each snapshot lists pushes, forks, issues, and the repositories and users behind them. gh-downloader consumes a file of those hourly URLs, so you point it at the windows of activity you care about.

Run gh-downloader yourself

A file of GitHub Archive URLs feeds gh-downloader, which writes deduplicated repository and user CSV files, then gh-investigator reads them and surfaces the interesting accounts and projects as a queryable output.

Run gh-downloader in Trickest View the workflow

Facts on this page come from the live Trickest tool library.

Name

Type

Flag

Description

urls-file

FILE

--urls-file

File containing the GitHub Archive URLs to process.

threads

STRING

--threads

Number of threads to use for processing the URLs.

one

STRING

--one

Collapse the results into one file under the given file name.

[+] Queued 24 GitHub Archive URLs across 20 threads[+] Parsed 41,882 events, wrote deduplicated repos and usersrepositories.csv (repo_name, repo_id)example-labs/payments-api, 208446120demo-sandbox/edge-proxy, 331902547octo-demo/terraform-modules, 417563980users.csv (user_login, user_id)example-labs, 10294857demo-sandbox, 20481923

Run gh-downloader yourself

Facts on this page come from the live Trickest tool library.

Turn GitHub Archive URLs into unique repository and user CSVs

What gh-downloader does

Where gh-downloader fits

Harvest active repositories from GitHub Archive

Build a target list of GitHub users

Process large archive sets in parallel

Seed a GitHub investigation pipeline

gh-downloader inputs and flags

Run gh-downloader

Choosing gh-downloader

gh-investigator

gh-enhancer

git-wild-hunt

gh-downloader questions

More Recon tools

aiodnsbrute

amass

amass-intel

amass-json

analyticsrelationships

assetfinder

Run gh-downloader yourself

Turn GitHub Archive URLs into unique repository and user CSVs

What gh-downloader does

Where gh-downloader fits

Harvest active repositories from GitHub Archive

Build a target list of GitHub users

Process large archive sets in parallel

Seed a GitHub investigation pipeline

gh-downloader inputs and flags

Run gh-downloader

Choosing gh-downloader

gh-investigator

gh-enhancer

git-wild-hunt

gh-downloader questions

More Recon tools

aiodnsbrute

amass

amass-intel

amass-json

analyticsrelationships

assetfinder

Run gh-downloader yourself