Recon
Harvest unique repos and users from GitHub Archive
GitHub Archive URLs in, unique repository and user CSVs out.
Python · GPL-2.0 license
overview
What gh-downloader does
gh-downloader reads a --urls-file of GitHub Archive hourly snapshots, fetches each one, and parses the event stream into deduplicated repository and user CSV files.
Raise --threads to process many archive hours in parallel. Use --one when the next stage wants a single combined file instead of split CSVs.
Trickest runs it as a managed Recon node that writes a folder and a file. Harvest first, then pass the CSVs to gh-investigator (or gh-enhancer) for cleanup and findings.
use cases
Where gh-downloader fits
Harvest active repositories from GitHub Archive
Feed a list of hourly GitHub Archive URLs and collect every unique repository that appears, building a deduplicated CSV of projects active in that time window.
Build a target list of GitHub users
Pull every unique user out of the same archive data so you have a clean account list to profile, cross-reference, or enrich in later OSINT stages.
Process large archive sets in parallel
Raise the thread count to work through a long list of hourly archives at once, turning days or weeks of GitHub activity into CSV files in a single run.
Seed a GitHub investigation pipeline
Generate the repository and user CSV files that gh-investigator consumes, so the discovery and analysis stages chain cleanly inside one workflow.
reference
gh-downloader inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| urls-file | FILE | --urls-file | File containing the GitHub Archive URLs to process. |
| threads | STRING | --threads | Number of threads to use for processing the URLs. |
| one | STRING | --one | Collapse the results into one file under the given file name. |
Showing key inputs. gh-downloader exposes 3 inputs in total.
example
Run gh-downloader
# harvest hourly GitHub Archive URLs into deduplicated repo and user CSVs
gh-downloader --urls-file archive-urls.txt --threads 20 --one github-activity.csv[+] Queued 24 GitHub Archive URLs across 20 threads
[+] Parsed 41,882 events, wrote deduplicated repos and users
repositories.csv (repo_name, repo_id)
example-labs/payments-api, 208446120
demo-sandbox/edge-proxy, 331902547
octo-demo/terraform-modules, 417563980
users.csv (user_login, user_id)
example-labs, 10294857
demo-sandbox, 20481923guidance
Choosing gh-downloader
Use gh-downloader when you already have GitHub Archive URLs and need clean, deduplicated repository and user CSVs. It harvests archive snapshots; it does not query live GitHub search. Run gh-investigator next for findings. Prefer git-wild-hunt when the source is live GitHub search instead of archive dumps.
gh-investigator
Consumes the repository and user CSVs and pulls out interesting accounts and projects. Run it after, not instead.
gh-enhancer
Enrichment stage in the same family. Use it after harvest when raw CSVs need GraphQL cleanup before investigation.
git-wild-hunt
Hunts leaked credentials across live GitHub search rather than archive snapshots. Different source, complementary coverage.
faq
gh-downloader questions
related
More Recon tools
amass
Multi-source subdomain discovery; hand names to httpx.
amass-intel
OWASP Amass intel: map an organization's root domains and ranges.
assetfinder
Find domains and subdomains potentially related to a given domain.
bbot
Modular OSINT recon that chains modules from a seed target.
bevigil
CLI client for the BeVigil OSINT API, keyed by domain or app package.
cloud-enum
Multi-cloud public name enumeration for AWS, Azure, and GCP.
Run gh-downloader yourself
A file of GitHub Archive URLs feeds gh-downloader, which writes deduplicated repository and user CSV files, then gh-investigator reads them and surfaces the interesting accounts and projects as a queryable output.
Facts on this page come from the live Trickest tool library.