loading
Loading content
loading
Recon
GitHub Archive URLs in, unique repository and user CSVs out.
overview
gh-downloader reads a --urls-file of GitHub Archive hourly snapshots, fetches each one, and parses the event stream into deduplicated repository and user CSV files.
Raise --threads to process many archive hours in parallel. Use --one when the next stage wants a single combined file instead of split CSVs.
Trickest runs it as a managed Recon node that writes a folder and a file. Harvest first, then pass the CSVs to gh-investigator (or gh-enhancer) for cleanup and findings.
use cases
Feed a list of hourly GitHub Archive URLs and collect every unique repository that appears, building a deduplicated CSV of projects active in that time window.
Pull every unique user out of the same archive data so you have a clean account list to profile, cross-reference, or enrich in later OSINT stages.
Raise the thread count to work through a long list of hourly archives at once, turning days or weeks of GitHub activity into CSV files in a single run.
Generate the repository and user CSV files that gh-investigator consumes, so the discovery and analysis stages chain cleanly inside one workflow.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| urls-file | FILE | --urls-file | File containing the GitHub Archive URLs to process. |
| threads | STRING | --threads | Number of threads to use for processing the URLs. |
| one | STRING | --one | Collapse the results into one file under the given file name. |
Showing key inputs. gh-downloader exposes 3 inputs in total.
example
# harvest hourly GitHub Archive URLs into deduplicated repo and user CSVsgh-downloader --urls-file archive-urls.txt --threads 20 --one github-activity.csv[+] Queued 24 GitHub Archive URLs across 20 threads[+] Parsed 41,882 events, wrote deduplicated repos and usersrepositories.csv (repo_name, repo_id)example-labs/payments-api, 208446120demo-sandbox/edge-proxy, 331902547octo-demo/terraform-modules, 417563980users.csv (user_login, user_id)example-labs, 10294857demo-sandbox, 20481923guidance
Use gh-downloader when you already have GitHub Archive URLs and need clean, deduplicated repository and user CSVs. It harvests archive snapshots; it does not query live GitHub search. Run gh-investigator next for findings. Prefer git-wild-hunt when the source is live GitHub search instead of archive dumps.
Consumes the repository and user CSVs and pulls out interesting accounts and projects. Run it after, not instead.
Enrichment stage in the same family. Use it after harvest when raw CSVs need GraphQL cleanup before investigation.
Hunts leaked credentials across live GitHub search rather than archive snapshots. Different source, complementary coverage.
faq
related
Multi-source subdomain discovery; hand names to httpx.
Find domains and subdomains potentially related to a given domain.
CLI client for the BeVigil OSINT API, keyed by domain or app package.
Multi-cloud public name enumeration for AWS, Azure, and GCP.
List assets from multiple cloud providers in one inventory.
Active multi-technique DNS enumeration for assessments.
A file of GitHub Archive URLs feeds gh-downloader, which writes deduplicated repository and user CSV files, then gh-investigator reads them and surfaces the interesting accounts and projects as a queryable output.
Facts on this page come from the live Trickest tool library.