Recon
Deduplicate archive events into repo and user lists
Parse GitHub Archive logs into deduplicated repository and user CSVs.
overview
What gh-scraper does
Parses GitHub Archive hourly event logs into two deduplicated CSVs: unique repositories and unique users. Output is a folder for the next recon stage.
Supply logs with --logs-file, --logs-folder, or --urls-file. Same output shape either way, so downstream nodes stay stable.
Trickest runs it as a managed Recon node. Repo CSV seeds secret scanners; user CSV seeds username enumeration such as enumerepo.
use cases
Where gh-scraper fits
Build a repository list from archive logs
Parse downloaded GitHub Archive logs into a deduplicated CSV of repositories, then pass it to a secret scanner or code hunter to mine those repos at scale.
Harvest user accounts for identity enumeration
Extract every unique user that appears in the archive window and feed the CSV into username enumeration to map the people behind an organization or ecosystem.
Process a curated set of log URLs
Hand gh-scraper a file of specific GitHub Archive log URLs so it pulls and parses only the hours you care about instead of a full local download.
Seed a GitHub recon pipeline
Run gh-scraper first to produce repo and user CSVs, then branch into leak scanning, endpoint discovery, and account enumeration from one shared source list.
reference
gh-scraper inputs and flags
| Name | Type | Flag | Description |
|---|---|---|---|
| logs-file | FILE | --logs-file | Path of a single file containing GitHub Archive logs to parse. |
| urls-file | FILE | --urls-file | Path of a file listing the log URLs to fetch and parse. |
| logs-folder | FOLDER | --logs-folder | Path of a folder containing GitHub Archive logs to parse in bulk. |
Showing key inputs. gh-scraper exposes 3 inputs in total.
example
Run gh-scraper
# parse a folder of downloaded GitHub Archive logs into repo and user CSVsgh-scraper --logs-folder ./gharchive/2024-06-14/[*] Parsing 24 archive files from ./gharchive/2024-06-14/[*] Wrote repositories.csv (4,812 unique repos) and users.csv (3,977 unique users)repo,url,ownerexample-labs/api-gateway,https://github.com/example-labs/api-gateway,example-labsgreenfield-io/terraform-modules,https://github.com/greenfield-io/terraform-modules,greenfield-ionorthwind-data/etl-pipeline,https://github.com/northwind-data/etl-pipeline,northwind-datausername,url,typedarroyo,https://github.com/darroyo,Usergreenfield-io,https://github.com/greenfield-io,Organizationguidance
Choosing gh-scraper
Use gh-scraper when you hold GitHub Archive data and need clean repo and user lists. It parses; gh-downloader fetches. Pair a fetch ahead and a leak scanner or enumerator behind.
gh-downloader
Sibling in the same archive pipeline. Pulls the logs gh-scraper then parses.
gh-enhancer
Sibling that enriches archive output. Run it after gh-scraper to add detail to the repo and user lists.
enumerepo
Lists public repositories for a set of GitHub usernames. Complements the user CSV gh-scraper produces.
faq
gh-scraper questions
related
More Recon tools
amass
Multi-source subdomain discovery; hand names to httpx.
amass-intel
OWASP Amass intel: map an organization's root domains and ranges.
assetfinder
Find domains and subdomains potentially related to a given domain.
bbot
Modular OSINT recon that chains modules from a seed target.
bevigil
CLI client for the BeVigil OSINT API, keyed by domain or app package.
cloud-enum
Multi-cloud public name enumeration for AWS, Azure, and GCP.
Run gh-scraper yourself
GitHub Archive logs feed gh-scraper, which dedupes them and writes a folder of repository and user CSVs ready for downstream scanning.
Facts on this page come from the live Trickest tool library.