loading
Loading content
loading
Recon
Turn raw GitHub Archive logs into deduplicated repository and user CSV files.
overview
gh-scraper reads GitHub Archive data and distills it into two clean CSV files: one of unique repositories and one of unique users. GitHub Archive records every public event on GitHub as hourly log files, so a few hours of data already names thousands of repos and accounts. gh-scraper walks those events, dedupes them, and writes the result as a folder you can hand to the next stage.
You point it at the logs three ways. Give it a single logs file, a folder of logs you have already downloaded, or a file listing the log URLs you want it to pull and parse. Whichever input you pick, the output shape is the same, so the tool slots into a pipeline without reshaping anything downstream.
On Trickest, gh-scraper is a Recon node that takes archive logs and writes a folder of CSVs. Use it to seed a GitHub-focused recon run: the repository list feeds secret scanners and endpoint hunters, and the user list feeds username and identity enumeration against an organization or ecosystem.
use cases
Parse downloaded GitHub Archive logs into a deduplicated CSV of repositories, then pass it to a secret scanner or code hunter to mine those repos at scale.
Extract every unique user that appears in the archive window and feed the CSV into username enumeration to map the people behind an organization or ecosystem.
Hand gh-scraper a file of specific GitHub Archive log URLs so it pulls and parses only the hours you care about instead of a full local download.
Run gh-scraper first to produce repo and user CSVs, then branch into leak scanning, endpoint discovery, and account enumeration from one shared source list.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| logs-file | FILE | --logs-file | Path of a single file containing GitHub Archive logs to parse. |
| urls-file | FILE | --urls-file | Path of a file listing the log URLs to fetch and parse. |
| logs-folder | FOLDER | --logs-folder | Path of a folder containing GitHub Archive logs to parse in bulk. |
Showing key inputs. gh-scraper exposes 3 inputs in total.
example
# parse a folder of downloaded GitHub Archive logs into repo and user CSVsgh-scraper --logs-folder ./gharchive/2024-06-14/[*] Parsing 24 archive files from ./gharchive/2024-06-14/[*] Wrote repositories.csv (4,812 unique repos) and users.csv (3,977 unique users)repo,url,ownerexample-labs/api-gateway,https://github.com/example-labs/api-gateway,example-labsgreenfield-io/terraform-modules,https://github.com/greenfield-io/terraform-modules,greenfield-ionorthwind-data/etl-pipeline,https://github.com/northwind-data/etl-pipeline,northwind-datausername,url,typedarroyo,https://github.com/darroyo,Usergreenfield-io,https://github.com/greenfield-io,Organizationguidance
Reach for gh-scraper when you have GitHub Archive data and need clean repository and user lists out of it. It is a parser, not a downloader or a scanner, so pair it with a fetch step ahead of it and a leak scanner or username enumerator behind it.
Sibling in the same archive pipeline. Pulls the logs gh-scraper then parses.
Sibling that enriches archive output. Run it after gh-scraper to add detail to the repo and user lists.
Lists public repositories for a set of GitHub usernames. Complements the user CSV gh-scraper produces.
faq
related
Asynchronous DNS brute-force tool for fast subdomain enumeration.
OWASP Amass: network mapping and external asset discovery.
OWASP Amass intel: find an organization's root domains and ranges.
OWASP Amass enumeration that produces structured JSON output.
Find related domains and subdomains via shared Google Analytics IDs.
Find domains and subdomains potentially related to a given domain.
GitHub Archive logs feed gh-scraper, which dedupes them and writes a folder of repository and user CSVs ready for downstream scanning.
Facts on this page come from the live Trickest tool library.