loading
Loading content
loading
Recon
Parse GitHub Archive logs into deduplicated repository and user CSVs.
overview
Parses GitHub Archive hourly event logs into two deduplicated CSVs: unique repositories and unique users. Output is a folder for the next recon stage.
Supply logs with --logs-file, --logs-folder, or --urls-file. Same output shape either way, so downstream nodes stay stable.
Trickest runs it as a managed Recon node. Repo CSV seeds secret scanners; user CSV seeds username enumeration such as enumerepo.
use cases
Parse downloaded GitHub Archive logs into a deduplicated CSV of repositories, then pass it to a secret scanner or code hunter to mine those repos at scale.
Extract every unique user that appears in the archive window and feed the CSV into username enumeration to map the people behind an organization or ecosystem.
Hand gh-scraper a file of specific GitHub Archive log URLs so it pulls and parses only the hours you care about instead of a full local download.
Run gh-scraper first to produce repo and user CSVs, then branch into leak scanning, endpoint discovery, and account enumeration from one shared source list.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| logs-file | FILE | --logs-file | Path of a single file containing GitHub Archive logs to parse. |
| urls-file | FILE | --urls-file | Path of a file listing the log URLs to fetch and parse. |
| logs-folder | FOLDER | --logs-folder | Path of a folder containing GitHub Archive logs to parse in bulk. |
Showing key inputs. gh-scraper exposes 3 inputs in total.
example
# parse a folder of downloaded GitHub Archive logs into repo and user CSVsgh-scraper --logs-folder ./gharchive/2024-06-14/[*] Parsing 24 archive files from ./gharchive/2024-06-14/[*] Wrote repositories.csv (4,812 unique repos) and users.csv (3,977 unique users)repo,url,ownerexample-labs/api-gateway,https://github.com/example-labs/api-gateway,example-labsgreenfield-io/terraform-modules,https://github.com/greenfield-io/terraform-modules,greenfield-ionorthwind-data/etl-pipeline,https://github.com/northwind-data/etl-pipeline,northwind-datausername,url,typedarroyo,https://github.com/darroyo,Usergreenfield-io,https://github.com/greenfield-io,Organizationguidance
Use gh-scraper when you hold GitHub Archive data and need clean repo and user lists. It parses; gh-downloader fetches. Pair a fetch ahead and a leak scanner or enumerator behind.
Sibling in the same archive pipeline. Pulls the logs gh-scraper then parses.
Sibling that enriches archive output. Run it after gh-scraper to add detail to the repo and user lists.
Lists public repositories for a set of GitHub usernames. Complements the user CSV gh-scraper produces.
faq
related
Multi-source subdomain discovery; hand names to httpx.
Find domains and subdomains potentially related to a given domain.
CLI client for the BeVigil OSINT API, keyed by domain or app package.
Multi-cloud public name enumeration for AWS, Azure, and GCP.
List assets from multiple cloud providers in one inventory.
Active multi-technique DNS enumeration for assessments.
GitHub Archive logs feed gh-scraper, which dedupes them and writes a folder of repository and user CSVs ready for downstream scanning.
Facts on this page come from the live Trickest tool library.