loading
Loading content
loading
Recon
Turn raw GitHub Archive events into clean, deduplicated repository and user lists.
overview
gh-enhancer is the cleanup stage of the github_archive_scraper toolkit. It reads the repository and user CSV files produced upstream, queries the GitHub GraphQL API to enrich and confirm them, and writes a folder of unique repositories and users.
It works in batches against the GraphQL API and spreads the load across threads. Supply a single --token for small runs, or --file-tokens so it can rotate across several credentials when one approaches its rate cap. --batch-size controls how many entries each GraphQL request asks for at once.
In a Trickest workflow gh-enhancer takes the repos and users CSV files plus GitHub credentials and emits a folder. Run it after the scraper and downloader stages, and before gh-investigator mines the cleaned lists.
use cases
Feed the raw repos and users CSV files from the scraper into gh-enhancer to collapse duplicate entries into one clean list per type.
Confirm and enrich each entry against GitHub's GraphQL API in batches, dropping stale or invalid records before investigation.
Point --file-tokens at a list of tokens so large archives keep enriching across several credentials instead of stalling on one rate limit.
Produce the clean repositories and users folder that the investigator stage reads to surface secrets, contributors, and other findings.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| repos-file | FILE | --repos-file | Path of the file containing the repos CSV files to process. |
| users-file | FILE | --users-file | Path of the file containing the users CSV files to process. |
| token | STRING | --token | GitHub token used for the API calls. |
| file-tokens | FILE | --file-tokens | File of GitHub tokens to rotate across for API calls. |
| batch-size | STRING | --batch-size | How many entries each GitHub GraphQL request asks for at once. |
| threads | STRING | --threads | Number of threads to run the enrichment with. |
Showing key inputs. gh-enhancer exposes 6 inputs in total.
example
# gh-enhancer: enrich and dedupe archive repos/users via GraphQLgh-enhancer --repos-file repos.csv --users-file users.csv --token "$GITHUB_TOKEN" --batch-size 50 --threads 4https://github.com/example-org/web-frontendhttps://github.com/example-org/payments-apihttps://github.com/example-org/infra-terraformhttps://github.com/example-dev/ci-runnershttps://github.com/example-app/mobile-clienthttps://github.com/example-test/docs-sitehttps://github.com/example-net/edge-proxyhttps://github.com/example-io/sdk-pythonguidance
Reach for GH Enhancer between scraping and investigating a GitHub Archive dataset, when you have raw repos and users CSV files and need them deduplicated and verified. Use gh-scraper or gh-downloader to gather the archive first, and gh-investigator afterward to mine the clean lists.
Upstream stage. Processes GitHub Archive URLs into the raw repos and users CSV files GH Enhancer cleans.
Sibling collector in the same toolkit. Pulls archive data down before enrichment runs.
Downstream stage. Reads the enriched CSV files to surface interesting GitHub findings.
faq
related
Asynchronous DNS brute-force tool for fast subdomain enumeration.
OWASP Amass: network mapping and external asset discovery.
OWASP Amass intel: find an organization's root domains and ranges.
OWASP Amass enumeration that produces structured JSON output.
Find related domains and subdomains via shared Google Analytics IDs.
Find domains and subdomains potentially related to a given domain.
Raw archive CSVs feed gh-scraper, which passes scraped repos and users to GH Enhancer; it deduplicates and enriches them through the GraphQL API into a clean folder of unique lists.
Facts on this page come from the live Trickest tool library.