loading
Loading content
loading
Discovery
Spider and scrape targets in search of cloud resources.
overview
cloudscraper spiders a target URL and scrapes page source for cloud resource strings such as S3 buckets, Azure blobs, and DigitalOcean spaces. Unlike keyword brute-force tools, it only reports references the application already links or embeds.
Depth (-d) caps how far the crawler follows links, and -p controls parallel processes. Feed -l for a file of seeds when you want the same scrape across many apps.
On Trickest the Discovery node takes a URL or target list and writes a file and a folder of findings. Hand bucket names to s3scanner or similar tools to test exposure.
use cases
Crawl a product site with -u so embedded S3 or Azure URLs surface without guessing names.
Pass scraped bucket names into s3scanner or cloud permission checkers to see which assets are public.
Use -l with a line-delimited URL list and raise -p when many apps need the same cloud scrape.
reference
| Name | Type | Flag | Description |
|---|---|---|---|
| url | STRING | -u | Target scope URL to spider. |
| depth | STRING | -d | Max link depth to crawl (default 5). |
| target-list | FILE | -l | File of line-delimited target URLs. |
| processes | STRING | -p | Number of parallel processes (default 2). |
| verbose | BOOLEAN | -v | Enable verbose output. |
| no-verify | BOOLEAN | --no-verify | Skip TLS verification. |
Showing key inputs. cloudscraper exposes 6 inputs in total.
example
# spider example.com two levels deep for cloud resource stringscloudscraper -u https://example.com -d 2 -p 2 -v[+] crawling https://example.com[+] found s3: example-static.s3.amazonaws.com[+] found s3: example-uploads.s3.amazonaws.com[+] found azure: examplemedia.blob.core.windows.net[+] found gcs: example-cdn.storage.googleapis.com[+] crawling https://www.example.com/docs[+] found s3: example-docs-public.s3.amazonaws.com[+] done: 7 cloud referencesguidance
Use CloudScraper when you want the cloud resources a site references rather than guessed names, so it pairs with the app you are already testing. To brute-force public names from a keyword instead, use cloud-enum. To list assets in accounts you own, use cloudlist.
Brute-forces public cloud names from a keyword. CloudScraper scrapes references from the target's own pages.
Tests S3 buckets for exposure. Feed it the references CloudScraper finds.
General-purpose crawler. CloudScraper focuses on extracting cloud resource strings while it spiders.
faq
related
Check whether a URL redirects to a masked 404 page.
Append lines to a file only if they are not already there.
Extract URLs and endpoints from Android APK files.
Visual inspection of websites across a large number of hosts.
Find suspicious files across a large set of AWS S3 buckets.
An automated tool that checks for backup artifacts that may disclose a web application's source code.
A target URL feeds CloudScraper, which scrapes cloud references from its pages and passes them to s3scanner so exposed buckets land as output.
Facts on this page come from the live Trickest tool library.