Launching today
Web Harvester
Go scraper: smart proxies, health checks, rate limits
2 followers
Go scraper: smart proxies, health checks, rate limits
2 followers
Web Harvester is an open-source, provider-agnostic web collection engine built in Go. It combines health-aware proxy rotation, authenticated proxies, adaptive rate limiting, cooldowns, runtime configuration reloads, Prometheus metrics, and Redis Streams. Built for resilient, high-throughput pipelines, it includes reproducible benchmarks for measuring performance and reliability. Clone the repo, run the benchmarks, inspect the code, and help build better web data infrastructure.
