Launching today
Web Harvester
Go scraper: smart proxies, health checks, rate limits
2 followers
Go scraper: smart proxies, health checks, rate limits
2 followers
Web Harvester is an open-source, provider-agnostic web collection engine built in Go. It combines health-aware proxy rotation, authenticated proxies, adaptive rate limiting, cooldowns, runtime configuration reloads, Prometheus metrics, and Redis Streams. Built for resilient, high-throughput pipelines, it includes reproducible benchmarks for measuring performance and reliability. Clone the repo, run the benchmarks, inspect the code, and help build better web data infrastructure.
Web Harvester Reviews
Reviews