Hey everyone! I built Crawlbot because I spent years working in CE brand management and kept running into the same problem: you have products listed on 15+ retailers, and you have zero visibility into what shoppers actually see. Is your product on page 1 at Currys? Is a competitor buying the top sponsored slot at Amazon? Did Argos drop your hero image? You only find out when sales dip, and by then it's too late. I started with a single Python script scraping one retailer. That turned into Playwright workers, then a distributed queue system across 5 machines, then a full monitoring platform with hourly SoV tracking, nightly content audits, and price change alerts. The hardest part was reliability. Retailers actively fight scrapers, so I had to build multi-layered anti-bot bypass, automatic retries, memory management, and a watchdog system that alerts me on Discord if anything goes down. Today Crawlbot tracks 35 retailers across 4 countries with near-zero downtime. Would love your feedback. Happy to answer any questions about the tech stack or the e-commerce intelligence space!

Crawlbot.pl

Real-time digital shelf analytics for CE brands

Real-time digital shelf analytics for CE brands