Veil - A web scraper that uses the lightest method that works

Veil is an open-source Python web scraper that tries the cheapest method first and only escalates when a site blocks it so you're not running a full browser for every page. • 3 tiers: plain HTTP → TLS impersonation → stealth browser • Resumable concurrent crawler (checkpoint, JSONL, dedupe) • Proxy rotation, rate limiting, robots.txt by default • Honest README: documents what it can't beat Python, MIT, tested. A learning project built in the open.

hey PH 👋 i'm still pretty new to python and built this mostly to learn, so honestly any feedback helps. the idea: most scrapers either send a plain http request and get blocked, or spin up a whole headless browser for every page (slow + expensive). veil tries the cheap stuff first and only moves up when it actually gets blocked: plain http → TLS fingerprint impersonation → stealth browser it's also got a crawler that resumes if it crashes, rotates proxies, and respects robots.txt by default. the part i'm actually proud of: i tested it against real sites and wrote down what it CAN'T do. it gets past JS challenges, but if a site throws a 403 before any javascript runs, that's an IP reputation thing and no code fixes it — you just need residential proxies. that's in the readme, i didn't wanna pretend it beats everything. it's open source (MIT), and there's a CONTRIBUTING guide + good-first-issues if anyone wants to build with me. would love to hear what i got wrong 🙏

Veil - A web scraper that uses the lightest method that works

Replies