Launched this week
Barcrawl
The web intelligence and execution layer for AI agents
12 followers
The web intelligence and execution layer for AI agents
12 followers
BarCrawl is a production-ready browser crawling and extraction API. It renders JavaScript-heavy websites, returns clean text, Markdown, HTML, metadata, JSON-LD, contacts and social profiles, and can generate screenshots and PDFs. It also supports batch crawling, asynchronous jobs, webhooks, API-key management, usage tracking, request history, audit logs and self-hosted deployment. Vision: BarCrawl’s long-term vision is to become the trusted web layer for AI agents.
The async jobs + webhook combo is a smart move for agents that need to crawl at scale without blocking. Nice to see audit logs and usage tracking baked in from the start rather than tacked on later.
Ran a few JS-heavy sites through it and the markdown output came back surprisingly clean, way less cleanup than I expected from other crawlers I have used.
The batch crawling with webhook support is a really thoughtful addition for AI agent workflows. Love that you thought through the async side instead of just tossing out a basic scraper endpoint.
@ivyleague Much appreciated mate. Being on this journey means we are leaning day by day and imrpvoing, next in this are the research and marketing agents along with pricing monitoring etc, and I guess thats something you would enjoy. If you can spare few more mins, do share your thoughts .
Cheers
Jugal