Firecrawl
The Web Data API for AI Agents and Developers
5.0•12 reviews•2.6K followers
The Web Data API for AI Agents and Developers
5.0•12 reviews•2.6K followers
The API to search, scrape, and interact with the web at scale.
This is the 8th launch from Firecrawl. View more
Prometheus by Firecrawl
Launched this week
An experimental Forward Deployed Agent for web data from Firecrawl. Describe the web data you need and it writes Firecrawl code to collect it. Run it yourself or let us host and automatically maintain it as pages change.
Free Options
Launch Team
Firecrawl
@ericciarla The 'run it yourself vs let us host & auto-maintain it' split is the clever part. Where does the aha actually land? First time it writes working scrape code, or first time a page changes and it self-heals without anyone touching it? Feels like that self-heal moment is the real hook
The automatic maintenance angle is clever. Having the agent detect and regenerate scraping code as page structures shift solves one of the biggest pain points in web data pipelines. We've spent significant time dealing with brittle scrapers that break silently and cause data quality issues downstream. How does Prometheus decide when a structural change warrants code regeneration vs. treating it as a transient anomaly?
Firecrawl
@anand_thakkar1 Yes that is my favorite part! We decide based on the historical differences in data
@anand_thakkar1 Would love to test one of your more brittle scraping processes on Prometheus!
the run it yourself or let us host split is interesting because those are pretty different value propositions. running it yourself means you're still responsible for maintenance even if the agent writes the code. hosting means you're trusting Firecrawl to maintain the extraction logic as sites change, which is a significant operational commitment to make on someone else's behalf. curious what the SLA looks like for hosted maintenance and what happens when a site change breaks extraction and you need the data today
Firecrawl
@ansari_adin good points. this is just an experiment but an SLA may come in the future!
This is neat. When page structures change, does Prometheus auto-detect the break or wait for you to flag it?
Firecrawl
@dhiraj_patel5 it automatically flags and fixes!
Prometheus generating the collector code is cool, but the part I’d love to see is whether it also generates validation checks with it. For example, after it samples the data, can it create simple assertions like required fields, expected value ranges, stable selectors, or row counts so teams can catch bad runs before the data reaches downstream systems?
The generated TypeScript, how readable is it in practice? The self-host pitch implies you can own and maintain the code, but "generated by an agent experimenting against a live site" often produces output that works until it doesn't, and is hard to debug when you need to patch it manually.
Is it structured around something like Playwright locators with clear selector logic, or more procedural? And on repair: does Prometheus regenerate from scratch, or try to surgically patch the existing collector?
Web scraping for agents is one of those problems that sounds simple until you actually try it. Curious how it handles sites with heavy JS rendering or login walls — that's usually where these tools fall apart. Will test it on a few of my usual sources.
@josedamian thats what Firecrawl does best!