Firecrawl
The Web Data API for AI Agents and Developers
5.0•14 reviews•2.9K followers
The Web Data API for AI Agents and Developers
5.0•14 reviews•2.9K followers
The API to search, scrape, and interact with the web at scale.
This is the 10th launch from Firecrawl. View more
The new Firecrawl /search
Launching today
Search is how AI agents ground themselves in the web, but reading full pages for every query burns tokens fast.
We trained a model that returns the excerpts from each /search result that best answer your query, giving your AI agents highly relevant context from every page.
It outperforms processing full pages while using 10x fewer tokens.
On SimpleQA, AI agents using Firecrawl /search now score 94.7%, higher than any other provider. It's live today on every /search call.
Free Options
Launch Team
Firecrawl
@ericciarla I think there are some people that underestimate how much bad retrieval hurts an AI agent. Better search often beats a better model.
Firecrawl
@reda_roqai_chaoui This is so true! Retrieval is everything
The excerpt model is a smart place to spend the effort. One thing I'd poke: it works on pages you already retrieved. For grounding, the failure I hit most isn't a found page being too long, it's the one page that actually holds the answer never making it into the result set at all.
SimpleQA rewards well-covered facts. The queries that hurt are the long-tail ones where the answer sits on a single thin, badly-linked page. Excerpt selection can't rescue a page that never surfaced.
Two things I'd want to know: does the new model touch recall, or only what you do with what's already retrieved? And on those thin pages, does aggressive excerpting ever drop the one passage that mattered, to save tokens?
Firecrawl
@jernej_jan_kocica Very fair points! This is something we're taking into account.
This model runs only on what's already retrieved currently but stay tuned!
Firecrawl
@nivotools It adds some latency, but it's very minimal and not noticeable! Let us know what you think :)
If the relevance model picks wrong excerpts for a query, what's my fallback? Can I still ask for the full page on the same call, or do I have to detect the miss myself and re-scrape? Asking because in agent chains a bad excerpt fails silently and the agent just confidently answers from partial context.
Firecrawl
@arya012 Great question. You can always turn off the highlighter model if you need the full page context or prefer to access the web page data in another way.
That's a solid benchmark score. Does the relevance model retrain continuously or is it a fixed release?
Firecrawl
@dhiraj_patel5 It's fixed for now, but we will retrain in the future.
Firecrawl
@thys_beesman great points!! There are definitely trade-offs, however, you can always turn this option off.
The 10x fewer tokens claim is the part that caught my attention. In agent workflows, the expensive failure is often not retrieval itself, but letting the model chew through irrelevant page structure. I’d be curious to see where the relevance model struggles most: tables, long docs, or queries with several equally plausible answers?