Firecrawl
The Web Data API for AI Agents and Developers
Start new thread

The new Firecrawl /search - Our most accurate Search API for AI agents.

by
•
Search is how AI agents ground themselves in the web, but reading full pages for every query burns tokens fast. We trained a model that returns the excerpts from each /search result that best answer your query, giving your AI agents highly relevant context from every page. It outperforms processing full pages while using 10x fewer tokens. On SimpleQA, AI agents using Firecrawl /search now score 94.7%, higher than any other provider. It's live today on every /search call.

Add a comment

Replies

Best
Maker
📌
Hey Product Hunt 👋 Eric, Caleb, and Nick from Firecrawl here. AI agents rely on search tools to find answers on the web, but those answers are often buried in noise. Navigation, boilerplate, and tangents wrap around the excerpt that matters, so an agent processes far more content than it needs to. That makes search slower, more expensive, and more likely to return the wrong answer. Firecrawl /search now runs every result through a new custom relevance model. It scores every paragraph, list, and table against your query, then returns the excerpts that best answer it. We evaluated the improved /search on SimpleQA, OpenAI's factuality benchmark for short, fact-seeking questions. AI agents using Firecrawl /search achieve state-of-the-art accuracy at 94.7%, the highest score of any search provider we tested. By returning relevant excerpts instead of full pages, each response uses 10x fewer tokens. That leaves more of your agent's context window free and makes the results cheaper and faster to reason over. AI agents with web search keep their context lean across long chains of searches. RAG pipelines get source-linked excerpts ready for a prompt without separate chunking or reranking. Research and enrichment workflows can pull the specific fact, figure, or quote they need. The upgrade is live across the API, SDKs, CLI, and MCP, and the API shape hasn't changed, so existing /search calls start returning better results automatically! If you're building products with agentic search, we'd love to hear how it improves your results.

 I think there are some people that underestimate how much bad retrieval hurts an AI agent. Better search often beats a better model.

Maker

 This is so true! Retrieval is everything

great idea and love the implementation! congrats on the launch to the firecrawl team! Does running every paragraph, list, and table through the custom relevance model introduce any noticeable latency overhead compared to traditional search providers?
Maker

 It adds some latency, but it's very minimal and not noticeable! Let us know what you think :)

someone in the reviews asked for multi-step search/enrichment, wonder if that's just a prompt problem or if it needs a different architecture entirely".

The excerpt model is a smart place to spend the effort. One thing I'd poke: it works on pages you already retrieved. For grounding, the failure I hit most isn't a found page being too long, it's the one page that actually holds the answer never making it into the result set at all.

SimpleQA rewards well-covered facts. The queries that hurt are the long-tail ones where the answer sits on a single thin, badly-linked page. Excerpt selection can't rescue a page that never surfaced.

Two things I'd want to know: does the new model touch recall, or only what you do with what's already retrieved? And on those thin pages, does aggressive excerpting ever drop the one passage that mattered, to save tokens?

Maker

 Very fair points! This is something we're taking into account.

This model runs only on what's already retrieved currently but stay tuned!

Congrats on your launch! 🚀

Firecrawl has been my default for feeding clean web data to LLMs, so seeing /search get more accurate at 10x fewer tokens is a big deal for anyone running retrieval at scale. One suggestion from the buyer side, put a quality-per-token comparison next to the raw token number. Technical teams evaluate cost and accuracy together, and you have a strong story on both.

Great idea and I love the token-efficiency angle. Once you're paying per-token for every agent step, stripping nav/footer cruft before it hits the model matters more than raw scrape speed...

If the relevance model picks wrong excerpts for a query, what's my fallback? Can I still ask for the full page on the same call, or do I have to detect the miss myself and re-scrape? Asking because in agent chains a bad excerpt fails silently and the agent just confidently answers from partial context.

Maker

 Great question. You can always turn off the highlighter model if you need the full page context or prefer to access the web page data in another way.

94.7% on SimpleQA is a strong number, but that benchmark is specifically short factual questions with a fairly clean right answer, which is close to the best case for a relevance model scoring paragraphs against a query. Curious how much that accuracy holds up on queries where the right answer isn't a single fact but requires synthesizing across multiple excerpts, since scoring individual paragraphs and lists against a query is a different task than judging whether the combination of several partial excerpts actually adds up to a complete answer. Also, returning excerpts instead of full pages means the model never sees the surrounding context that a paragraph originally sat in. If a relevant looking paragraph is actually a counterexample, an outdated caveat, or a quote the source page was explicitly refuting, does the excerpt carry any signal that it needs to be read skeptically, or does stripping the surrounding page context risk agents treating pulled quotes as more authoritative than the original author intended.
Maker

 great points!! There are definitely trade-offs, however, you can always turn this option off.

That's a solid benchmark score. Does the relevance model retrain continuously or is it a fixed release?

Maker

 It's fixed for now, but we will retrain in the future.

123
Next