mcp-server-webcrawl

mcp-server-webcrawl

MCP server for web crawl data

5 followers

AI search and retrieval for web crawlers
mcp-server-webcrawl gallery image
mcp-server-webcrawl gallery image
mcp-server-webcrawl gallery image
Free
Launch tags:MarketingSEODeveloper Tools
Launch Team / Built With
Auth0
Auth0
Start building with Auth0 for AI Agents, now generally available.
Promoted

What do you think? …

Ben Caulfield
Maker
📌
As the developer of a web crawler myself, I struggled to connect to crawled data with MCP in a practical manner. Generalized solutions sort of worked, but needed guidance and/or a few queries just to get warmed up to the schema. They always seemed a bit behind the curve. mcp-server-webcrawl works with WARC, wget, InterroBot, Katana, and SiteOne. It understands it is a query engine on top of website data from the get-go, and effectively communicates this to the LLM. Using fulltext and field searching along with a full boat of Boolean operators, mcp-server-webcrawl can surgically filter anything you need from your crawl, be it HTML webpages, images, CSS, or HTTP metadata. Use it to tune up your website, for technical research, or for any use case where you require a dependable search and retrieval API over your crawled data.