Ben Caulfield

Ben Caulfield

software developer

About

I'm building an MCP server for websites, to connect your crawled website data to Claude Desktop via advanced search with support for field searching (HTTP status, headers, content, etc.) and Boolean operators. Markdown and search result snippet extras, effectively HTTP content distillations, are available for maximum token efficiency. Multi-crawler support is core to the server, which supports wget, InterroBot, Katana, and SiteOne crawlers, as well as the WARC web archive format.

Badges

Tastemaker
Tastemaker

Maker History

Forums

Ben Caulfield

8mo ago

mcp-server-webcrawl - MCP server for web crawl data

AI search and retrieval for web crawlers
View more