Hey Everyone 👋 I built OpenFetcher to solve a problem I kept hitting while building RAG apps: getting clean, structured web data without overpaying or over-scraping. OpenFetcher is an LLM-native crawler that converts full domains into RAG-ready Markdown, even when sitemaps are broken or missing. It’s optimized for signal over noise, so your embeddings and agents actually work. ⚠️ Current Limitations (being transparent) Hosted version has page limits due to free-tier infra 🚀 Power users: run it locally If you want unlimited pages, faster crawling, and full control, you can deploy OpenFetcher locally or on your own server: No crawl limits Higher concurrency Perfect for large knowledge bases and internal docs This project is built in the open, and feedback means everything 🙏 Happy to answer questions, compare with Firecrawl/Jina, or help you plug it into your RAG pipeline.

OpenFetcher

Turn entire websites into RAG-ready Markdown — automatically

Turn entire websites into RAG-ready Markdown — automatically