TheCrawler

Web scraping with AI extraction included — we own the GPU

5 followers

Web scraping with AI extraction included — we own the GPU

5 followers

Scraping APIs bill you twice: once for the page, again for the AI that reads it. We bought GPUs instead - so TheCrawler scrapes any site to clean JSON/markdown with AI extraction INCLUDED. 1 credit = 1 page, no per-call AI surcharge, structurally impossible for metered rivals to match. Grounded extraction returns null instead of inventing values. Plus web search, async batch + webhooks, change-monitoring, and an MCP server for Claude Code/Cursor. 500 free pages, no card.

Free Options

Launch tags:API•Developer Tools•Artificial Intelligence

Launch Team

Tines The single, secure environment for agents, apps, and automations.

Promoted

Maker

📌

Hi Product Hunt 👋 I'm Manchit, and I built TheCrawler because scraping APIs bill you twice — once for the page, again for the AI that reads it. Their marginal cost is an OpenAI invoice, so they meter every extraction. I bought GPUs instead (128GB box running a Qwen MoE via llama.cpp). Extraction became a sunk cost, so I price it like a plain scrape: 1 credit = 1 page, AI included. That's the whole pitch — the rarest thing here isn't a feature, it's the cost structure. A metered competitor can't match it without re-architecting their COGS. What it does: scrape any URL to markdown/JSON (+ brand data, contacts, tables) · AI extraction to your schema with a grounding validator — every value must trace to the page text or it returns null instead of inventing · you only pay for valid results: /diagnose scores extractability before you spend a credit, and failed/empty extractions are auto-refunded · bring your own LLM — point extraction at any OpenAI-compatible endpoint (Ollama, llama.cpp, LM Studio) and your data never leaves your infra · web search · async batch with signed webhooks · change-monitoring watchlists · an MCP server so it works inside Claude Code and Cursor. Honest limits: no stealth/residential-proxy tier yet (heavily bot-protected sites will block it), answer-synthesis search is on the roadmap, MCP is stdio-only. Free tier is 500 pages, no card. I'd genuinely love your teardown - I'll be here all day answering everything.

Report

2mo ago

Maker

Hey Product Hunt 👋 Manchit here, maker of TheCrawler.

I built this because every scraping API I used billed me twice — once to fetch the page, then again in AI credits for the model that reads it. Those extraction bills were unpredictable and grew faster than my actual usage.

So I did the unreasonable thing and bought the GPUs. TheCrawler runs the extraction model on hardware we own — so structured JSON extraction is part of the scrape, not a separate metered line item.

What you get:

• One call: URL in → clean markdown or validated JSON out

• Page-grounded extraction — it returns null for anything it can't verify on the page, instead of inventing values

• A drop-in MCP server (claude mcp add) for AI agents, plus LangChain / LlamaIndex

• 500 free pages to try, no card required

It's day one and I'm here all day — genuinely want your feedback, edge cases, and "does it handle X?" questions. Throw a gnarly page at it; if it chokes, tell me and I'll fix it.

Thanks for checking it out 🕷️