Hey Product Hunt 👋 I built ChiselForge because I needed a crawler that could tell me more about a website than just what was visibly rendered on the page. It originally started as infrastructure for a larger research/data-collection problem. While building it, I kept running into information that conventional page extraction could easily overlook: metadata, structured schemas, inline hidden content, linked resources, deeper site structure, and even pages that were still connected to a website despite not being part of the obvious public navigation. That pushed ChiselForge in a different direction. Instead of treating a website as a collection of pages to turn into text, ChiselForge tries to understand the broader web property and extract it into structured, usable data. The extraction path is deterministic-first. You provide a URL and schema, ChiselForge attempts to produce validated structured output, and heavier browser or LLM-assisted paths are used only where they’re actually needed. I’ve open-sourced it because I think there’s a lot more that can be explored here, especially for AI pipelines, research, web intelligence, data collection and agents that need reliable information from the web. It’s still evolving, so I’d genuinely love feedback from people building crawlers, agents, search systems or data pipelines. What would you want a crawler like ChiselForge to discover that current tools don’t give you easily? — Subrojit, maker of ChiselForge

ChiselForge

Open-source web intelligence beyond visible pages

Open-source web intelligence beyond visible pages