Created SupaCrawl.com this weekend for the ElevenLabs x 16z Worldwide Hackathon. Supacrawl was inspired by tools like Selenium and Puppeteer. Wanted to scale crawling websites for data extraction while making it into an easy to use API. The service simplifies crawling pages for LLMs by turning websites into clean, easy-to-read markdown. It’s a straightforward tool that gathers the web content you need, fast and hassle-free. Made with Nextjs and website crawling with Puppeteer. The service creates clean Markdown for LLMs and uses Elevenlabs to turn a webpage into audio automatically. This allowed us to use Elevenlabs to convert the result of the webpage into audio. Web scraping is very difficult to do at scale. Many websites have anti scraping measures set up. In the future we will improve access to more sites. We are going to continue to improve the types of websites we can crawl. Some websites block requests so we will create new ways to access them.

SupaCrawl

Feed your LLMs clean, easy-to-read markdown from any URL

Feed your LLMs clean, easy-to-read markdown from any URL