Launching today
Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
129 followers
Free Local AEO & SEO Spider and a Markdown content extractor
129 followers
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Crawler.sh
Hey Product Hunt! I built crawler.sh because I kept running into the same problem: every time I needed to audit a website's SEO or extract its content, I had to choose between bloated enterprise tools, slow cloud services, or stitching together a bunch of scripts.
crawler.sh is a single tool that does all three: crawling, SEO analysis, and content extraction, from the terminal or a native desktop app. It's built in Rust so it's fast, it runs locally so your data stays private, and it outputs standard formats so you can take the results anywhere.
@mehmetkose Congrats on your launch! You got me at bloated enterprise tools. After an SEO/AEO analysis, does this also offer suggestions to fix the detected issues?
Crawler.sh
@jerrybyday Thank you! and YES. You can see them on the UI or download them. I'm adding new features every day in our core modules to discover issues smartly
Loving the ease of this. Is there a team plan?
Crawler.sh
@jacklyn_i Thank you Jacklyn William I'm thrilled! I'm adding this on the roadmap! Will let you know when is implemented. Also the cloud functionality is on the way
@mehmetkose Awesome! I’d follow your account to stay connected on your journey as you continue to grow. Congrats on your launch again.
Rumora
Man, markdown extraction is such a lifesaver when dealing with giant websites. Way easier than my old 'copy-paste marathon' method. 😂
Crawler.sh
@djordjevic_nikola Oh that's exactly what i would like to hear. Give me some more :)) Also check it out that you can copy your SEO issues as a prompt so your llm can go fix.
Triforce Todos
Fast, private, and outputs standard formats, that’s how tools should be. Congrats on the launch 🙌