Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
456 followers
Free Local AEO & SEO Spider and a Markdown content extractor
456 followers
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Free Options
Launch Team / Built With
Render Sandboxes: a computer for every agentIsolated compute, persistent state and a real filesystem
Promoted
The Markdown extraction and multi-format export (JSON, CSV, Sitemap XML) make it especially useful for content audits and AEO workflows. Its real strength will depend on crawl depth accuracy, JavaScript rendering support, and how comprehensive its automated SEO checks are compared to established crawlers.
Finally, a crawler that doesn’t require allocating 16GB of RAM just to spin up! Screaming Frog is great for deep enterprise audits, but having a lightweight, Rust-based CLI tool that instantly dumps clean Markdown for LLM testing is exactly what the industry needs right now. This is bridging the gap between technical SEO and AI workflows perfectly. Great job shipping this!
Well done and congrats for the launch!
@weiss_arnaud Thank you Arnaud!
Here is my poor-mans open source alternative: https://github.com/pors/sitemapplus. It works for me, but PRs are always welcome
I've been looking to do this for my site and products! Exactly what I've been searching for - the AEO tools. Found one that was called "Profound". But will check out crawler for sure
你好， 相关seo的问题参考数据来源哪里？
Congratulations on the launch! A small but useful service.