Hey 👋
AI assistants like ChatGPT, Claude, Perplexity, and Gemini are browsing and citing websites more and more when answering questions. But most of them still have to parse your entire HTML page to figure out what your site is about.
llms.txt is a markdown file at the root of your domain that gives AI models a summary of your site about what it does, and which pages matter most.
Geekflare llms.txt generator is a free tool to generate llms.txt file in seconds:
1. Enter your website or sitemap URL
2. We crawl your pages by respecting robots.txt and group them into logical sections
3. You can copy or download your `llms.txt` file, ready to publish
Would love for you to try it on your own site and let us know what you think 🙌
Love,
Chandan
Report
love how clean the api looks, especially the mcp server angle. one thing that would save me a ton of time is webhook support for long-running jobs, so my agents can move on and pick up results later instead of waiting on a polling loop.
honestly the no-headless-browser approach is such a smart move, basically sidesteps all the usual scraping headaches. love that the free tier actually lets you try the MCP server too.
@enayocakllsubz thank you for the feedback. Yes, we wanted to give as many features as possible in the free plan to test it out.
Report
It would be great if you could add scheduled scraping jobs that automatically push fresh content to a webhook or S3 bucket. Right now I have to manually re-trigger extractions to keep my dataset current, and automating that would make Geekflare fit much more naturally into a production pipeline.
@bayramzndzow4z thank you for the feedback. It is a great feature to add, and we will prioritize this. We will do the webhook first.
Report
Love how clean the API is and the fact that it sidesteps CAPTCHAs entirely. One thing that would make my life easier: a built-in scheduling option so I can set recurring scrapes on a cron and just get the diff delivered to a webhook. Right now I have to wire up an external scheduler to keep my data fresh, which feels like a missing piece for an otherwise solid tool.
Replies
Geekflare
love how clean the api looks, especially the mcp server angle. one thing that would save me a ton of time is webhook support for long-running jobs, so my agents can move on and pick up results later instead of waiting on a polling loop.
Geekflare
@semra266920 thank you
honestly the no-headless-browser approach is such a smart move, basically sidesteps all the usual scraping headaches. love that the free tier actually lets you try the MCP server too.
Geekflare
@enayocakllsubz thank you for the feedback. Yes, we wanted to give as many features as possible in the free plan to test it out.
It would be great if you could add scheduled scraping jobs that automatically push fresh content to a webhook or S3 bucket. Right now I have to manually re-trigger extractions to keep my dataset current, and automating that would make Geekflare fit much more naturally into a production pipeline.
Geekflare
@bayramzndzow4z thank you for the feedback. It is a great feature to add, and we will prioritize this. We will do the webhook first.
Love how clean the API is and the fact that it sidesteps CAPTCHAs entirely. One thing that would make my life easier: a built-in scheduling option so I can set recurring scrapes on a cron and just get the diff delivered to a webhook. Right now I have to wire up an external scheduler to keep my data fresh, which feels like a missing piece for an otherwise solid tool.
Geekflare
@melahatemicyh7 thank you for the feedback. We will add scheduling option very soon.