... browser), and Training (absorbing your content into a model). Until now, the "Block AI bots" preset deliberately excluded crawlers that do more than one of those jobs. That exclusion is what kept Googlebot out of scope, because Googlebot crawls for search and for training with the same bot. On 15 September that exclusion goes away. Mixed-purpose crawlers that combine Search and Training get blocked by every configuration that blocks AI training, including the legacy preset. Cloudflare names the three ... ... care about: Googlebot, Applebot, BingBot. So if you turned that on and forgot, in 25 days you are telling Google not to crawl

... URLs in the sitemap, 225 indexed, checked this morning against the GSC coverage report. Most of the rest have an obvious excuse: too new, not enough internal links, wait it out. One doesn't. It was actually crawled by Google over a month ago (last crawl

... what Search Console shows: 294 indexable pages, 236 of which have never received a single impression. 230 URLs sit in "Discovered, currently not indexed", which is Google's own label for an address it knows about and has not crawled. Median position 28.5. Our working assumption was the usual one. Write more, publish more often, cover more queries. We never questioned it, because production was busy. Busy is not the same as working. When we finally looked at the serving ...

... blog posts flipped from "Submitted and indexed" to "Crawled, currently not indexed" this week. No edits, no redirects, nothing changed on my end. I went through everything GSC actually exposes: robots meta says index,follow, the canonical points to itself, HTTP status is 200. Technically clean by every check available. Google indexed them once, then quietly pulled them back out, with no error and no explanation anywhere in Search Console. Has anyone tracked a previously indexed page getting pulled like ...

... eight free browser tools as a public build-in-public SEO portfolio. Today I stopped doing homepage-only checks and crawled every sitemap URL across all eight sites. Audit scope: 743 URLs, HTTP status, redirects, canonical tags, noindex, titles, descriptions, H1s, duplicate text, and internal-link reachability. Two defects survived retry: 1. iWorkViewer had an Android Pages growth page in its sitemap but no internal link. I added a contextual homepage link. 2. ZonePlan listed two legacy meeting-planner URLs ... ... sitemap even though both canonicalized to the global planner. I removed both old URLs from the sitemap. Both repos were rebuilt, deployed through GitHub Actions, and verified on production. The post-fix crawl

... what we gotta do. Relentless : Do they stop and relax when they score a goal/basket/touchdown? No, they go harder. I will not take a single win for granted, greedy with winning... Unshakable: Do they crawl back into their shell after an airball? No. They demand the next shot. Jordan missed 26 game-winners. He kept asking for the ball. Bad demo, ghosted lead, feature that flopped, I'm taking the next shot. Short memory: They forget the last play instantly ...

... page built from original data , one of our teardowns where we read a few hundred real reviews and counted what people said. None landed on a page that sells anything. That matches what we see in the crawl logs too: assistants fetch our data-built pages roughly nine times more often than our product pages. Same pattern, two different instruments. So the headline finding is not that AI ignores you. It is that AI reads the pages where ... ... this is not a block. Something else was going on, and I found the explanation on OpenAI's own help page, in a section most people never open: "to be included it is important to allow OAI-Searchbot to crawl

... talking about brands, not brands talking about themselves. Which means the conversations happening today are the citations of tomorrow. If your brand isn't in them, you're not in the answers either. So we built Conversation Radar. It crawls 6 platforms (Reddit, Quora, LinkedIn, Hacker News, Medium, dev.to ), finds the live conversations relevant to your brand and category, tags each one as a Question, Discussion, Problem, or Comparison, and links you straight to the thread so you can join ...

... everyone actually wants, the Request Indexing button, has no API at all. So my daily job does everything the API allows, inspects up to 160 URLs, resubmits the sitemap, then writes a shortlist of URLs stuck in Discovered or Crawled but not indexed. The last step is me clicking Request Indexing by hand off that list. Has anyone found a legitimate way around this, or is the manual click just the toll Google charges? Comment from Abdullah Javaid(@abdullah_javaid3 ... ... Google fix. What changed how I read that clicklist is what the click actually buys. Discovered and not indexed is usually a judgement about the URL rather than a queue position, so a manual request buys one crawl

... Google Search Console showed far fewer indexed pages than sitemap URLs across eight small Next.js tools, so I crawled every URL rather than checking only the homepages. The 745-URL audit found 15 live pages with a placeholder canonical host and 22 sitemap pages with no internal crawl

... crisis is playing out across enterprise SEO. Aggressive programmatic AI initiatives are stalling, collapsing, or triggering manual penalties. This is not happening because Google hates AI content. It happens because these initiatives break the foundational mechanics of Google's crawl ecosystem, indexing thresholds, and quality controls. Google does not have infinite infrastructure. The most dangerous assumption in programmatic SEO is that publishing a page guarantees Google will evaluate it. Crawling

... just published a free interactive Technical SEO Audit Checklist on ToolMint. It includes 60 checks across 20 sections , covering: Crawling and indexing Robots.txt and XML sitemaps Canonicals and redirects Structured data Core Web Vitals JavaScript rendering HTTPS and security headers Search Console verification You can track progress, add notes, filter issues by severity, export the checklist as CSV or TXT, and print or save it as a PDF. No account is required, and your checklist data stays in your browser ...

... akarsh_hegde   @kaavya_prasad  The three-platform test controls the content variable, but I think the same confound can still eat it unless you also control for time. If all three go out simultaneously and Medium is crawled fastest, Medium wins every fetch on recency, and you've measured crawl

... optimize for LLM context windows and brand entity recognition (Answer Engine Optimization, or AEO). The problem? There was no quick, automated way to check if an LLM is actually recommending your brand, or if a rogue server-level crawl block in your robots.txt is quietly locking out AI search bots. So, I built a tool to solve my own headache. It's called AEO Audit Pro ! Here is what you can check in 30 seconds: AI Visibility Radar: Track exactly ... ... major engines (ChatGPT, Gemini, Perplexity) see and cite your brand. Knowledge Graph Entity Checker: Instantly map your site's semantic data and schema to see if search engines can logically connect your brand to your target keywords. Semantic & Crawl

... height first and only take targeted shots where they matter hero, mid-page, and footer. The data pipeline went from three separate passes over the page to one, cutting redundant work by ~2/3. When asking setup questions (export target + crawl scope), we now ask once instead of twice. Better analysis quality The extractor now captures data it previously missed: how often each font size actually appears, which spacing values dominate, button styling patterns, and how much area each background color ...

... properties including claude.ai , docs.anthropic.com , and anthropic.com ensuring our products and developer resources are discoverable, performant, and well-positioned in an evolving search landscape." The responsibilities include sitemaps, structured data (JSON LD), canonical tags, redirect strategies, robots.txt, Core Web Vitals, crawl efficiency, Google Search Console, log file analysis, international SEO, and "defining our strategy for emerging search experiences like AI Overviews and answer engines." This is not a company that thinks SEO is dead. This is a company that thinks ... ... role is not decreasing. What the job description actually says The role is a "high-impact, hands-on individual contributor role at the intersection of marketing, engineering, and data." It requires "deep understanding of how search engines crawl

... tools you don't give up rather than another one in the rotation. Curious which of yours are keepers versus glue. Comment from Adrian Witaszak(@adrian_witaszak): Mine, honestly: Linear for tracking, Claude for drafting and code, Stripe for anything involving money, Figma, and a notes app I keep abandoning and crawling

... good work! Comment from SaltyFishh(@saltyfishh): Good product. We are building ServBay, an AI-native local development base that simplifies web dev and empowers AI coding agents. What we're building: ServBay. It s a lightweight, native alternative to Docker on macOS that bundles 50+ dev services (databases, languages, web servers) with one-click setup, alongside a local-first AI Gateway and built-in MCP server. Who it's for: Indie hackers, solo developers, and dev teams who want ... ... Comment from Divv Saxena(@divvsaxena): @nolan_vu sure man :) I ll keep you updated about my app progress. Comment from Divv Saxena(@divvsaxena): @nolan_vu  Thanks man! I'm doing a lot of research with other competitors

... dominate the top rankings, but they all look and sound alike "If the humans are not involved anymore, what does this mean for the industry in the future?" asked Professor Thomas Reutterer, who led the research. A Graphite/Common Crawl analysis found that 86% of articles ranking high on Google Search are human-written. AI-generated content gets more impressions, but people still do not engage with it the way they engage with human-written content . The human premium ... ... things AI cannot generate. Real examples. Hard-earned insights. A point of view that is not just a remix of the top 10 results. What I am curious about Have you noticed your content becoming more similar to your competitors

... while ago, I noticed a huge frustration for many indie hackers: With the rise of AI search engines (like ChatGPT Search, Claude, and Perplexity), traditional SEO is shifting. Everyone is trying to figure out how to get their products crawled and cited by AI agents. To solve this, I built AI Tools Scoop. We heavily optimized our directory's technical semantic structure and Schema data so that AI crawlers can digest the listings instantly. And the results have been incredible ... ... directory pages are now being intensely crawled

... shipped multiple scrapers, Chrome extensions, and Apify actors for lead gen and e commerce, and I sell them through Fiverr, Gumroad, and other platforms. Most of my work is about turning manual grind workflows into simple tools: crawling data, cleaning it up with Python, and wrapping it in a clean UI so non technical sellers can use it. My latest product is Etsy Winner Finder a Chrome extension that sits on top of Etsy and turns the search results into ...

... After improving AI visibility signals, this started appearing: GPTBot visits ClaudeBot activity PerplexityBot crawling Clear AI discovery patterns This is the part most SEO tools completely miss. Here s a real dashboard snapshot TruboRankAI.com If you re not seeing this on your site, you re likely invisible ...

... Product Hunt I built LogicGapAI after watching agency BD reps spend 4 6 hours per prospect just to build a pitch. Crawl the site, format the findings, translate to business language, find the right contact, write the follow-up. By the time it was done, someone faster had already sent theirs. LogicGapAI collapses that into ~60 seconds. Paste any prospect URL and you get: Lighthouse performance scores (Performance, Accessibility, Best Practices, SEO) Gaps translated into revenue not rankings ...

... moments that matter most: The Career Leap: A CV maker that doesn t just look good it actively scans for jobs you qualify for and matches you instantly. The Entrepreneur s Edge: Our Business Plan maker uses AI to crawl the web for real-time market research, so you don't have to spend weeks on spreadsheets. The Cultural Connection: We ve built a collaborative design suite specifically for African, Indian, and European traditions. Custom, high-end invitations that feel ...

... just exposure, but the degree to which AI treats your brand as a credible source . Share of Voice (SOV): Represents your brand s voice share within AI narratives the probability that AI mentions your brand rather than a competitor when answering a topic-related question. Sentiment: Captures the emotional tone of AI mentions positive, neutral, or negative showing how AI portrays your brand. Together, these four metrics answer the two most fundamental GEO questions: How frequently does AI mention ... ... brand? and In what tone and context is the brand represented? 1.2 Trends and Competitor