Why AI search can’t find your site (and how to fix it fast)

AI-powered search tools like ChatGPT Search, Perplexity, and Google AI Overviews are reshaping how people discover content online. But what if your site is invisible to them? A recent audit revealed a widespread issue: websites using Cloudflare’s free tier often block AI crawlers by default, cutting off a critical channel for reach and influence. The fix is straightforward—but it requires knowing where to look.

The hidden barrier blocking AI from your content

Many developers trust Cloudflare to handle security and performance, but few realize its default settings can unintentionally block AI crawlers. Even when users disable the “Block AI bots” option, a separate feature called “Manage your robots.txt” may still inject directives that prevent AI agents from accessing the site. This happens because Cloudflare modifies the robots.txt file at the edge, adding rules that disallow access for AI crawlers like GPTBot, ClaudeBot, and Google-Extended.

Consider a recent audit of vibescoder.dev, a developer blog using Cloudflare. The site’s original robots.txt file was just 100 bytes, but Cloudflare served an expanded version totaling 1,838 bytes. The discrepancy included lines like:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

These rules effectively blocked AI search engines from indexing the site, despite the owner’s intent to maximize visibility. The issue stemmed from the “Instruct AI bots to not scrape content” setting in Cloudflare’s Security panel, which was enabled by default and added AI-blocking directives to the robots.txt file without updating the Content-Length header.

SEO vs. AEO: two discovery channels, one blind spot

Traditional search engine optimization (SEO) focuses on ranking in Google’s results, but Answer Engine Optimization (AEO) ensures content is discoverable by AI assistants and answer engines. While SEO prioritizes keywords and backlinks, AEO requires clear signals for AI agents, including structured data, canonical URLs, and properly formatted sitemaps.

The audit identified 20 issues across vibescoder.dev, ranging from critical to minor. Among the most severe were:

• - Blocked AI crawlers due to Cloudflare’s robots.txt modifications
• - Broken RSS feed links causing 15 dead endpoints
• - A missing sitemap.xml file returning 404 errors
• - Duplicate User-agent: * blocks in the robots.txt file

None of these issues were intentional, yet they collectively undermined the site’s visibility. The good news? All were resolved in a single session with 17 files changed and 428 lines added.

How to check your site’s AI discoverability in 60 seconds

If your site uses Cloudflare, follow these steps to ensure AI crawlers can access your content:

• - Navigate to Security → Settings in your Cloudflare dashboard
• - Disable the “Block AI Bots” setting if it’s enabled
• - Check the “AI Labyrinth” setting and disable it
• - Review “Manage your robots.txt” and select “Disable robots.txt configuration” to prevent Cloudflare from modifying your file

If you prefer to keep some AI-blocking directives, consider the “Content Signals Policy” option. This adds Content-Signal: ai-train=no to your robots.txt, which prevents AI models from using your content for training. However, this also reduces the likelihood that your work will be cited or referenced by AI tools.

Prepare for the rise of AI-native content

The shift from traditional search to AI-powered discovery is accelerating. Websites that rely solely on SEO may soon find themselves invisible to users querying AI assistants. To stay ahead, content creators and engineers should:

• - Audit their site’s AI crawler accessibility regularly
• - Implement structured data and canonical URLs for better AEO
• - Consider creating an llms.txt file to guide AI agents
• - Monitor Cloudflare’s AI-related settings, as defaults can change without notice

The tools and platforms we use today may not always align with tomorrow’s discovery methods. By proactively addressing these blind spots, creators can ensure their work remains visible in an evolving digital landscape.