AI crawlers blocked by your host, and how to fix it.

Your robots.txt can say yes while your host says no. A CDN or bot-protection layer in front of your site answers each request before robots.txt is ever read, and several of them block AI crawlers by default. When that happens, Verand's robots.txt allows AI crawlers check passes and the AI assistants still never see a page.

The AI crawlers reach your pages check in Site Health tests for exactly this. It fetches your home page once as a normal browser, then once with each AI crawler's user agent, and compares the answers. If the browser gets the page and a crawler gets a 403, a 429 or a challenge page, the check fails and names the host where it can.

What the check tells you

• Pass. Every crawler got the same answer a browser did. This proves user-agent blocking only; see the limit below.
• Fail, high. A search or user-fetch crawler was refused: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot or Perplexity-User. Those are the crawlers that put your pages into AI answers, so this is the one that costs visibility.
• Fail, medium. Only training crawlers were refused, such as GPTBot, ClaudeBot or CCBot. That can be a deliberate choice.
• n/a. Your host refused the plain-browser request too, or did not answer. Nothing can be said per crawler, so the check reports nothing rather than a pass it did not earn.

Blocking training crawlers is a legitimate business decision. Blocking search and user-fetch crawlers is what costs AI-answer visibility. The guide below lets you decide; Verand recommends allowing all three for visibility.

Cloudflare

These are the exact steps, in the order they were performed on verand.ai on 1 October 2026, where Cloudflare's default policy had been refusing GPTBot, ClaudeBot, CCBot, Amazonbot and Bytespider, and refusing Claude-User 164 times in a week, without anyone having touched a setting.

1. Security → Settings → Configure AI bot policies. Set Search to Allow, Agent to Allow and Training to Allow. Switch Bot Preference Sync off. Save.
2. AI Crawl Control → Security. Tick Show inactive crawlers, then switch every Block Crawler toggle off.
3. Security → Settings → Client side abuse. Switch Email Address Obfuscation off. It rewrites every email address on your pages so crawlers read "[email protected]" where your contact details should be.
4. Leave Bot fight mode and AI Labyrinth off.

Training is the setting that matters most: Cloudflare files GPTBot, ClaudeBot, CCBot, Amazonbot, PetalBot, Google-CloudVertexBot and, oddly, Claude-User under training, so "Training = Disallow" silently refuses a user-fetch crawler too.

🎬 Screen recording: cloudflare-ai-bot-policy.mp4
Dan walks through steps 1 to 4 on a live Cloudflare dashboard. Link to be added.

Re-run the audit afterwards from Scheduled Tasks → Deep Full-Site Crawl → Run now, or wait for the next scheduled crawl. The check re-tests every crawler. You can start a crawl by hand once every 6 hours per site, and 4, 8, 15 or 30 times a month across your sites on Starter, Growth, Professional or Agency (2 during a trial).

Other hosts

Cloudflare is the most common cause because it changed its default: since 1 July 2025, a newly added Cloudflare domain blocks known AI crawlers unless its owner chooses otherwise. It isn't the only one. What each host says it does by default, from its own documentation where it publishes one:

Bot-management products block wherever their owner has set them to:

• Akamai, Imperva, DataDome. Each has a console with a category for AI crawlers or automated traffic. Find the rule that covers GPTBot or ClaudeBot and set it to allow, or add exceptions for the crawlers you want.
• A WordPress security plugin (Wordfence, Sucuri and similar) can block by user agent. Search its rules for "bot", "GPT" or "crawler".
• Your own firewall or .htaccess. Search for the crawler names from the finding.

If the check names no host, it couldn't identify the edge from the response headers. The finding still lists which crawlers were refused, which is enough to search your host's settings or ask its support.

Rate limiting: when we can read only part of your site

A different block, with a different fix. The check above is about AI crawlers reaching your pages. This one is about Verand's own crawler, the one that builds Site Health.

If Site Health shows a banner like "Cloudflare refused 1,405 of your 1,505 pages, so this crawl is partial", your host's rate limiting answered our crawler with HTTP 429 after a burst of requests. We crawl at one request per second and retry after the cooldown, which clears it on most hosts without you doing anything. When it doesn't, the scores you see are built from the pages we could read, and the checks that need the whole site (orphan pages, internal links, interlinking, cannibalization) are held rather than guessed.

The fix is one rule, the same idea on every host: allow requests whose User-Agent contains VerandBot, above the host's rate limiting and bot protection. Our full user agent is VerandBot/1.0 (+https://verand.ai).

• Cloudflare. Security → WAF → Custom rules → Create rule. Field User Agent, operator contains, value VerandBot. Action Skip, and tick Rate limiting rules, All remaining custom rules, Bot Fight Mode and Managed rules. Deploy. On the Free plan, Bot Fight Mode can't be skipped per rule; switch it off or upgrade.
• Vercel. Firewall → Custom rules → New rule. Condition User Agent contains VerandBot, action Bypass. Save.
• AWS WAF. Add a rule to your web ACL: single header user-agent contains VerandBot, action Allow, with a priority above your rate-based rule.
• Any other host. Send its support: "Please allowlist the user agent VerandBot/1.0 for rate limiting and bot protection on our site. It fetches at 1 request per second."

Then open the banner's Let our crawler through and press Test the rule: it fetches two dozen of your pages at full speed and tells you in seconds whether any were refused. When nothing is refused, Run the full crawl now appears, so you don't have to wait for the schedule. That crawl is the real proof: the banner, the Home card and the Get started task clear on their own once a crawl reads 90% or more of your site.

The same status lives on Settings → Integrations in the Site Audit Crawler Access card, under Site access: how many pages the last crawl read, the host that refused us if there was one, Copy user agent, and the same rule steps (Let our crawler through, or Allow rule steps when nothing was refused). It's there whether or not a crawl was partial, so you can add the rule before you need it.

The limit of this test, and why it is stated on a pass

The check sends each crawler's user agent from Verand's own servers. A host that recognises a crawler by its network addresses, as Cloudflare does for verified bots, can still refuse the real crawler while letting this test through. That is why a pass behind Cloudflare carries the line "Cloudflare can still block verified crawlers such as Claude-User by their network origin, which this test can't see", and why step 2 above is worth doing once even when the check passes.

Verand never claims that every AI assistant can reach your site. The check proves that user-agent blocking is not in the way; the host's own crawler log is the only complete answer.