Cloudflare can block Googlebot through a training block or another security rule. Check the current policy and the actual request before changing either. Disallow AI Training preserves supported search access, but it does not override every other restriction.
Cloudflare changed its AI crawler controls on September 15, 2026. The final rollout introduced Disallow AI Training, which differs from Block. Older articles describe an earlier plan, so start with the setting currently recorded for your domain. Cloudflare’s launch announcement
For a service business, the practical question is whether a genuine crawler can retrieve the public pages prospective clients need to find. A training preference, a dashboard bot name and a successful browser visit each answer only part of that question.
This guide follows documentation checked on October 8, 2026. We have not inspected your Cloudflare account. Use the access workbook to keep policy, observations and unresolved questions together.
Start with the three decisions
Decide what your business wants before changing a control.
| Category | Decision to record |
|---|---|
| Search | Should crawlers collect content for a search index? |
| Training | Should content be available for model training or fine-tuning? |
| Agent | Should an automated visitor fetch or use a page on a person’s behalf? |
These categories describe different uses. Cloudflare now offers separate controls, with proposed defaults that depend on whether a new domain carries advertising. Those defaults are a starting point, not evidence of your account’s current configuration. Cloudflare’s control overview
| Training selection | Meaning for access |
|---|---|
| Allow | Adds no training block; other controls still matter. |
| Disallow AI Training | With Bot Preference Sync, publishes supported preferences; retains Accountable mixed-use search access and blocks other training crawlers. |
| Block | Blocks training crawlers, including mixed-use Googlebot, Bingbot and Applebot. |
| Block on pages with ads | Applies that block on pages Cloudflare detects as serving ads. |
Disallow is available for Training, not Search or Agent. There is no ads-only Disallow option. “Accountable” is Cloudflare’s designation and includes operator commitments alongside existing capabilities. Final setting definitions
Check the migration before assuming an outage
For domains that had selected legacy Block or Block on pages with ads without configuring granular controls, the final rollout migrated those settings to Search Allow, Training Disallow AI Training and Agent Block on pages with ads. Previously configured granular Training blocks also moved to Disallow; existing granular Search and Agent choices remained. That migration is why an old screenshot is insufficient evidence of what is blocking a request today. September migration tables
Select the correct domain and find Security Settings > Configure AI bot policies. Record all three values and Bot Preference Sync, with a timestamp. Cloudflare’s settings documentation still contains older rollout wording, so use the final announcement for behavior and the labels available in your account for navigation. This is a documented path, not a walkthrough we have verified inside your account. Settings documentation
Read the robots.txt actually returned for the affected hostname. Bot Preference Sync adds generated preferences while preserving existing directives. Existing restrictions therefore still need review; a category selection does not describe every custom rule. If Sync is off, do not assume it is publishing your preferences. Save the returned file alongside your record. Bot Preference Sync
Collect one request you can explain
First confirm the affected hostname is proxied through Cloudflare. These controls apply to requests passing through its proxy; a DNS-only hostname needs its own hosting investigation. See the AI Crawl Control setup requirements.
Choose one affected public URL, such as a service page, and an unaffected comparison. Include the exact hostname, path and time window. A working homepage does not establish that the service page, sitemap or robots.txt is accessible.
For the AI crawlers, our free AI Crawler Checker requests that URL under the names OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and GPTBot, and shows what the returned robots.txt allows each one: a first observation for the record, not an identity check. It never requests a page as Googlebot, so Google’s access still needs the authenticated request described below.
In the current navigation, Analytics > Events shows security events. Filter around the reported time and look for the action, product and rule. Records can be sampled, and one request can produce several events. An empty result is inconclusive. Use available traffic or server records to investigate successful requests too. Security Events
| Record | What it helps establish |
|---|---|
| URL and timestamp, including timezone | Whether observations describe the same request. |
| Claimed crawler and identity check | Whether the request came from the named operator. |
| Edge action, status, product and rule | Whether Cloudflare took a blocking action and which control did it. |
| Origin status and returned content | Whether the origin refused access or the response contained the expected page. |
Origin status may be unavailable, including when a response is served from cache. Record “unknown” rather than filling the gap with an assumption. Security Analytics
Verify the crawler before naming the cause
Cloudflare’s Free-plan AI Crawl Control identifies crawlers using user-agent strings. A row labelled Googlebot therefore needs more evidence before it supports a claim about Google’s access. Changing a command-line request’s user-agent to Googlebot tests that string’s treatment, not a genuine Google request. AI Crawl Control setup
For Google, obtain the original client IP from the request record. Match it against the appropriate published crawler ranges, or follow Google’s reverse-DNS check and then confirm that the hostname resolves back to the same IP. An origin log containing only a Cloudflare proxy address cannot complete that check. Keep sensitive log exports private. Google request verification
For Bing, use its Bingbot verification guidance. For OpenAI, distinguish OAI-SearchBot for search, GPTBot for training and ChatGPT-User for user-triggered visits; the official crawler documentation provides the corresponding address lists. These identities answer different access questions.
Google-Extended is different again: it is a robots.txt product token with no separate HTTP user-agent. It controls both future Gemini model training and grounding, which supplies source content to answers in Gemini Apps and Vertex AI. It does not determine inclusion or ranking in Google Search. A business seeking Gemini discovery should weigh the grounding effect as well as its training preference. Do not invent a Google-Extended fetch test. Google crawler documentation
A fictional record with an unresolved cause
The following example is entirely synthetic. It is not a customer incident, a live account inspection or a measured recovery. The worked CSV retains the same unresolved fields.
| Field | Fictional record |
|---|---|
| Page and time | https://consultancy.example/services/operations/, October 8, 2026, 10:15 UTC. |
| Proxy status | Not recorded. Confirm whether the request passed through Cloudflare. |
| Business intent | Allow search, disallow training, allow agents. |
| Recorded configuration | Search Allow; Training Disallow AI Training; Agent Allow; Sync on. |
| Team’s initial claim | “Cloudflare blocked Googlebot.” |
| Evidence behind the claim | A Free-plan crawler label and a request reported with HTTP 403. |
| Candidate event in the illustration | Same URL and time window; block attributed to a custom country restriction. Exact request correlation is incomplete. |
| Missing identity evidence | Original client IP and provider authentication have not been obtained. |
| Other missing evidence | Origin response and returned body have not been established. |
| Alternative explanation | The request may be impersonating Google; a separate origin refusal may also exist. |
| Proposed next action | Authenticate the request and correlate the exact event before proposing a scoped rule amendment. |
| Change and retest | No change completed. Rule owner review, rollback plan and genuine-crawler retest pending. |
The defensible conclusion is limited: the illustration contains a reported denial and an attributed custom rule, but it does not establish that genuine Googlebot was denied.
If identity and exact event correlation establish that genuine required Googlebot traffic was blocked, review the identified rule. If the request impersonated Google, this observation does not justify a Googlebot exception. Missing evidence means collecting the missing record first.
That distinction changes the next action. Turning every AI category to Allow would not resolve the missing identity evidence. A request already blocked by an earlier WAF rule can remain blocked despite a per-crawler Allow selection. Cloudflare’s WAF interaction documentation
Propose a narrow change, then retain the retest
When authenticated request evidence identifies a restriction that conflicts with the business decision, record the exact proposed change, the responsible owner, the previous value and the rollback condition. Keep “proposed,” “applied” and “verified” separate. A broad security bypass is not a substitute for identifying the responsible rule.
After an authorized change, inspect the affected URL again. Save the response status and expected content, then correlate subsequent genuine crawler traffic. If that traffic has not occurred or the available logs cannot establish it, leave the record pending.
A Search Console live test provides a useful additional observation, but it uses Google’s inspection tool. Record that identity separately from natural Googlebot crawling. A successful test also does not establish indexing. URL Inspection documentation
Likewise, a 200 response is not an indexing result. Continued errors can affect crawling and indexing, but a single request cannot predict a removal or recovery date. Google’s HTTP status guidance
Keep the Bing limitation visible
As of this review, Cloudflare says Bing’s domain-level robots.txt no-training support is targeted for early 2027. Disallow AI Training does not yet automatically convey that preference to Bing through robots.txt. Cloudflare’s Bing explanation
Do not add NOARCHIVE automatically as a workaround. Bing’s published guidance attaches consequences for Bing Chat answers and links as well as future training. A business seeking discovery needs to weigh both effects. Bing’s content-use controls
Download the access workbook
Download the blank access CSV, the fictional worked CSV or the Markdown workbook with instructions. The files retain policy, request identity, rule evidence, unknowns and the retest status. They do not scan your website or change its settings.
Use the record to choose the next investigation
If the record identifies an access restriction, bring the URL, time window and rule evidence into a technical SEO review. If access is successful but traffic is down, continue with the Google traffic drop diagnosis. For the separate question of search access in ChatGPT, use the ChatGPT search readiness guide.
For a free Clarity Call, bring the affected page and the conclusion your evidence currently supports. Keep private logs in your team’s controlled records until an appropriate review is arranged. “Identity unverified” is a useful starting point. It prevents an uncertain diagnosis from becoming an unnecessary configuration change.