# Cloudflare crawler access workbook

Version: 2026-10-08. Documentation reviewed: 2026-10-08. Companion files: `cloudflare-access-blank.csv` and `cloudflare-access-worked.csv`.

This workbook records an investigation. It does not scan a site, authenticate a crawler, change Cloudflare or verify indexing. Keep private logs and client IP addresses in controlled storage; use references here. The worked example is entirely fictional and contains no customer outcome or successful test.

## How to use the files

1. Copy the blank record for one affected URL and request time. Keep the exact hostname and timezone. Confirm whether the hostname is proxied through Cloudflare; DNS-only traffic requires a separate hosting investigation. Add a comparison URL when useful.
2. Record business intent separately from current Search, Training and Agent values. Capture Bot Preference Sync and the returned robots.txt. An unknown value must stay unknown.
3. Retain the reported request and any matching security event. Authenticate the original client IP using the operator's official method. User-agent names alone are insufficient.
4. Correlate URL, time, request reference, edge action/status and the responsible product/rule. Check the origin status and body where evidence is available.
5. Record alternative explanations and the narrow conclusion the evidence supports. Propose a change only when the responsible restriction conflicts with the intended access.
6. Assign the rule owner and retain the prior configuration and rollback condition. Keep proposal, application and verification as separate states.
7. Retest and retain the exact tool or crawler identity. A live inspection test, a natural crawl and an indexing observation are separate evidence. Missing subsequent traffic remains pending.

## Evidence status vocabulary

| Status | What it permits you to say |
| --- | --- |
| Not recorded | The field has not been investigated. |
| Unknown | Evidence was sought but is insufficient or unavailable. |
| Policy observed | A current setting or returned robots directive was captured. Actual access remains separate. |
| Identity unverified | A crawler label or user-agent was seen without authentication. |
| Request blocked | A request was exactly correlated with its blocking action and rule. Its claimed operator remains a separate identity check. Claim a verified Googlebot block only when both are established. |
| Fetch succeeded | The identified request returned the expected content. Indexing remains separate. |
| Test tool succeeded | The named inspection tool fetched the URL. Do not generalize to natural crawling. |
| Proposed / applied | Record the corresponding action only; neither means a successful retest. |
| Retest pending | Adequate subsequent evidence is absent. |
| No observation | Logs are missing, sampled, expired or contain no matching observation. This is inconclusive. |

The CSV example prefixes statuses with `synthetic` so it cannot be mistaken for a real investigation. A real record should use the evidence states appropriate to its observations and preserve the private supporting records.

## Blank record

| Field | Your record |
| --- | --- |
| Record type and review date | |
| Affected hostname and URL | |
| Proxy status and supporting observation | |
| Comparison URL | |
| Request timestamp and timezone | |
| Intended Search / Training / Agent access | |
| Current Search / Training / Agent selections, capture reference | |
| Bot Preference Sync and returned robots capture | |
| Initial claim and who made it | |
| Claimed crawler / user-agent | |
| Private original-client-IP reference | |
| Identity verification method, date and result | |
| Edge action and HTTP status | |
| Product / rule and private event reference | |
| Exact request correlation | |
| Origin status and returned content | |
| Unresolved alternative explanations | |
| Supported conclusion and evidence status | |
| Conditional proposed change | |
| Rule owner | |
| Previous value and rollback condition | |
| Change status and time | |
| Inspection test identity, URL, time and result | |
| Natural crawler retest evidence | |
| Separate indexing evidence | |
| Retest status and next review | |
| Source URLs and review date | |

## Fictional worked record

**All account details, timestamps, requests and events below are synthetic. No account was inspected, no change was made and no successful retest was observed.** The reserved `.example` address does not identify a customer. Primary documentation supports the method, not the invented event.

| Field | Fictional value |
| --- | --- |
| Record type / review | Fictional illustration, 2026-10-08. |
| Page | `https://consultancy.example/services/operations/` |
| Proxy status | Unknown. Confirm whether the request passed through Cloudflare. |
| Comparison | Not selected. |
| Time | 2026-10-08 10:15 UTC, invented for the example. |
| Business intent | Search allowed, training disallowed, agents allowed. |
| Recorded configuration | Search Allow; Training Disallow AI Training; Agent Allow; Sync on. These are fictional account values. |
| Robots capture | Unknown. No returned file supplied. |
| Initial claim | A fictional team member says Cloudflare blocked Googlebot. |
| Initial evidence | A Googlebot label in Free-plan AI Crawl Control and a reported HTTP 403. |
| Identity | Unverified. Original client IP and provider check are missing. No fake bot IP is supplied. |
| Candidate event | Same URL and time window; block attributed to a custom country restriction. Exact request correlation is incomplete. |
| Origin and body | Unknown. |
| Alternative explanations | The source may impersonate Google; a separate origin refusal may exist. These are hypotheses. |
| Supported conclusion | The fictional record describes a reported denial and custom-rule attribution. It does not establish a genuine Googlebot denial. |
| Proposed next action | Obtain identity and correlation evidence. If genuine required crawler traffic is implicated, have the owner review a scoped rule amendment. |
| Rule owner | Unassigned; review pending. |
| Previous value / rollback | Not captured. Define both before any approved change. |
| Change status | No change completed. |
| Inspection test | Not run. |
| Natural crawler retest | Pending. |
| Indexing | Unknown. |
| Retest status | Pending. No recovery claim. |

The next useful result could be an authenticated crawler denial, an impersonating request, or insufficient evidence. Do not force the record into a success/failure answer when the evidence supports uncertainty.

## Primary references

Reviewed 2026-10-08. Recheck behavior and available account controls when using this workbook later.

- [Cloudflare final September 15, 2026 rollout and migration](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)
- [AI Crawl Control setup and Free-plan detection limitation](https://developers.cloudflare.com/ai-crawl-control/get-started/)
- [WAF interaction and rule precedence](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/)
- [Security Events and sampling](https://developers.cloudflare.com/waf/analytics/security-events/)
- [Google request authentication](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [Google crawler identities and product tokens](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Search Console URL Inspection](https://support.google.com/webmasters/answer/9012289?hl=en)
- [OpenAI crawler roles and address lists](https://developers.openai.com/api/docs/bots)
- [Bingbot verification](https://www.bing.com/webmasters/help/verify-bingbot-2195837f)

No ranking, citation, referral, enquiry or recovery result follows from completing this workbook. Those outcomes require their own observations.