AI Visibility Benchmark · Crawler Access
AI Crawler Access Study: OAI-SearchBot, Claude, Perplexity, Bing, and Googlebot
Inspect the HTTP responses and robots flags recorded in the June 2026 sample, then distinguish those observations from actual platform visits and current access decisions.
Benchmark record
- Edition
- 2026.06
- Sample
- Reviewed 50-site sample
- Scope
- Robots, WAF, and challenge signals
Benchmark corpus
2026.06 Access Findings
| Recorded field or flag | Count | Percent |
|---|---|---|
| Primary service page returned HTTP 200 | 50/50 | 100% |
| Googlebot allowed | 50/50 | 100% |
| Bingbot allowed | 50/50 | 100% |
| OAI-SearchBot allowed | 50/50 | 100% |
| GPTBot allowed | 49/50 | 98% |
| ChatGPT-User allowed | 49/50 | 98% |
| ClaudeBot allowed | 50/50 | 100% |
| Claude-SearchBot allowed | 50/50 | 100% |
| Claude-User allowed | 50/50 | 100% |
| PerplexityBot allowed | 50/50 | 100% |
| Perplexity-User allowed | 50/50 | 100% |
| Noindex present | 0/50 | 0% |
| Nosnippet present | 0/50 | 0% |
These are stored permission flags and responses to the study’s collector, not authenticated visits by the named platforms. Robots-file responses were 200 for 48 records, 406 for one and 404 for one. Both non-200 rows still have all allowed flags set to true; the response-handling implementation is not supplied. The sample selected accessible sites and did not measure actual citations.
Access Quality Signals
| Signal | Count | Percent |
|---|---|---|
| Sitemap found | 48/50 | 96% |
| Key audited pages found in sitemap | 6/50 | 12% |
| Service page self-canonicalized | 34/50 | 68% |
| WAF block signal observed | 6/50 | 12% |
| CAPTCHA signal observed | 8/50 | 16% |
| JavaScript challenge signal observed | 0/50 | 0% |
The collector confirmed key pages in sitemap evidence for six records. Discovery was bounded to eight candidates, 20,000 URLs and an eight-second request timeout. A negative flag therefore does not establish absence from every sitemap. WAF and CAPTCHA flags are signals to inspect, not proof of a blocked platform request.
Bot Taxonomy
| Agent | Role |
|---|---|
| Googlebot | Google Search crawling and indexing. |
| Bingbot | Bing crawling and Microsoft search discovery. |
| OAI-SearchBot | ChatGPT Search surfacing. |
| GPTBot | OpenAI training crawler, tracked separately from search. |
| ChatGPT-User | User-directed ChatGPT fetches. |
| ClaudeBot | Anthropic training crawler. |
| Claude-SearchBot | Claude search retrieval. |
| Claude-User | User-directed Claude fetches. |
| PerplexityBot | Perplexity search/indexing surfacing. |
| Perplexity-User | User-directed Perplexity fetches. |
Current policies also differ for user requests: ChatGPT-User may not follow robots.txt and Perplexity-User generally ignores it. Anthropic describes its bots as honoring standard robots directives. These historical flags do not establish identical control behavior. The important principle is separation. If a business wants search and answer visibility, blocking a training crawler is a different decision from blocking a search/retrieval crawler.
Access Checks
For a current investigation, choose the checks relevant to the intended access policy. This list is not a claim that private server logs were part of the June dataset.
- robots.txt allow/block status
- HTTP status per target page
- WAF/CDN challenge signals
- CAPTCHA signals
- JavaScript challenge signals
- key page access through sitemap and internal links
- server-log evidence where available
- page-level noindex and snippet restrictions
- X-Robots-Tag status
- canonical target clarity
Common Access Mistakes
- Leaving an unintended search-crawler restriction in place when access is desired.
- Confusing training crawler consent with search/retrieval access.
- Using WAF rules that return 403, 429, CAPTCHA, or challenge pages to legitimate fetches.
- Important content missing from the intended crawler’s fetched or rendered result because of a rendering failure or restriction.
- Treating robots.txt as an indexing control instead of a crawl-control file.
- Publishing research statistics without accessible supporting records or explaining the method.
Source Notes
Current primary documentation and its review date are linked from the benchmark methodology. For a present-day investigation, use the ChatGPT access guide or important-page indexing guide.
OpenAI documents separate roles for OAI-SearchBot, GPTBot, and ChatGPT-User. Google documents that robots.txt controls crawler access and that noindex/snippet controls require crawlable pages. Anthropic documents separate ClaudeBot, Claude-SearchBot, and Claude-User agents. Perplexity documents PerplexityBot behavior for robots.txt.
Checklist Download
Access checklist: /downloads/ai-crawler-access-checklist-2026-06.csv
Related pages:
The Outside View
See where you stand with Google, AI and your buyers.
Send us your website. In two business days you have it in writing: how you look from the outside, who really competes with you, and the ten fixes that matter. Free.