Can ChatGPT reach the service information you want buyers to find?
Separate search access, training preferences and user visits, then check what the evidence supports for your important pages.
Free AI Crawler Checker
Check whether AI crawlers can reach and read your page.
ChatGPT, Claude, Perplexity, Google and Bing read websites through crawlers: automated visitors that fetch pages. Enter one page to see what your robots.txt file allows each crawler, how your server answers requests made in their names, and how much text arrives without JavaScript.
Free. No email needed to see your result. We do not keep the address you check.
What the checker looks at.
Four checks, run on the page you enter. Every result names what we saw, so you can confirm it yourself.
-
Robots rules for each crawler
We read your robots.txt file, the public file that tells crawlers which pages they may fetch, and apply its rules to your page for eleven crawler names. We follow the published robots standard: the longest matching rule wins, and a tie goes to Allow.
If it needs work: If a crawler you want is blocked, change the matching Disallow line or add an Allow line for it, then check again. OpenAI and Perplexity say changes can take about 24 hours to reach their systems.
-
How your server answers crawler names
We request the page under five crawler names, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and GPTBot, and once under our checker’s own name. Every request also names our checker, so your records show where it came from. Then we compare the answers.
If it needs work: If a crawler name is refused or challenged while our own request is not, look for a firewall or bot protection rule. Real crawlers come from published or verified addresses that our requests cannot use, so confirm with your firewall or CDN records before you change anything.
-
Text without JavaScript
We count the words in the HTML your server sends, before any JavaScript runs. Google renders JavaScript before indexing. A December 2024 analysis by Vercel and MERJ found that the crawlers of OpenAI, Anthropic and Perplexity did not.
If it needs work: If the count is low but the page looks full in a browser, its text is added by JavaScript. Ask your developer to send the main text in the HTML, for example through server rendering or prerendering.
-
Page signals
We record the page’s answer code and redirects, any noindex or snippet limits in its HTML or headers, and the main address it names in a canonical tag. Google says a page must be indexed, and eligible to show a snippet, to appear as a supporting link in AI Overviews or AI Mode.
If it needs work: Remove noindex from pages you want found. Lift snippet limits where you want AI features to quote the page. Point the canonical tag at the page’s own address, unless another page really is the main version.
How we label results
- Fix
- Something stops crawlers reaching or reading the page: a robots.txt block on a search crawler, an error answer, a noindex instruction, a robots.txt file that does not load, or fewer than 50 words without JavaScript.
- Review
- Worth a look, depending on what you intend: a crawler name your server treats differently, 50 to 199 words without JavaScript, snippet limits, a canonical tag naming another page, a block on a fetcher used at a person’s request, or a Google-Extended block.
- Fine
- We found nothing in the way for this check.
- Your choice
- A training control, set as you prefer. OpenAI and Anthropic document their training crawlers separately from their search crawlers.
The 50 and 200 word marks are our thresholds for a first look, not a published standard.
The crawlers we check
Eleven names, grouped by what a block would change. Each row links to the company's own documentation.
Search and answers
These crawlers fetch the pages that search results and AI answers draw on. If you want to be found, allow them.
| Crawler | Used for | If robots.txt blocks it |
|---|---|---|
| OAI-SearchBot ChatGPT search, by OpenAI | Finds pages to show in ChatGPT search answers. | OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. Source: OpenAI: Overview of OpenAI crawlers |
| Claude-SearchBot Claude search, by Anthropic | Indexes pages to improve Claude's search results. | Anthropic says blocking it prevents your content being indexed for Claude's search, which can reduce your visibility there. Source: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? |
| PerplexityBot Perplexity search | Finds and links websites in Perplexity's search results. Perplexity says it is not used to collect content for training AI models. | Perplexity recommends allowing PerplexityBot so your pages can be surfaced and linked. Source: Perplexity: Perplexity crawlers |
| Googlebot Google Search | Crawls pages for Google Search, including AI Overviews and AI Mode. | Google cannot read a page Googlebot is blocked from. Google says a page must be indexed, and eligible to show a snippet, to appear as a supporting link in AI Overviews or AI Mode. Source: Google: List of Google's common crawlers |
| Bingbot Bing search, by Microsoft | Crawls pages for Bing search. | Bing cannot read a page Bingbot is blocked from. Source: Microsoft Bing: Verify Bingbot |
When a person asks an assistant to open your page
These fetch a page at a person’s request. Some do not follow robots.txt.
| Crawler | Used for | If robots.txt blocks it |
|---|---|---|
| ChatGPT-User ChatGPT, at a person’s request | Visits a page when someone using ChatGPT asks it to. | OpenAI says robots.txt rules may not apply to these visits, because a person started them. It also says ChatGPT-User does not decide whether a page appears in ChatGPT search. Source: OpenAI: Overview of OpenAI crawlers |
| Claude-User Claude, at a person’s request | Fetches a page when someone using Claude asks a question that needs it. | Anthropic says blocking it prevents Claude retrieving your content in response to a person's question. Source: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? |
| Perplexity-User Perplexity, at a person’s request | Visits a page to help answer a person's question in Perplexity. | Perplexity says this fetcher generally ignores robots.txt, because a person requested the visit. Source: Perplexity: Perplexity crawlers |
AI training and Gemini grounding
These are controls for how your content is used. Set them as you prefer.
| Crawler | Used for | If robots.txt blocks it |
|---|---|---|
| GPTBot OpenAI model training | Collects content that may be used to train OpenAI's models. | Blocking it tells OpenAI not to use your content for training. It is a separate control from OAI-SearchBot. Source: OpenAI: Overview of OpenAI crawlers |
| ClaudeBot Anthropic model training | Collects content that may be used to train Anthropic's models. | Blocking it asks Anthropic to leave the site's future content out of training. Claude-SearchBot and Claude-User are separate. Source: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? |
| Google-Extended Gemini training and grounding, by Google | A robots.txt control, not a separate crawler. It covers training of future Gemini models and grounding, which supplies content from Google's index to Gemini Apps and Vertex AI when they answer. | Google says blocking it does not affect inclusion or ranking in Google Search. It does limit grounding in Gemini Apps and Vertex AI. Source: Google: List of Google's common crawlers |
What this checker cannot tell you
- Whether an assistant will cite or recommend you. Access lets a crawler read a page; it does not decide which sources an answer uses.
- How your firewall treats the real crawlers. Our requests use crawler names but not their verified addresses, so a rule that checks addresses can treat the two differently.
- Whether the crawlers have visited. Your server or CDN logs can show that. OpenAI and Perplexity publish the addresses their crawlers use; Anthropic says it does not publish its IP ranges.
- Anything beyond this page at this moment. Rules can differ by page and change with your settings. We count words in the first 512 KB of HTML.
Sources, reviewed 12 October 2026
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: List of Google's common crawlers
- Google: AI features and your website
- Google: How Google interprets the robots.txt specification
- Google: Understand the JavaScript SEO basics
- Microsoft Bing: Verify Bingbot
- IETF: RFC 9309, the Robots Exclusion Protocol
- Vercel and MERJ: The rise of the AI crawler (17 December 2024)
Read the whole website from the outside.
The checker covers access for one page. A free Outside View looks at how buyers, Google and ChatGPT find and understand your business. Ten priorities arrive in a PDF within two business days of your request.
Free, with no call required and no obligation to buy.
No passwords, analytics exports or account access.
Your details are used to prepare and send the review, and to reply if you choose to continue.
Already want to discuss the work? Request a free Clarity Call
Questions
Is the AI Crawler Checker free?
Yes. You see the result on this page without giving an email address. To prevent abuse, one connection can run up to ten checks an hour.
Which crawlers does it check?
Eleven names from OpenAI, Anthropic, Perplexity, Google and Microsoft. They are the crawlers behind search and answers, the fetchers used when a person asks an assistant to open a page, and the training controls GPTBot, ClaudeBot and Google-Extended. We also request the page under five of those names to see how your server answers.
If every crawler is allowed, will AI assistants cite my site?
Not necessarily. Access lets a crawler read the page; it does not decide whether an assistant chooses it as a source. What the page says, and the evidence behind it, matter once access is in place. We do not promise citations or rankings.
Should I block GPTBot or ClaudeBot?
It is a business decision about AI training. OpenAI and Anthropic document their training crawlers separately from their search crawlers, so a site can block training and still allow OAI-SearchBot and Claude-SearchBot. Google-Extended is different. Google says it covers grounding in Gemini Apps and Vertex AI as well as training, while Google Search is unaffected.
What if my server answers a crawler name differently?
Your server answered a request made in a crawler's name differently from our checker's own request, for example with a 403 error or a security challenge. That can be a rule that also stops the real crawler, or protection that only stops impostors. Real crawlers come from published or verified addresses that our requests cannot use, so check your firewall or CDN records before changing anything.
Why does JavaScript matter?
Google renders JavaScript before indexing. A December 2024 analysis by Vercel and MERJ found that the crawlers of OpenAI, Anthropic and Perplexity did not render it. Text that appears only after JavaScript runs may not reach those crawlers.
Does the checker look at llms.txt?
No. llms.txt is an optional file, and Google says it ignores these files for Search. Our llms.txt guide in Labs explains when a file is worth keeping.
What happens to the address I enter?
Our server uses it to request your robots.txt file and the page, then shows the result in your browser. We do not save the address or the result. Short-lived counters, made from scrambled versions of your connection's address and the checked website's name, limit repeat checks.
How can I confirm real crawler visits?
Look in your server or CDN logs. OpenAI and Perplexity publish the IP addresses their crawlers use, so you can match a visit to their lists. Anthropic says it does not publish its IP ranges. Our guide to Cloudflare's AI bot settings shows how to keep the evidence.
Guides for what the result shows
Each one picks up where the checker stops.
Is Cloudflare blocking Googlebot? Check the evidence.
Record your crawling policy, identify what blocked a request, and separate a successful fetch from a search recovery claim.
Are AI bots reaching the pages that explain your services?
Separate repeated bot requests from coverage of important pages, then use the evidence to choose an access or measurement investigation.
llms.txt and SEO: what it does and how to check yours
Separate an optional agent-oriented file from search requirements, then check whether the file matches the pages it describes.
Found something in the way?
The result shows where to look: robots.txt, your firewall or the page itself. For a wider view of how buyers and AI assistants find your business, start with a free Outside View: ten priorities in a PDF, within two business days.
Get your free Outside View Ready to discuss the work? Request a Clarity Call