Free AI citations check: see what ChatGPT, Google AI and Gemini say about your business. Claim your check

AI search

AI crawlers explained: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest

A year or two ago, a lot of websites added a rule to block the AI bots. It felt like the safe choice. Now the same owners want to know why ChatGPT never mentions them.

The Kihan Marketing AI crawler checker, a form that takes a website address and reads its robots.txt

One line in a small text file can keep a business out of every search answer ChatGPT gives, and nothing on the site will ever tell you. The same file, left wide open, lets your pages train models you may not want to feed. Both are choices. Most owners have made them by accident.

You cannot choose well until you know which crawler does what. Every crawler here is sorted by its job and quoted from its vendor's own page.

What are AI crawlers, and what do they do?

An AI crawler is a program that requests pages from your site on behalf of an AI company. The name it gives, such as GPTBot, is its user agent. Every one of them does one of three jobs, and the job decides what happens when you block it.

JobWhat it doesIf you block itExamples
TrainingCollects pages that may be used to train future modelsYour pages are left out of training. You can still appear in answersGPTBot, ClaudeBot, Applebot-Extended, and Google-Extended (which also covers the Gemini app, see below)
SearchBuilds the index an assistant searches when it answersYou stop appearing as a source in that assistant's search answersOAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot
User requestFetches one page because a person asked about itThe assistant may not be able to open your page for that personChatGPT-User, Claude-User, Perplexity-User

Training and search are separate switches.

What are GPTBot, OAI-SearchBot and ChatGPT-User?

They are OpenAI's crawlers, described on OpenAI's crawler page.

NameJobWhat OpenAI says
GPTBotTraining"Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
OAI-SearchBotSearch"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
ChatGPT-UserUser requestIt "is not used for crawling the web in an automatic fashion". Because a user starts these visits, "robots.txt rules may not apply".
OAI-AdsBotAd checksVisits only pages submitted as ads on ChatGPT, to check them.

OpenAI adds that a robots.txt change can take about 24 hours to reach its search systems.

What are ClaudeBot, Claude-SearchBot and Claude-User?

They are Anthropic's, described on Anthropic's crawler page. Anthropic says its bots honor robots.txt, including the Crawl-delay line, and do not try to get past CAPTCHAs.

NameJobWhat Anthropic says
ClaudeBotTrainingBlocking it "signals that the site's future materials should be excluded from our AI model training datasets".
Claude-SearchBotSearchBlocking it "may reduce your site's visibility and accuracy in user search results".
Claude-UserUser requestBlocking it "prevents our system from retrieving your content in response to a user query".

What are PerplexityBot and Perplexity-User?

Perplexity runs two, described on its crawler page.

NameJobWhat Perplexity says
PerplexityBotSearchIt is "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."
Perplexity-UserUser request"Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

What is Google-Extended, and does it affect AI Overviews?

Google-Extended is not a crawler. It is a name you can use in robots.txt to tell Google how it may use pages that Googlebot has already crawled. Google's crawler list says it "doesn't have a separate HTTP request user agent string".

NameWhat it controlsWhat Google says
GooglebotCrawling for Google Search, which includes AI Overviews and AI Mode"AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search."
Google-ExtendedTraining of Gemini models, and grounding in the Gemini app and Vertex AIIt "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".
Search generative AI controlWhether your pages appear in AI Overviews, AI Mode and the generative AI features in Discover. A setting in Search Console, not a crawler"This control doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended."

So blocking Google-Extended does not take you out of AI Overviews. It can keep your pages out of what the Gemini app draws on. To stay in Google's AI answers, Google's AI features page says a page must be indexed and able to show a snippet, with no extra technical requirements. The Search Console control, which Google says had reached all websites by August 31, 2026, includes a site by default.

Which other AI crawlers visit a website?

Bingbot, Applebot, Amzn-SearchBot, DuckAssistBot and MistralAI-Index are search crawlers, the kind that affects whether you appear in answers. The rest train models or fetch single pages.

JobCompanyNameWhat the company says
SearchMicrosoftBingbotBuilds the Bing index. Bing's crawler list names no separate Copilot crawler.
SearchAppleApplebotPowers search in Siri, Spotlight and Safari.
SearchAmazonAmzn-SearchBotMakes content eligible for search experiences such as Alexa. Not used for training.
SearchDuckDuckGoDuckAssistBotCrawls "in real-time for our AI-assisted answers". "This data is not used in any way to train AI models."
SearchMistralMistralAI-IndexIndexes content for Mistral's search. Not used for training.
TrainingAppleApplebot-ExtendedIt "does not crawl webpages". It lets you opt out of training Apple's models.
TrainingAmazonAmazonbotIt "may be used to train Amazon AI models".
TrainingMistralMistralAI-TrainingBuilds datasets for training Mistral's models.
Open archiveCommon CrawlCCBotSays it checks robots.txt before it fetches a page, and that it obeys the Crawl-delay line.
User requestAmazonAmzn-UserStarted by a user, so it "may not follow all robots.txt directives".
User requestMistralMistralAI-UserVisits a page when a user asks a question. Not used for crawling or training.

Source: each company's own page.

How does Microsoft Copilot decide what it may use?

Through Bing. Since September 2023, Microsoft has read two page tags: NOCACHE limits Bing's AI answers to your URL, title and snippet, and NOARCHIVE keeps a page out of those answers. Both leave normal Bing results alone.

That is changing. On October 30, 2026 Microsoft announced new preferences for AI training and for grounding, which it will honor from February 1, 2027. After March 1, 2027 the old two tags will no longer be read for this purpose. The new settings default to allow, so a site that relies on the old tags needs to publish the new ones. Microsoft says the new settings "do not change eligibility for traditional Bing search results".

Should you allow or block AI crawlers?

For a business that wants customers to find it, our view is simple: allow the search crawlers and the user request fetchers. They are how you get into answers. Blocking them hides you from the people asking.

Training is a real choice, and reasonable owners land on both sides. If your pages are your product, such as paid research or original reporting, you may want training off. If your pages exist to win customers, there is little to protect and some reason to be known.

If you areSearch crawlersUser request fetchersTraining crawlers
A local business that wants leadsAllowAllowAllow, or your call
A publisher selling its contentAllow, if you want citationsYour callMany choose to block
A site that wants no AI use at allBlockBlockBlock
The frequently asked questions section of the Capital Park Apartments website
Being allowed in is only the start

A crawler that gets in still needs something to read. The home page of Capital Park Apartments, a Kihan Marketing client, states rents, floor plans and answers to renter questions as plain text. Google's AI Overview for "1 bedroom apartments olympia" linked to it.

How do you allow or block AI crawlers in robots.txt?

robots.txt is a text file at yoursite.com/robots.txt. Each group names a crawler and says what it may fetch. If the file names no AI crawler and has no rule that blocks everyone, they are all allowed already.

Appear in AI answers, but opt out of training

# Search and user requests: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Allow everything, the simplest file

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

There is also a new, shared way to state a training preference: a draft signal, written as a single group named __usage_ai-training. Anthropic says it began recognizing a shared training signal on October 30, 2026. Microsoft will honor it from February 1, 2027, and Google's crawler page says it intends to support it in early 2027. Anthropic says a rule that names ClaudeBot still wins over the general one.

The shared training signal

User-agent: __usage_ai-training
Disallow: /

What can robots.txt not do?

  • It is a request, not a lock. The companies above say they follow it. A crawler that chooses to ignore it will not be stopped by it.
  • User request fetchers may skip it. OpenAI, Perplexity and Amazon each say their user-triggered fetcher may not follow robots.txt, because a person asked for the page.
  • Names can be faked. Common Crawl warns of crawlers "falsely identifying themselves as CCBot". Most vendors publish their IP ranges so a server can check.
  • It does not remove what was already collected. Anthropic's wording is about "future materials". Blocking today does not reach back.
  • Changes are not instant. OpenAI says about 24 hours and Perplexity up to 24. DuckDuckGo says 72.

How do you check what your site allows today?

Do the check today, before you change anything else on the site. If a search crawler is blocked, that one line is costing you more than any page you could write this month.

AI crawler checkerFree. Type your domain and see which AI crawlers your robots.txt allows or blocks.

Questions about AI crawlers

What is GPTBot?

GPTBot is OpenAI's training crawler. OpenAI says it crawls content that may be used to train its generative AI models, and that disallowing it signals a site's content should not be used for that training. It is separate from OAI-SearchBot, which decides whether a site appears in ChatGPT search answers.

Does blocking GPTBot remove my site from ChatGPT?

No. OpenAI says each setting is independent. ChatGPT's search answers use OAI-SearchBot, and a rule that names only GPTBot does not apply to it. Blocking OAI-SearchBot is what removes a site from ChatGPT search answers.

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler. Anthropic says restricting it signals that a site's future materials should be left out of its AI model training datasets. Claude's search uses Claude-SearchBot, and user requests use Claude-User.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect a site's inclusion in Google Search. AI Overviews are part of Search and depend on Googlebot. Google-Extended controls training of Gemini models and grounding in the Gemini app and Vertex AI.

Should a small business block AI crawlers?

Usually not the search crawlers or user request fetchers, because they are how a business appears in AI answers. Whether to allow training crawlers is a choice. A business whose pages exist to win customers has little reason to block them.

How do I see which AI crawlers my site blocks?

Read your robots.txt at yoursite.com/robots.txt, or use Kihan Marketing's free AI crawler checker, which reads the file and lists each crawler as allowed or blocked. Also ask your host about firewall rules, which do not show in robots.txt.

An apartment community website shown on a phone

What I appreciate most is that they understand the property management industry, not just marketing.

MichaelProperty management company
Kihan Marketing

See who AI recommends in your market

Send us your website. We ask AI models what your buyers ask, then show you who gets named. The check is free.

Or call (425) 954-3452 or email info@kihanmarketing.com.