If I block GPTBot in robots.txt, do I disappear from ChatGPT?

No. GPTBot only collects content for model training. ChatGPT search is fed by OAI-SearchBot, a different bot with its own independent setting. You can block GPTBot and still be cited in ChatGPT if you allow OAI-SearchBot. OpenAI notes that robots.txt changes may take around 24 hours to be processed for search.

Do AI crawlers see content loaded with JavaScript?

No. According to the Vercel and Merj study from December 2024, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do not execute JavaScript: they read only the HTML your server returns, and they download JS files without running them. Only Gemini, via Googlebot infrastructure, and AppleBot render JS. Content that depends on client-side JavaScript is invisible to AI assistants.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended is not a crawler but a robots.txt token that controls whether your content can be used to train Gemini models and for grounding. Google states explicitly that it does not affect inclusion or ranking in Google Search. Crawling continues through Google's normal user-agents.

What is the difference between training, search and user bots?

Training bots such as GPTBot and ClaudeBot collect content to train models and account for roughly 80 percent of AI crawling according to Cloudflare. Search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so engines can cite you, and blocking them reduces your AI visibility. User bots such as ChatGPT-User fetch your page live when a person asks; this category grew more than 15x in 2025 according to Cloudflare Radar.

How much real traffic do AI bots send back to a website?

Little, but it varies a lot by sector. Cloudflare, measuring its own network, reported crawl-to-refer ratios — pages crawled per referred human visit — for a July 2025 window in its post of 29 August 2025: Anthropic crawled about 38,000 pages per referred visit, OpenAI about 1,100, Perplexity about 195 and Google 5.4. In a separate Cloudflare post of 28 August 2025 covering news publishers specifically, the ratios are much better: 152:1 for OpenAI and 32.7:1 for Perplexity. The GEO case rests on visibility and citations more than on click volume.

Do all AI bots respect robots.txt?

The declared crawlers from OpenAI, Anthropic and Perplexity say they respect it, and Anthropic also supports Crawl-delay. There are documented exceptions: Perplexity-User generally ignores robots.txt by design because a user initiates the request, and Cloudflare documented stealth crawling by Perplexity in August 2025 using undeclared user-agents and rotating IPs. Treat robots.txt as a courtesy request, not a technical barrier.

AI Crawlers Guide: GPTBot, ClaudeBot and robots.txt in 2026

AI crawlers are the automated bots that OpenAI, Anthropic, Perplexity, Google and Microsoft send to read your website for three distinct jobs: training models, building search indexes, and fetching pages live when a user asks. Each bot obeys its own robots.txt token, so blocking the wrong one can quietly remove your site from AI answers. And per Vercel’s December 2024 tests with Merj, none of the major AI crawlers execute JavaScript: what your server sends as HTML is what they see.

Why do AI crawlers matter for your visibility?

Because being crawled is the entry ticket: if an AI search bot never reads your page, ChatGPT, Claude or Perplexity cannot cite it. According to Cloudflare (July 2025), combined AI and search crawler traffic grew 18% between May 2024 and May 2025. GPTBot’s share of that traffic jumped from 2.2% to 7.7% — a 305% increase — while Googlebot rose from 30% to 50%, ClaudeBot fell from 11.7% to 5.4% and Bingbot stood at around 8.7%.

Most site owners have not reacted. In the same study, only 14% of 3,816 top domains analyzed had robots.txt directives specific to AI bots, and GPTBot was the single most-blocked bot (312 domains). Being crawled is step one; turning crawls into citations is a different problem, which we unpack in how AI engines choose what to cite.

What are the three types of AI bots?

Every documented AI crawler does one of three jobs: model training, search indexing, or live fetches triggered by users. The split matters because each type carries a different trade-off, and robots.txt lets you treat them separately.

According to Cloudflare (August 2025), model training accounts for roughly 79-80% of AI bot activity (up from 72% in July 2024), search for 17% and user actions for 3.2%. The user-action category is the fastest-growing: Cloudflare’s Radar 2025 Year in Review measured user-triggered fetching growing more than 15x during 2025, with ChatGPT-User peaking at up to 16x its level at the start of the year.

Here is the full list of documented bots and what blocking each one actually does:

Bot	Operator	Job	If you block it in robots.txt	Runs JS
GPTBot	OpenAI	Model training	Content excluded from OpenAI training datasets	No
OAI-SearchBot	OpenAI	ChatGPT search index	Your site can no longer appear in ChatGPT search results	No
ChatGPT-User	OpenAI	Live fetch on user request	May not apply — visits are user-initiated	No
OAI-AdsBot	OpenAI	Validates ad landing pages	—	Not tested
ClaudeBot	Anthropic	Model training	Future material excluded from Claude training datasets	No
Claude-SearchBot	Anthropic	Claude search index	Lower visibility in Claude search results	Not tested
Claude-User	Anthropic	Live fetch on user request	Claude can no longer fetch your pages during user conversations	Not tested
PerplexityBot	Perplexity	Search index (not training)	You can no longer be shown or linked in Perplexity results	No
Perplexity-User	Perplexity	Live fetch on user request	Generally ignored, per Perplexity’s own docs	Not tested
Google-Extended	Google	Token controlling Gemini training and grounding	No Gemini training or grounding; Google Search unaffected	n/a
Bingbot	Microsoft	Bing index, also reported to feed Copilot answers *	You leave Bing, and reportedly Copilot, at once *	Not tested

Purposes per each vendor’s official documentation (2026), except the Bingbot row (*): its Copilot link comes to us from Bing Webmaster Tools’ documentation via secondary sources — we could not open the primary page, so treat it as reported, not verified. The JavaScript column reflects Vercel’s late-2024 tests, with untested bots marked accordingly.

Which crawlers does OpenAI run, and what does each one control?

OpenAI documents four crawlers, each with its own robots.txt token, user-agent string and published IP ranges, per OpenAI’s crawler documentation (2026):

GPTBot (user-agent GPTBot/1.4, IPs published at openai.com/gptbot.json) collects content for model training. Disallowing it excludes your site from OpenAI’s training datasets.
OAI-SearchBot (OAI-SearchBot/1.4) powers search in ChatGPT. Allowing it is what lets your site appear in ChatGPT search results, and OpenAI notes it may take ~24 hours to process robots.txt updates.
ChatGPT-User (ChatGPT-User/1.0) fetches pages during user conversations. Because a human initiates each visit, OpenAI states that robots.txt may not apply to it.
OAI-AdsBot validates ad landing pages.

The settings are independent between bots. The practical consequence: you can refuse OpenAI’s training and keep full visibility in ChatGPT search — they are separate levers.

What do ClaudeBot, Claude-SearchBot and Claude-User each do?

Anthropic documents three crawlers, all of which respect robots.txt, per the Claude Help Center (2026). ClaudeBot collects web content that may be used to train Claude models, and it supports the Crawl-delay directive. Claude-SearchBot indexes pages to improve the quality of Claude’s search results. Claude-User accesses a page when a user’s question to Claude requires it.

Blocking each bot has a distinct effect and needs its own robots.txt entry. Anthropic says that once ClaudeBot is disallowed, future materials should be excluded from its model training datasets. Published IP ranges live at claude.com/crawling/bots.json, so you can verify that traffic claiming to be Claude really is.

Does Perplexity respect robots.txt?

Its declared crawler does; its user agent does not have to. Per Perplexity’s crawler docs (2026), PerplexityBot (PerplexityBot/1.0, IPs at perplexity.com/perplexitybot.json) indexes pages to show and link sites in results — not to train models — and respects robots.txt. Perplexity-User handles user-initiated visits and, in Perplexity’s own words, “generally ignores robots.txt rules” because a person originates each request.

There is also a documented credibility problem. On August 4, 2025, Cloudflare accused Perplexity of stealth crawling: an undeclared, generic Chrome user-agent generating 3-6 million requests per day on top of the declared crawler’s 20-25 million, rotating IPs and ASNs to evade blocks and ignoring robots.txt on test domains. Cloudflare removed Perplexity from its verified-bots list. The lesson generalizes: robots.txt is a courtesy request, not a technical barrier.

Is Google-Extended a crawler, and does Bingbot feed Copilot?

Google-Extended is not a crawler at all. It is a standalone robots.txt product token with no user-agent of its own — the crawling is done by Google’s existing user-agents. The token controls whether your content can be used to train future Gemini models (Gemini Apps and Vertex AI) and for grounding, per Google’s crawler documentation (2026):

“Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” — Google for Developers, 2026

Site owners have noticed how cheap that trade is: Cloudflare’s Radar 2025 Year in Review found that allow directives for Google-Extended tripled during 2025, even as GPTBot, ClaudeBot and CCBot led the list of bots most often blocked outright.

Bingbot is the opposite case — with a caveat about our own evidence. As reported second-hand from Bing Webmaster Tools’ documentation, a page we could not open directly to verify, the same Bingbot crawl feeds both the Bing Search index and Microsoft Copilot’s AI answers, and Microsoft publishes validated IPs in bingbot.json and recommends DNS verification. We flag it as reported rather than confirmed. What is documented first-hand is the official user-agent — a Chrome-like string containing bingbot/2.0, with a mobile variant — announced by Microsoft on April 28, 2022. If the shared-crawl account holds, then unlike OpenAI or Anthropic there is no separate lever here: Bing visibility and Copilot exposure come bundled.

How much do AI bots crawl — and how much traffic do they send back?

They crawl a lot and refer little. In late 2024, Vercel measured monthly volumes on its network: GPTBot made 569 million requests, Anthropic’s Claude 370 million, AppleBot 314 million and PerplexityBot 24.4 million — together about 20% of Googlebot’s 4.5 billion over the same period.

The same study found AI crawlers strikingly inefficient: ChatGPT spent 34.82% of its fetches on 404 pages and Claude 34.16%, against 8.22% for Googlebot. ChatGPT prioritizes HTML (57.70% of fetches) while Claude leans toward images (35.17%).

Referral traffic is the sobering part. Cloudflare’s crawl-to-refer ratios — pages crawled per referred human visit — come from two different posts that measure different cuts, so each column is labelled with its own:

Operator	All sites — Cloudflare, 29 Aug 2025 (Jul 2025 window)	News publishers — Cloudflare, 28 Aug 2025
Anthropic	38,066:1	2,500:1
OpenAI	1,091:1	152:1
Perplexity	195:1	32.7:1
Google	5.4:1	—

“…still crawled 38,000 pages for every referred page visit in July 2025.” — Cloudflare on Anthropic, August 29, 2025

Per Cloudflare’s industry breakdown (August 2025), ClaudeBot and GPTBot together account for nearly half of observed AI crawling, and in the news sector ChatGPT-User reaches 14.9% of crawling — a signal of heavy real-user demand. The strategic read for a small business: AI visibility currently pays in citations and brand presence, not in mass clicks.

Do AI crawlers render JavaScript?

No — none of the major AI crawlers execute JavaScript. A study by Vercel with Merj (December 2024) tested them directly:

“ChatGPT and Claude don’t execute JavaScript.” — Vercel, December 2024

Neither GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider nor CCBot rendered JS. Only Gemini — via Googlebot’s infrastructure — and AppleBot did. The bots still download JavaScript files (11.50% of ChatGPT’s fetches, 23.84% of Claude’s); they just never run them.

The implication is blunt: if your pricing, product descriptions or FAQs are injected client-side, they are invisible to AI assistants. Critical content must arrive in the server’s HTML response, via server-side rendering or static generation. This is the same discipline classic search rewards — see our technical SEO foundations — and it is the first thing we check in a technical SEO audit.

How should you configure robots.txt for AI bots?

Bot by bot, line by line — there is no single AI switch. Disallowing GPTBot blocks only OpenAI’s training crawler and leaves OAI-SearchBot untouched; disallowing ClaudeBot does not affect Claude-SearchBot or Claude-User. Both OpenAI and Anthropic warn that blocking their search bots reduces your visibility and citations in their answers.

A sensible starting point for most SMEs: allow search and user bots, then make a deliberate decision about training bots.

# Option A — default: allow everything (no AI entries needed)

# Option B — block model training, keep AI search visibility
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Option C — throttle training instead of blocking it
# (Anthropic documents Crawl-delay support for ClaudeBot)
User-agent: ClaudeBot
Crawl-delay: 1

Two operational details. First, OAI-SearchBot may take ~24 hours to process robots.txt updates, per OpenAI’s documentation. Second, each provider publishes IP lists (openai.com/gptbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json) so you can verify that a visitor claiming to be a bot is genuine. Note also that robots.txt governs who may crawl; the separate llms.txt file proposes what AI systems should read first — see our llms.txt guide.

What should you do this week?

A practical checklist for a small team:

Pull your server logs and search for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and bingbot. Baseline first, opinions later.
Audit your robots.txt against the table above. Confirm no search or user bot is blocked by accident — remember that only 14% of top domains had any AI-specific directive at all (Cloudflare, July 2025).
Decide your training stance. Blocking GPTBot and ClaudeBot costs no AI-search visibility: OpenAI and Anthropic keep search on separate levers. Google-Extended is a different trade — Google states it does not affect inclusion or ranking in Google Search, but the same token also switches off grounding, so it is the one training block with an AI-visibility cost attached.
Test one key page with JavaScript disabled or fetch it with curl. If the main content is missing from the raw HTML, AI crawlers cannot see it.
Re-check logs after 24-48 hours, since OpenAI says OAI-SearchBot may take around 24 hours to process robots.txt changes.

That sequence — baseline, implement, measure before and after — is exactly what we run in our AI visibility service, with every published number labeled as measured, calculated or assumed.