--- title: "Technical SEO Foundations for AI Search in 2026" description: "No major AI crawler rendered JavaScript in Vercel's Dec 2024 study; within AI Overview queries a citation earns +120% more clicks per impression than none." lang: en canonical: https://upgradepro.eu/blog/technical-seo-foundations published: 2026-07-16 tags: ["SEO", "technical"] --- # Technical SEO Foundations for AI Search in 2026 > No major AI crawler rendered JavaScript in Vercel's Dec 2024 study; within AI Overview queries a citation earns +120% more clicks per impression than none. Technical SEO is the entry requirement for AI visibility, not a separate discipline. Google states there are no additional requirements to appear in AI Overviews or AI Mode: a page must be indexed and eligible for a snippet, nothing more. Meanwhile, no major AI crawler renders JavaScript, so content that only exists client-side is invisible to ChatGPT, Claude, and Perplexity. Core Web Vitals, clean sitemaps, and bidirectional hreflang remain the base everything else — including GEO — builds on. ## Why is technical SEO the entry ticket to AI search? Because Google's AI features reuse the classic Search infrastructure, a page that cannot be crawled, indexed, and shown with a snippet cannot be cited. Google's own documentation is unusually blunt about this: > "There are no additional requirements to appear in AI Overviews or AI Mode." — [Google Search Central](https://developers.google.com/search/docs/appearance/ai-features), consulted July 2026 No machine-readable AI files, no AI text files, no special markup, no specific schema.org type. To be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Google Search with a snippet, meeting Search's technical requirements ([Google Search Central](https://developers.google.com/search/docs/appearance/ai-features), consulted July 2026). The same controls apply in both worlds: `noindex`, `nosnippet`, `data-nosnippet`, and `max-snippet` also govern the AI features. A forgotten `noindex` or an aggressive `max-snippet` silently removes you from AI answers too. Two consequences follow. First, GEO does not replace SEO — the two overlap at the infrastructure layer, as we unpack in [GEO vs SEO](/blog/geo-vs-seo). Second, structured data keeps its own role in how machines interpret content — see our guide to [structured data for AI](/blog/structured-data-ai) — but Google is explicit it is not an entry ticket to AI features. ## What are the official Core Web Vitals thresholds in 2026? The official thresholds are: LCP within the first 2.5 seconds, INP of 200 milliseconds or less, and CLS of 0.1 or less, according to [web.dev](https://web.dev/articles/vitals) (Google, consulted July 2026). A page passes when it meets each target at the 75th percentile of real page loads, segmented between mobile and desktop. | Metric | What it measures | Good threshold | How it is evaluated | | --- | --- | --- | --- | | LCP (Largest Contentful Paint) | Loading speed of the main content | ≤ 2.5 s | 75th percentile, mobile and desktop separately | | INP (Interaction to Next Paint) | Responsiveness to user input | ≤ 200 ms | 75th percentile, mobile and desktop separately | | CLS (Cumulative Layout Shift) | Visual stability during load | ≤ 0.1 | 75th percentile, mobile and desktop separately | What do they buy you in rankings? Google's position is careful: good Core Web Vitals, with other page experience aspects, align with what its core ranking systems seek to reward — but they do not guarantee better positions alone ([Google Search Central](https://developers.google.com/search/docs/appearance/core-web-vitals), consulted July 2026). Anyone selling a guaranteed ranking jump per millisecond saved is selling smoke. ## Which AI crawlers should you allow or block in robots.txt? Allow the search and indexing bots if you want to appear in AI answers. Each operator documents separate user agents with independent robots.txt controls, so training and search are decisions you can take one at a time. [OpenAI](https://developers.openai.com/api/docs/bots) (consulted July 2026) runs GPTBot for model training, OAI-SearchBot for visibility in ChatGPT search, and ChatGPT-User for visits a user triggers. OpenAI explicitly recommends allowing OAI-SearchBot in robots.txt to appear in ChatGPT search results — and you can block GPTBot while still appearing there. [Anthropic](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (consulted July 2026) mirrors the pattern with ClaudeBot (training), Claude-User (user-requested visits), and Claude-SearchBot (search indexing); blocking Claude-SearchBot can reduce your site's visibility and accuracy in Claude's search answers. [Perplexity](https://docs.perplexity.ai/guides/bots) (consulted July 2026) states that PerplexityBot exists to surface and link sites in its results and is not used to train foundation models — while Perplexity-User, acting on a direct user request, generally ignores robots.txt rules. | Bot | Operator | Purpose | Allow it for AI visibility? | | --- | --- | --- | --- | | GPTBot | OpenAI | Model training | Your call — blocking it does not remove you from ChatGPT search | | OAI-SearchBot | OpenAI | ChatGPT search indexing | Yes — OpenAI explicitly recommends allowing it | | ChatGPT-User | OpenAI | User-triggered visits | Yes, if you want ChatGPT users reaching your pages | | ClaudeBot | Anthropic | Model training | Your call — a separate user agent from Claude-SearchBot, with its own control | | Claude-SearchBot | Anthropic | Search indexing | Yes — blocking it can reduce visibility in Claude answers | | Claude-User | Anthropic | User-requested visits | Yes | | PerplexityBot | Perplexity | Surfacing and linking sites in results | Yes — not used for foundation-model training | | Perplexity-User | Perplexity | Direct user actions | Generally ignores robots.txt; control it at server level if needed | For user-agent strings and verification details bot by bot, see our full [AI crawlers guide](/blog/ai-crawlers-guide). ## How much do AI bots crawl compared with the visits they send back? Far more than they return, and the gap is enormous. According to [Cloudflare](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) (July 2025 data, published August 2025), the crawl-to-refer ratio was 38,065:1 for Anthropic, 1,091:1 for OpenAI, and 194:1 for Perplexity — against 5.4:1 for Google. Training drove nearly 80% of AI bot activity, up from 72% a year earlier, and GPTBot's share of AI crawler traffic rose from 4.7% to 11.7% year over year (July 2024 to July 2025). Treat these ratios as a dated snapshot, not a constant. The direction of travel is hard to ignore. A compilation of Cloudflare Radar data by [Digital Applied](https://www.digitalapplied.com/blog/ai-crawler-bot-traffic-statistics-2026-data-reference) (June 2026) reports that bots already generate 57.5% of HTML web traffic, against 42.5% for humans — a third-party figure, cited with that caveat. If machines are the majority reader, serving them clean, complete HTML stops being optional. ## Do AI crawlers render JavaScript? No — and this is the single biggest technical blind spot for AI visibility. The latest large-scale public evidence is Vercel's study (with MERJ) of real crawler traffic: > "none of the major AI crawlers currently render JavaScript" — [Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler), December 2024 That covers GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and the Meta and ByteDance bots. The exceptions that do execute JavaScript are Google's Gemini, via Googlebot's infrastructure, and AppleBot. The scale makes this matter: in one month on Vercel's network, GPTBot made 569 million requests and ClaudeBot 370 million; AI crawlers combined reached roughly 1.3 billion fetches, about 28% of Googlebot's 4.5 billion. GPTBot downloads JavaScript files in 11.5% of its requests and ClaudeBot in 23.84% — but neither executes them. Content that only appears after client-side rendering simply does not exist for these systems. Googlebot itself does process JavaScript, in three phases — crawling, rendering, indexing — using an evergreen version of Chromium, but pages wait in a render queue that can take from a few seconds to considerably longer ([Google Search Central](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), consulted July 2026). The practical answer for an SMB is the same in both cases: server-side rendering, static generation (Astro, Next.js, Nuxt), or prerendering, so the critical content ships in the initial HTML. One honest caveat: the Vercel study is from December 2024 and crawler capabilities can change; it remains the best public evidence available in mid-2026. ## Do XML sitemaps still matter in the AI era? Yes — arguably more than before, because AI crawlers navigate poorly. In the same [Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) study (December 2024), 34.82% of ChatGPT's fetches ended in a 404, against 8.22% for Googlebot. Bots that waste a third of their requests on dead URLs need every bit of help a clean sitemap provides. The official rules are simpler than most tools suggest ([Google Search Central](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap), consulted July 2026). A single sitemap holds up to 50,000 URLs or 50 MB uncompressed; beyond that, use a sitemap index. Google ignores the `` and `` values entirely, and only uses `` when it is consistent and verifiably accurate against the page's last significant update — bumping it without changing the page defeats its purpose. List canonical URLs only, and keep redirects, parameters, and duplicates out. ## How do you set up canonical and hreflang without breaking a multilingual site? Pick one of the three supported methods and make every annotation reciprocal. Google accepts hreflang via `` tags in the head, HTTP headers, or the XML sitemap — the three are equivalent ([Google Search Central](https://developers.google.com/search/docs/specialty/international/localized-versions), consulted July 2026). The rule that breaks most implementations is bidirectionality: > "If two pages don't both point to each other, the tags will be ignored." — [Google Search Central](https://developers.google.com/search/docs/specialty/international/localized-versions), consulted July 2026 Two more rules from the same documentation. Specifying a region alone is not valid — the language code is required. And x-default marks the fallback version served when no language matches the user. On a site like ours, publishing in English, Slovenian, Spanish, and Italian, every page version must reference all its siblings consistently; one missing return link and Google discards the pair. Canonical tags solve a different problem — consolidating duplicates within one language — while hreflang connects equivalents across languages. Don't use one to patch the other. ## What is a citation in an AI answer actually worth in clicks? Within queries that trigger an AI Overview, being cited more than doubles your clicks compared with being left out. [Seer Interactive](https://www.seerinteractive.com/insights/aio-impact-on-google-ctr-2026-update)'s 2026 update — 53 brands, 5.47 million queries, and 2.43 billion organic impressions between January 2025 and February 2026 — found that being cited in an AI Overview delivers +120% more organic clicks per impression than not being cited. The honest reading cuts both ways. For informational queries, per million impressions, the study reports roughly 33,500 clicks when no AI Overview appeared, roughly 20,743 when the site was cited inside one, and roughly 9,445 when it was not cited. AI Overviews shrink total clicks even for the winners — but once an AI Overview exists, citation is the difference between those last two figures. Everything in this article — indexability, snippet eligibility, server-rendered content, clean sitemaps, correct hreflang — is what makes that citation technically possible. ## What should your SMB fix this week? Run this ten-point check, in order. Most items cost nothing but attention; none require buying a tool. 1. **Indexation and snippet eligibility.** In Google Search Console, confirm your key pages are indexed, then audit for stray `noindex`, `nosnippet`, `data-nosnippet`, and `max-snippet` directives — they also remove you from AI Overviews and AI Mode. 2. **Core Web Vitals with field data.** Check the 75th percentile against LCP ≤ 2.5 s, INP ≤ 200 ms, and CLS ≤ 0.1, mobile and desktop separately. 3. **robots.txt policy per bot.** Allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot; make a conscious decision about GPTBot and ClaudeBot — training and search are separate controls. 4. **The JavaScript test.** Load your key pages with JavaScript disabled. If the main content vanishes, ChatGPT, Claude, and Perplexity cannot read it. 5. **Rendering fix.** Move critical content to server-side rendering, static generation, or prerendering so it ships in the initial HTML. 6. **Sitemap hygiene.** Canonical URLs only, within the 50,000-URL / 50 MB limits, honest `lastmod`, no `priority` or `changefreq`. 7. **Kill your 404s.** AI crawlers hit far more dead URLs than Googlebot (34.82% versus 8.22% in Vercel's data); fix broken internal links and stale sitemap entries. 8. **hreflang reciprocity.** Verify every language pair links both ways and that an x-default fallback exists. 9. **Valid language codes.** Language is required; region alone is invalid and gets ignored. 10. **Baseline before touching anything.** Record AI bot hits in your server logs, indexation counts, and Core Web Vitals, so every change is measured against a before — not a feeling. That baseline-first sequence — audit, implement, measure again — is how our [technical SEO service](/services/seo) works, and the only way to know whether any of this moved the needle. ## Frequently asked questions ### Do I need special markup or an AI file to appear in Google AI Overviews or AI Mode? No. Google states there are no additional requirements: no machine-readable AI files, no new markup, no specific schema. A page must be indexed and eligible to show in Google Search with a snippet, meeting the usual technical requirements. The same fundamental SEO covers AI features. ### Do AI crawlers like GPTBot or ClaudeBot execute JavaScript? No. Vercel's study of real crawler traffic (December 2024) found that none of the major AI crawlers from OpenAI, Anthropic, Perplexity, Meta or ByteDance render JavaScript; they download JS files but never execute them. Only Gemini, via Googlebot infrastructure, and AppleBot render. If key content only exists after client-side rendering, it is invisible to ChatGPT, Claude and Perplexity. ### What are the official Core Web Vitals thresholds? According to Google's web.dev documentation: LCP within 2.5 seconds, INP of 200 milliseconds or less and CLS of 0.1 or less, evaluated at the 75th percentile of real page loads, segmented for mobile and desktop. Google says good Core Web Vitals align with what its ranking systems seek to reward, but they do not guarantee positions on their own. ### Should I allow or block AI bots in robots.txt? It depends on the bot, because each operator documents different purposes and independent robots.txt controls. For visibility in AI answers, allow the search bots: OAI-SearchBot (OpenAI recommends allowing it to appear in ChatGPT search), Claude-SearchBot (Anthropic says blocking it can reduce your visibility and accuracy in Claude's search answers) and PerplexityBot. Training bots are a separate decision: OpenAI documents that you can block GPTBot and still appear in ChatGPT search, while Anthropic lists ClaudeBot (training) and Claude-SearchBot (search indexing) as distinct user agents you control separately. ### Do XML sitemaps still matter in the AI era? Yes, arguably more than before. AI crawlers are inefficient: 34.82% of ChatGPT's fetches ended in a 404 in Vercel's study, versus 8.22% for Googlebot, so a clean sitemap with canonical URLs helps them find your content. Official limits are 50,000 URLs or 50 MB per file. Google ignores priority and changefreq, and only uses lastmod when it is verifiably accurate. ### How do I set up hreflang on a multilingual site without breaking it? Choose one of three equivalent methods: link tags in the head, HTTP headers or the XML sitemap, and make every annotation bidirectional. If page X links to page Y but Y does not link back, Google ignores the tags. Use valid language codes, since region alone is not valid, and add x-default as the fallback when no language matches. ## Sources Every claim in this article links to the source it comes from, with its publication date. - [Google Search Central — AI Features and Your Website](https://developers.google.com/search/docs/appearance/ai-features) — 2026-07-16 - [web.dev (Google) — Web Vitals](https://web.dev/articles/vitals) — 2026-07-16 - [Google Search Central — Understanding Core Web Vitals and Google search results](https://developers.google.com/search/docs/appearance/core-web-vitals) — 2026-07-16 - [Vercel — The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler) — 2024-12-17 - [Google Search Central — JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) — 2026-07-16 - [OpenAI Developers — Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) — 2026-07-16 - [Anthropic Privacy Center — Does Anthropic crawl data from the web](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — 2026-07-16 - [Perplexity Docs — Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) — 2026-07-16 - [Google Search Central — Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) — 2026-07-16 - [Google Search Central — Localized versions of your pages (hreflang)](https://developers.google.com/search/docs/specialty/international/localized-versions) — 2026-07-16 - [Cloudflare Blog — The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) — 2025-08-29 - [Seer Interactive — AIO Impact on Google CTR: 2026 Update](https://www.seerinteractive.com/insights/aio-impact-on-google-ctr-2026-update) — 2026-03 - [Digital Applied — AI Crawler & Bot Traffic Statistics 2026 (citing Cloudflare Radar)](https://www.digitalapplied.com/blog/ai-crawler-bot-traffic-statistics-2026-data-reference) — 2026-06