There’s a gap between how confidently technical AI-search advice gets sold and how little of it anyone has actually demonstrated. Add this schema type. Ship an llms.txt. Tighten your entity signals. The pitch is specific, the price is real, and the evidence behind most of it is thin enough to see through.
That doesn’t mean the technical work is pointless. It means the honest version of the checklist is more useful than the confident one, because it tells you where to spend first. This piece separates what Google has actually said, what third parties have measured and hedged, and what is still just reasoning. Where we’re reasoning rather than citing, we say so.
Technical SEO for AI search is the machine-readable layer that decides whether an answer engine can reach, parse, and reuse your page: crawl access, indexation, server-rendered HTML, structured data, and entity clarity. It governs eligibility, not selection. Getting it right makes you available to be cited. It does not make you cited.
That distinction runs through everything below. If you want the content side of the problem, how to structure pages so they’re worth quoting once you’re eligible, we covered it in structuring content for AI citation.
What AI Answer Surfaces Actually Run On
The useful starting point is that answer engines aren’t parallel infrastructure. They sit on top of search infrastructure you already understand.
Google says so directly. Its May 2025 Search Central post on performing well in AI experiences states that meeting the standard technical requirements for Search, being crawlable, indexable, and returning a 200, “covers you for search generally, including AI formats.” Google frames all nine of its recommendations there as long-standing Search advice that carries across, not as a new AI-specific discipline. By Google’s own account, the generative features themselves are built on a Gemini model customized for Google Search, combined with its existing Search systems rather than standing beside them. That’s Google describing its own architecture, not an independent audit of it. AI Mode, generally available across the US and expanded to more than 200 countries and territories, draws on that same stack. We went through the mechanics in our AI Overviews explainer.
The pattern repeats outside Google. When ChatGPT search launched, CNBC reported that it drew on partnerships with news and data providers plus third-party search providers. OpenAI has never published the full list, and when asked directly whether ChatGPT search ran on Bing, an OpenAI engineering VP said only that they use a set of services and Bing is an important one. Seer Interactive later found that more than 87% of ChatGPT search citations matched a page in Bing’s top organic results for the same query. That’s a directional finding from roughly 100 queries and 500-plus citations, not a controlled study, and Seer presents it as such. Read it as a strong hint rather than a law: if Bing hasn’t indexed you, ChatGPT search has a hard time reaching you, and most startups have never opened Bing Webmaster Tools.
None of that is exotic. It’s the same SEO fundamentals work that has always decided whether a page is reachable, with a new consequence attached.
Crawlability, Indexation, and Rendering
This is the part that’s genuinely load-bearing, and it’s the least glamorous section on the page.
AI Crawlers Don’t Run Your JavaScript
If you take one technical fact from this article, take this one. A joint Vercel and MERJ analysis of AI crawler traffic found that the major AI crawlers fetch JavaScript files but never execute them. Across roughly 569 million GPTBot requests in the month they measured, they found no evidence of JavaScript execution at all: GPTBot fetched JS files in 11.50% of requests and Claude’s crawler in 23.84%, in both cases without running them. The scope matters and the authors state it: the data comes from Vercel’s own network including nextjs.org, plus two job-board sites on different stacks for cross-validation, from US data centers only, over a window they describe as the past few months without giving a start date. Microsoft Copilot was excluded outright because it lacks a unique user agent to track.
So this isn’t a web-wide census. But the direction is unambiguous and the fix is old news: if your content only exists after client-side hydration, an AI crawler sees an empty shell. Server-render or statically generate anything you want cited. A React app that renders its article body in the browser is invisible to the crawlers, no matter how good the article is.
Controlling How Your Content Appears
Google’s May 2025 post also lays out the levers it says it honors for AI experiences: nosnippet, data-nosnippet, max-snippet, and noindex, alongside general page-experience guidance about cross-device display and latency. Treat those as Google’s stated mechanism rather than a guaranteed outcome, and note the tradeoff before you reach for them. Suppressing snippets to keep your content out of an AI answer also suppresses the citation that comes with it.
One thing we won’t claim: there’s no credible dated evidence tying Core Web Vitals to AI citation odds. Speed and stability are worth fixing because they affect crawl budget and the humans who arrive, not because anyone has shown they move citations. Anybody selling you a performance sprint as an AI-visibility play is ahead of the evidence.
Does Schema Markup Help You Get Cited?
The honest answer: the evidence is thin, mixed, and platform-dependent, and Google has never claimed it helps. That’s the whole finding, and it’s worth sitting with before you buy a structured-data package.
Start with what Google actually says. The May 2025 Search Central post lists structured data among its recommendations, but the claim it makes is narrow: structured data “is useful for sharing information about your content in a machine-readable way that our systems consider and makes pages eligible for certain search features and rich results.” Follow the guidelines, keep marked-up content visible on the page, validate it. Nowhere on that page does Google tie structured data to AI Overviews or AI Mode citation, ranking, or visibility. The connection people sell you is not asserted by the source they’re implicitly leaning on.
Third-party evidence points the same direction. AccuraCast analyzed over 2,000 prompts across ChatGPT, Google AI Overviews, and Perplexity, examining roughly 9,000 citation sources, and found 81% of cited pages carried some schema markup. Their own read of that number is the important part: “one could jump to the conclusion that including schema is a requisite for getting cited in AI answers. But that wouldn’t be an accurate assumption,” and they conclude schema “matters to a very limited extent as a ranking, visibility, or citation factor.” It’s also a single vendor’s own methodology, using Screaming Frog for schema detection plus Otterly.ai and manual query testing by their own analysts, across three platforms, with no significance testing reported. Eighty-one percent of cited pages having schema mostly tells you that most decent pages have schema.
Search Engine Land ran a controlled three-page experiment in September 2025: one page with well-implemented schema, one with poor schema, one with none, all built on a single matched target keyword. The well-implemented page peaked at position 3 and was the only one whose query triggered an AI Overview appearance. The no-schema page was crawled but never indexed. Before anyone builds a strategy on that, note the sample size is three pages and the authors say plainly, “We don’t consider these results to be absolute proof that well-implemented schema plays a role in AI Overview presence,” adding that “there’s always the possibility that our results were simply a coin-flip-style fluke of the Google algorithm.” Their earlier 100-site healthcare correlation study found a relationship that was not statistically significant, didn’t assess schema quality, and couldn’t control for site differences.
What the Schema Type Breakdown Shows
AccuraCast’s type distribution is the most practically useful thing in their study, because it cuts against what’s usually recommended. Person schema was by far the most common on cited pages at 58.9% overall, rising to 70.4% on ChatGPT specifically and 56.0% on Google AI Overviews. Product came in at 6.9%, Question at 3.1%, FAQPage at 1.8%, with HowTo and Review each under 1%. Their causation warning applies to all of it: this describes what cited pages happen to have, not what got them cited.
The platform divergence matters more than any single number. ChatGPT gave more weight to ListItem schema than the other platforms did, while on Perplexity, Person, Organization, and Article schema had no bearing on citation likelihood at all, and pages without those types were actually more visible. That’s the opposite pattern from ChatGPT. There is no single universal schema recommendation to give you, and anyone who gives you one is filling a gap in the evidence with confidence.
Google Already Walked Back the Types Being Pitched
Worth remembering when someone sells you an FAQ schema rollout: Google cut FAQ rich results back to well-known, authoritative government and health sites in August 2023, and restricted How-To rich results to desktop. A September 2023 update to the same post finished the job, removing How-To from desktop too. As of now, FAQ rich results remain limited to authoritative government and health sites and How-To rich results are gone entirely. Google’s own framing was that this wasn’t a ranking change: the visible effects were limited to search-appearance metrics and impression counts in Search Console, and there’s no need to proactively remove the markup, since unused structured data doesn’t cause problems for Search.
Our position, stated as interpretation rather than evidence: implement structured data. It’s cheap, standards-based, and it makes your entities and content legible to machines, which is a reasonable thing to want regardless of what any specific engine does with it. Just don’t implement it expecting an AI citation lift, and don’t pay a premium for it as an AI-search service, because nobody has demonstrated that lift exists.
Entity Signals: Talked About Constantly, Measured Almost Never
Entities are real infrastructure. Google’s Knowledge Graph launched in May 2012 with, by Google’s own launch-day description, more than 500 million objects and more than 3.5 billion facts about and relationships between those objects. Those are Google’s self-reported figures from 2012, not an independent count or a current one.
There’s also decent evidence about which sources answer engines lean on. Ahrefs analyzed the top 1,000 pages ChatGPT cites and found Wikipedia accounted for 29.7% of pages by content type, the largest single category, ahead of homepage and landing pages at 23.8% and educational pages at 19.4%. That’s a September 2025 snapshot from Ahrefs’ own Brand Radar tool, covering one platform and the top 1,000 pages only, with content-type classification that isn’t foolproof. Useful for direction, not for planning.
E-E-A-T and Author Signals
The E-E-A-T literature is well dated and worth knowing. Google added Experience to E-A-T in December 2022, recognizing content that demonstrates first-hand use. It updated the quality rater guidelines again in November 2023, noting that none of those changes involved major or foundational shifts. The August 2022 helpful content update post set out the people-first framework, including the line “SEO is a helpful activity when it’s applied to people-first content. However, content created primarily for search engine traffic is strongly correlated with content that searchers find unsatisfying.” Google uses “strongly correlated” rhetorically there, not as a reported statistic. And the standing caveat on all of it: rater guidelines describe what Google wants its systems to reward, and Google is explicit that they don’t directly influence ranking.
The Gap, Stated Plainly
Here’s what we could not find, after checking specifically for it: no dated Google Search Central blog post published before December 2025 ties E-E-A-T, author signals, or entity signals to AI Overviews or AI Mode. The May 2025 post about succeeding in AI experiences doesn’t mention E-E-A-T, expertise, authoritativeness, trustworthiness, or author attribution anywhere in its body. We also found no dated study linking NAP consistency or brand-name consistency to AI citation behavior.
That’s an absence of evidence in our search, not proof of absence. But it should change how you buy. Consistent entity markup, accurate Organization and Person data, and the same brand name and details everywhere are worth doing because inconsistency creates real ambiguity for any system trying to resolve who you are. That’s our reasoning about a mechanism, not a measured result, and we’d rather label it than dress it up. The guidance for sameAs, Organization, and Person markup lives on Google’s continuously revised reference documentation, which is why we’re describing the practice rather than citing a dated page for it.
llms.txt: Cheap, Harmless, and Unproven
llms.txt is a proposal from Jeremy Howard at Answer.AI, published in September 2024: a curated markdown file at your site root with an H1 site name, an optional summary, and H2-delimited link lists pointing to further markdown resources. Read the original and note what it doesn’t claim. It’s framed as an inference-time context aid for language models, making no claims about ranking, citation frequency, or SEO benefit. The visibility promises attached to it since were added by other people.
The evidence has been consistently unencouraging. Google’s John Mueller, as reported by Search Engine Journal in April 2025, compared it to the keywords meta tag: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag.” His “AFAIK” and “To me” are doing real work there, and it’s a personal read rather than an official Google position. Search Engine Land’s March 2025 introduction was already hedging hard, noting adoption “is definitely not even near critical mass” and framing the benefits as theoretical, with no guarantees and no harm in trying.
The most direct measurement came in November 2025. SE Ranking analyzed roughly 300,000 domains and found only 10.13% had implemented an llms.txt file, then tested whether its presence predicted AI citation frequency using correlation testing and an XGBoost model. Removing the llms.txt variable from the model actually improved its accuracy, meaning the file added noise rather than signal. GPTBot does occasionally fetch these files, but infrequently, with no apparent connection to citation outcomes. Their own hedge is worth carrying: the file “doesn’t seem to directly impact AI citation frequency. At least not yet.”
Duane Forrester wrote about it in November 2025 too, saying “I’m personally not a fan” and putting the standard in a “grey zone between promise and problem.” That’s explicitly his personal opinion rather than industry consensus. His structural point is the interesting one: Schema.org worked because Bing, Google, Yahoo, and Yandex backed it together with shared vocabularies, and llms.txt has no equivalent governance or verification, which leaves room for cloaking, showing bots a different file than humans see.
Our position: llms.txt is cheap and harmless. If it takes an hour, ship it. Don’t pay anyone for it as a visibility lever, don’t let it displace crawl, index, and rendering work, and be skeptical of any proposal that leads with it. It’s a bet on a standard that might get adopted, not a lever that currently does anything measurable.
Crawler Policy: Who’s Fetching Your Content
Two separate decisions get conflated constantly. Whether an AI company may use your content to train a model, and whether its retrieval systems may fetch your pages to answer a live question, are controlled by different tokens in robots.txt. Blocking training access doesn’t remove you from AI answers, and blocking retrieval access does, which is the opposite of what most people want.
One correction, because it circulates as folklore: Google-Extended governs training use for Gemini app responses and Vertex AI products. It does not govern AI Overviews. AI Overviews are generated from the standard Search index, so ordinary Googlebot crawling and indexing is what controls your presence there. Blocking Google-Extended does not remove you from AI Overviews, and allowing it does not add you.
Most sites haven’t made either decision deliberately. Cloudflare found that only 37% of the top 10,000 domains maintain a robots.txt file at all, and GPTBot appears in disallow directives in just 7.8% of the ones that exist. Fewer than 100 new robots.txt files were added across those top domains between January and June 2025. A companion analysis of the 3,816 domains where a robots.txt could be found showed 546, about 14%, carried directives specifically targeting AI crawlers, with GPTBot the most blocked at 312 domains and the most explicitly allowed at only 61. Cloudflare notes a methodological wrinkle: some robots.txt tokens don’t correspond to any actual HTTP user agent, so entries don’t always map to request logs.
The Crawl-to-Refer Gap
The economics are worth understanding before you decide how open to be. Cloudflare’s crawl-to-refer analysis measured how many pages a crawler fetches for every referral visit it sends back. For the week of June 19 to 26, 2025, Anthropic’s Claude crawler made roughly 70,900 HTML page requests per referral, while Mistral sat at the other extreme around 0.1, referring roughly ten times more than it crawled. Cloudflare’s own caveat belongs with those numbers: traffic referred by native apps doesn’t include a Referer header, so the counts cover web-based tools only and “may overstate the respective ratios, but it is unclear by how much.”
A later Cloudflare dataset covering monthly figures from January to July 2025 found the gap narrowing at the top: Anthropic’s ratio fell 86.7% over that period to roughly 38,066 crawls per referred visit in July, down from about 286,930 in January. OpenAI’s sat around 1,091 in July, down about 10%. Perplexity’s rose 256.7%, from about 55 to about 195. Google’s was lowest throughout at 5.4. These are a different measurement over a different window than the June week above, so read them as two separate snapshots rather than one trend line. Cloudflare classifies crawler purpose from operator disclosures and industry sources rather than independent verification, and notes that Anthropic’s crawler lags on verification, making its traffic easier to spoof.
When Crawler Policy Turns Adversarial
Cloudflare accused Perplexity in August 2025 of using stealth, undeclared crawlers to get around no-crawl directives: when the declared crawler was blocked, Cloudflare observed follow-up requests using a generic browser user agent, rotating IPs and switching between ASNs, ignoring or failing to fetch robots.txt entirely. Cloudflare removed Perplexity from its verified bots list and added detection rules. Their own framing limits how far you should take it: “Once this post is live the behavior we saw will almost certainly change.” It’s a snapshot of adversarial behavior at one moment, not a permanent characterization. The practical lesson is that robots.txt is a request, and if you need enforcement you need enforcement tooling.
Content Signals
Cloudflare added a Content Signals Policy to robots.txt in September 2025, letting site owners state machine-readable preferences about how content is used after it’s been fetched: search, ai-input, and ai-train, each set to yes, no, or left blank. They auto-updated robots.txt for more than 3.8 million domains already using their managed file, defaulting to Content-Signal: search=yes, ai-train=no. Cloudflare is upfront that these “express preferences; they are not technical countermeasures against scraping,” and that “some companies might simply ignore them.” If you’re on Cloudflare, check what your robots.txt says right now, because it may have been updated on your behalf.
Measuring Any of This
Measurement is still the weakest link. Microsoft’s Bing Webmaster team argued in November 2025 that brands should track impressions, placement within AI answers, and citations rather than clicks alone, since a visible citation that doesn’t produce a click still does work. Their view is that structured content helps AI systems interpret and summarize a page, which is a vendor’s stated position rather than a measured result.
Practically: run a monthly prompt audit against the questions that actually matter to your pipeline, log which sources each engine cites, and watch the trend. Our AI search visibility audit guide walks through the pre-work, and the AEO fundamentals piece covers the citation-earning side.
The Honest Checklist
Three tiers, labeled by how much evidence sits behind each. Spend top-down.
| Tier | What it covers | Evidence behind it |
|---|---|---|
| Do it first | Crawlable, indexable, returns 200. Server-rendered or static HTML for anything you want cited. Snippet and display controls set deliberately. Valid structured data with marked-up content visible on the page. | Google’s own stated guidance, plus the Vercel and MERJ finding on JavaScript, with its stated scope limits. |
| Reasonable, unproven | Consistent Organization and Person data. Clean internal linking. Accurate entity details everywhere your brand appears. Presence in the sources answer engines demonstrably lean on. | Our reasoning about how retrieval and entity resolution work. No dated study measures these against AI citation. |
| Cheap, no demonstrated upside | llms.txt. Content Signals declarations. Additional schema types added specifically for AI. | Measured and found to do nothing detectable, or never measured at all. Do them if they’re free. Don’t buy them. |
The uncomfortable summary: the tier with the most evidence behind it is the tier that was already standard technical SEO practice, and the tier being marketed hardest is the tier with the least. That’s not an argument for doing nothing. It’s an argument for doing the boring things properly before buying the interesting ones.
Questions We Keep Getting
Does schema markup improve your odds of being cited in AI Overviews? Nobody has shown that it does. AccuraCast found 81% of AI-cited pages carried some schema across roughly 9,000 citation sources, but warned directly that “one could jump to the conclusion that including schema is a requisite for getting cited in AI answers. But that wouldn’t be an accurate assumption,” concluding schema “matters to a very limited extent.” Google’s own May 2025 guidance ties structured data only to eligibility for certain search features and rich results, and makes no AI-specific claim. Implement it for machine comprehension, not for a citation lift.
Do I need an llms.txt file? No. SE Ranking’s analysis of roughly 300,000 domains found only 10.13% have one, and that removing llms.txt from their citation-prediction model improved its accuracy, meaning it added noise rather than signal. Their hedge: it “doesn’t seem to directly impact AI citation frequency. At least not yet.” It costs an hour and does no harm, so ship one if you want. Don’t pay for it as a visibility service.
Do AI crawlers render JavaScript? The major ones don’t. Vercel and MERJ found no evidence of JavaScript execution across roughly 569 million GPTBot requests in the month they measured, with GPTBot fetching JS files in 11.50% of requests and Claude’s crawler in 23.84% without running them. That study covers Vercel’s own network plus two validation sites, from US data centers, and excludes Microsoft Copilot for lacking a trackable user agent. Server-render anything you want cited.
Should I block AI crawlers? Decide training and retrieval separately, because different tokens control them. Blocking retrieval crawlers removes you from AI answers, which is the opposite of AEO. Blocking training crawlers is a separate call about how your content gets used. And note that Google-Extended governs training for Gemini app and Vertex AI, not AI Overviews, which run on ordinary Googlebot indexing.
Is technical SEO for AI search different from regular technical SEO? Mostly no, and that’s the finding. Google states that meeting Search’s technical requirements “covers you for search generally, including AI formats”. The genuinely new parts are narrow: rendering matters more because AI crawlers don’t execute JavaScript, and crawler policy is now a deliberate decision with two independent halves rather than one setting.
Where to Spend
The technical layer buys eligibility. It’s necessary, it’s cheap relative to content work, and most of it is the same work that made pages findable before answer engines existed. What it won’t do is select you. Once you’re reachable, parseable, and indexed, whether you get cited comes down to whether your page is the best available answer, which is a content problem rather than a markup problem.
So do the boring tier properly, label the unproven tier as unproven, and put the money you didn’t spend on llms.txt into something that has evidence behind it. If you want help sorting out which tier your site is actually stuck in, that’s the work our AI Search and Answer Engine Optimization service starts with.
