Why AI search ignores your site
A page can rank first in Google and still be invisible to an answer engine. These are the seven causes we see most often, in the order it is worth checking them — the early ones are cheap to rule out and fatal if true.
Updated 30 October 2026
The seven causes
1. The retrieval crawler is blocked
The usual suspect is a robots.txt rule written for training crawlers and applied to everything, or a CDN toggle labelled “block AI bots” that also stops the crawlers which fetch pages for answers. A WAF rule can do it without anyone noticing, because a challenge page is an ordinary 403 to a crawler.
How to confirmRequest a page with the crawler's own User-Agent and read the status code. 403, 429 and a JavaScript challenge all look identical to the model: nothing to read.
2. The text only exists after JavaScript runs
Most AI crawlers do not execute JavaScript. A page built by React, Vue or a client-side fetch returns a shell with an empty container, so the crawler sees the navigation and nothing else. Google can render, which is why this failure hides: the page ranks and is still skipped.
How to confirmFetch the URL with a plain HTTP client and search the raw response for a distinctive sentence from the page. If it is not there, it is not there for the crawler either.
3. The page is not indexed
Retrieval has an upstream: a normal search index, or the product's own crawl. A page blocked by noindex, left out of the sitemap, linked from nowhere, or simply never crawled is not a candidate no matter how good the writing is.
How to confirmSearch the exact URL in Google and Bing. If neither returns the page, fix indexing before rewriting anything.
4. A canonical, a redirect or a duplicate points somewhere else
Two versions of the same page — http and https, www and bare domain, with and without a tracking parameter — split whatever signal the page has. A canonical that points at a different URL tells every engine which one to trust, and it is not the one you are measuring.
How to confirmLook at the canonical tag on the page and at the redirect chain for its URL. Both should end at one HTTPS address, once.
5. The passages do not survive being cut out
Retrieval scores passages, not pages. A section that opens with “It works the same way here” or “As we saw above” is a fragment once the page around it is gone, and the model drops it.
How to confirmCover the rest of the page and read each paragraph on its own. Anything that no longer makes sense is a passage you are not getting credit for.
6. There is nothing to quote
Answer engines need a fact to carry: a figure, a date, a version, a named comparison. Marketing language without specifics gives the model nothing to use, even when the page is the best answer.
How to confirmHighlight every number, date and proper noun on the page. If a section has none, the model has no reason to cite it rather than paraphrase it without a link.
7. The page answers a different question
The question being asked in the chat is often narrower than the page's topic. A page about “SEO tools” will not be pulled for “what does AI Overviews do with noindex”, even if it mentions the answer in passing. Retrieval matches the question to a passage, so the passage has to look like an answer to that question.
How to confirmWrite down five questions your buyers ask, then look for a heading that asks the same one. If the heading is a product category, the answer is somewhere else.
Why the order matters
Causes 1 to 4 are about access; 5 and 6 are about writing; 7 is about which pages exist at all. Rewriting a page whose text never reaches the crawler is wasted work, and that is the most common way to spend a quarter on AI visibility without moving anything.
Work top to bottom: confirm the crawler gets a 200 with your text in it, confirm the page is indexed under one canonical URL, then look at whether the passages are quotable and whether the page competes for the question being asked.
What the report checks
A Lumicite audit maps to the same list: robots.txt for all 19 crawlers, what the page returns, the text present in the HTML, titles, canonical and structured data, and a grade for every passage. The crawler reference explains which tokens decide what, and the writing guide covers causes 5 and 6 in detail.
Find out which cause is yours
Free, three pages, no sign-up. The report tells you what to fix first, not everything at once.