AI Search Access Audit: How to Check robots.txt, Sitemaps, and llms.txt Without Overclaiming

The short answer

An AI Search access audit checks whether important pages are discoverable, allowed under the relevant crawl policy, delivered successfully, and represented by the content you intend systems to read. It should inspect robots.txt, XML sitemaps, canonical and indexing signals, response behavior, and—where a team chooses to publish one—llms.txt.

It does not prove that a page was retrieved for an answer, cited in a response, recommended to a user, or responsible for a conversion. Keep the evidence chain explicit:

crawl policy → page delivery → answer capture → visible citation → referral → conversion

An access audit identifies technical conditions. It does not turn a successful request into proof of a citation or business outcome.

This guide targets searches such as “how do I audit robots.txt for AI crawlers?”, “does llms.txt improve AI visibility?”, “can blocking a bot prevent citations?”, and “how do I check whether AI Search can access my pages?” The practical answer is a layered audit, not a visibility guarantee.

Why access checks matter—and why they are easy to overstate

A page cannot be evaluated under a particular retrieval path if that path cannot reach the intended response. A blocked request, redirect loop, authorization wall, server error, or empty rendered document can be a useful technical finding.

But availability is only one condition. A system may use another source, a cache, a different fetcher, or a separate retrieval process. A page that returns 200 OK may still be irrelevant to a particular question, less useful than another source, or absent from a sampled answer. Google says its AI features may show links to supporting web resources and recommends foundational Search practices (AI features and your website, checked September 29, 2026). That guidance does not publish a rule that makes every accessible page appear in an AI answer.

Use access data to diagnose access. Use answer captures to measure answer-level visibility.

1. Define the page set and decision

Do not begin by scanning every URL and calling the result an AI Search score. Define what the audit must help you decide:

Decision	Page set	Evidence needed
Are important pages technically reachable?	Canonical product, service, documentation, and editorial URLs	Policy, response, redirect, and delivered-content checks
Are high-risk facts available to verify?	Pricing, limits, integrations, safety, and availability pages	Current HTML, structured facts, dates, and source ownership
Did a deployment change access?	Pages affected by the release plus a stable control set	Before/after response and policy records
Are answers citing the right sources?	Pages mapped to a fixed prompt panel	Answer text, source URLs, claims, and collection conditions
Should we publish `llms.txt`?	A deliberately scoped documentation or content set	A documented editorial goal and a post-publication test, not a promise

Record the hostname, URL patterns, market, language, authentication assumptions, and collection date. A result for www.example.com does not automatically describe an app subdomain, a CDN hostname, or a localized site.

2. Audit `robots.txt` as a policy document

The Robots Exclusion Protocol specification, checked September 29, 2026, describes a protocol for crawlers that choose to follow rules in robots.txt. It does not provide a universal identity system for bots, guarantee that every client follows the file, or prove that a compliant request was used in an answer.

Google’s robots.txt documentation, checked September 29, 2026, explains how Google Search interprets the file. Apply vendor-specific guidance to the crawler or fetcher you are evaluating rather than assuming that one user-agent group represents every AI product.

Check:

The file is available at the site root, such as https://example.com/robots.txt.
The response status and content type are expected.
User-agent groups are parsed according to the documented matching rules.
Important page paths are not disallowed for the crawler relevant to the decision.
Rules did not change unexpectedly after a deployment.
Sitemap declarations point to the intended absolute URLs.
Rules for assets or APIs are not being mistaken for rules for canonical HTML pages.

A Disallow line is evidence of a declared policy, not proof that a specific product could never access the content. Conversely, the absence of a matching rule is not evidence that a page was retrieved or cited.

What to save

Keep the raw file, retrieval timestamp, response headers relevant to the audit, parser version, and a normalized list of rules. If a policy is generated by infrastructure, record the deployment or configuration version. A screenshot alone is difficult to compare and audit.

3. Validate XML sitemaps for discovery hygiene

A sitemap is a discovery and URL-set signal, not a citation feed. Google’s sitemap overview, checked September 29, 2026, describes sitemaps as a way to provide information about pages, videos, and other files and their relationships. It does not say that inclusion guarantees crawling, indexing, ranking, or AI citation.

For the pages that matter, check:

The sitemap index and child sitemaps return successfully.
URLs use the intended canonical host and protocol.
Entries are not dominated by redirects, error pages, parameter variants, or thin duplicates.
lastmod is maintained only when it reflects a meaningful modification under your publishing process.
Important pages are not omitted because the generator uses the wrong collection or environment.
The sitemap is referenced in robots.txt or submitted through the relevant search-engine workflow where appropriate.

Do not inflate lastmod on every build to suggest freshness. A changed timestamp can be useful metadata only when it describes a real change; it is not evidence that an AI system will prefer the page.

4. Treat `llms.txt` as an optional proposal, not a switch

The llms.txt proposal, checked September 29, 2026, describes a convention for placing a curated Markdown summary at /llms.txt, with links to fuller material. It is not the same thing as the standardized Robots Exclusion Protocol, and the proposal itself should not be represented as proof of adoption by every AI Search system.

If you publish one, make its purpose explicit:

Help a human or compatible system find the canonical documentation set.
Provide concise context about the site and its sections.
Link to stable, high-value pages rather than every URL.
Keep product names, limits, dates, and terminology aligned with the source pages.
Version or review it when the linked documentation changes.

Avoid claims such as “adding llms.txt makes ChatGPT cite your site” unless you have a tightly controlled, reproducible experiment that supports that exact claim. A file can be available and well-written while having no observable effect in a particular answer sample.

A useful `llms.txt` review

Ask:

Is the file at the intended origin and returning the intended content?
Does it identify the site and link to canonical documentation?
Are links absolute, live, and not blocked by an unrelated access layer?
Are deprecated pages, temporary URLs, or unsupported claims excluded?
Can a reviewer trace each summary statement to a source page?
What answer-level test will be run after publication, and what result would remain inconclusive?

The last question protects against a common mistake: publishing a new file and interpreting any later answer change as proof that the file caused it.

5. Test the actual page response

A policy file is not a page test. Fetch the URLs in the defined set and record:

HTTP status and redirect chain
Final URL and canonical link
noindex or other relevant robots meta signals
Content type and response size
Whether important text is present in delivered HTML
Authentication, WAF, rate-limit, or geo restrictions
Cache and edge behavior when it affects the observation
Fetch timestamp and user-agent used for the test

A successful HTML response is a technical observation. It does not show what a hidden retrieval system extracted, whether a page was selected, or whether a user saw it. If rendering is required to expose the meaningful content, test the delivered experience deliberately and document the method; do not assume that a browser screenshot represents every crawler.

For a server-log perspective, see AI Crawler Analytics: What Server Logs Can—and Cannot—Tell You. A matching user-agent in a log is still an access observation, not a citation record.

6. Run a separate answer and citation test

If the decision is whether a brand or page appears in AI Search answers, build a fixed prompt panel. Include category, comparison, branded, product-fact, and risk questions as relevant. Record:

prompt ID, exact wording, surface and mode, market, language,
model when disclosed, date, answer, visible source URLs,
claim supported, and unavailable state

Tools such as PromptWatch, AI Search Console, Peec AI, Otterly.AI, and Scrunch AI may support different combinations of prompt monitoring, answer capture, crawler data, or source analysis. Their metrics should not be treated as interchangeable without checking the prompt set, surface, refresh method, denominator, and evidence retained.

A useful result is bounded:

The target URL appeared as a visible source in 6 of 30 captured US-English answers in the fixed panel collected on September 29, 2026. This describes the sample and conditions; it does not estimate all AI Search visibility or prove that the access policy caused the citations.

Keep “brand mentioned,” “brand recommended,” “owned URL cited,” and “source supports the claim” as separate fields. A page can be cited for one sentence while the answer contains an inaccurate price or feature description.

Access audit versus answer audit

Observation	What it supports	What it does not prove
`robots.txt` allows a path	A declared policy permits a compliant crawler path	That a product fetched or cited the page
Sitemap includes a URL	The URL is listed in the submitted discovery set	Crawling, indexing, ranking, or citation
`llms.txt` returns `200`	The file is available at the tested origin	Adoption, retrieval, citation, or traffic
Page returns `200` with useful HTML	The tested request received the intended response	Selection in an answer or user recommendation
Named crawler requests a page	A request matching the classification was observed	Purpose, answer use, or downstream conversion
Answer displays the URL	A citation was visible under the recorded conditions	Click, trust, or revenue

This separation is the core of the audit. A technical fix can be worthwhile even when citation impact is unknown, because it improves reliability or removes a documented access problem. Report that technical outcome directly instead of attaching an unsupported business result.

A practical seven-step workflow

Choose the decision and URL set. Prioritize pages whose access or factual accuracy matters.
Snapshot policy files. Save robots.txt, sitemap indexes, child sitemaps, and optional llms.txt with timestamps.
Parse and flag. Identify blocked paths, invalid URLs, redirects, stale entries, and policy changes.
Fetch representative pages. Test canonical URLs, high-risk facts, and a control group with a documented method.
Inspect logs separately. Classify matching requests with a versioned rule; retain unknowns and failures.
Run the answer panel. Capture exact prompts, answers, visible source URLs, and claim relationships.
Make one bounded change and retest. State what changed, preserve the denominator, and label associations rather than causal conclusions.

If a deployment changes robots.txt, do not simultaneously rewrite the pricing page, launch a campaign, change the prompt panel, and call the next result an experiment. Too many moving parts make the evidence weaker.

What the audit can and cannot prove

An audit can identify:

A declared crawl-policy conflict
Broken or misleading sitemap entries
A page that returns an error, redirect loop, or unexpected content
A missing or stale documentation pointer
A visible citation gap in a defined prompt sample
A mismatch between a claim and the source page that was cited

It cannot, from access evidence alone, prove:

That a model read or retained a page
That a page will be cited in future answers
That llms.txt caused a visibility change
That removing a block creates traffic or revenue
That a crawler request came from a user-facing answer
That a citation caused a click or conversion

The independent paper GEO: Generative Engine Optimization (Aggarwal et al., arXiv, checked September 29, 2026) provides research context for generative-engine visibility experiments. It does not validate a particular access file, commercial dashboard, current model behavior, or causal business outcome.

Trial and implementation checklist

Before buying a product or assigning a technical project, verify:

Can the workflow export raw policy, response, log, or answer evidence?
Does it distinguish crawler requests from user-facing citations and referrals?
Are prompt, surface, model, market, language, date, and unavailable states visible?
Does it preserve exact source URLs and the claims they support?
Can you compare a fixed control panel before and after a change?
Are tool-specific credit and refresh units documented?
Does the workflow retain failed, blocked, redirected, and unknown observations?
Is llms.txt presented as an optional convention rather than a guaranteed ranking or citation mechanism?
Can a reviewer reproduce the result without relying on a blended score?

FAQ

Does allowing an AI crawler in `robots.txt` guarantee citations?

No. It records a policy that a compliant crawler may follow. Citation requires separate answer-level evidence showing the page or URL in a captured response.

Does blocking a crawler remove a brand from AI Search?

Not as a universal conclusion. The result depends on the crawler, product, retrieval path, cache, other sources, and policy compliance. A block can establish an access condition; it does not by itself establish a visibility or traffic outcome.

Is `llms.txt` required for GEO?

There is no basis in the sources reviewed for calling it universally required. It is an optional proposal that some site owners may use to curate documentation links. Test its operational usefulness and do not promise a citation effect.

Should every URL be placed in `llms.txt`?

Not necessarily. The proposal is oriented around concise, curated context and links. A short, maintained set of canonical, useful pages is easier to review than an unbounded URL dump. The right scope depends on the site’s documentation and governance needs.

Can Google Search Console prove ChatGPT access or citations?

Not by itself. Search Console data describes the Search properties and reports available through Google’s systems. It is not a universal log of every AI product’s crawler request or answer citation. Use each platform’s own documented data and preserve your answer sample separately.

What should I do first on a small site?

Start with a small critical URL set: home, category or service page, pricing or availability page, key documentation, and a control page. Snapshot policy files, test responses, and run a fixed prompt panel before expanding the audit.

Evidence snapshot

Source	Public signal	What it supports	Confidence
IETF RFC 9309	Standards-track Robots Exclusion Protocol specification	What `robots.txt` standardizes and what it does not prove	High for the protocol; not vendor behavior
Google: Introduction to robots.txt	Official Google Search documentation	Google-specific robots.txt interpretation context	High for Google documentation; not every AI system
Google: Sitemaps overview	Official sitemap guidance	Sitemap purpose and discovery boundaries	High for Google guidance; not citation guarantees
llms.txt proposal	Public convention proposal	What the proposal suggests and its non-universal status	Medium; proposal, not a universal standard
Google Search Central: AI features	Official guidance that AI features may show supporting links	Boundary between AI-feature links and technical access evidence	High for Google guidance; not a universal inclusion formula
Aggarwal et al., GEO	Independent academic research	Context for generative-engine visibility experiments	Medium; not validation of a production access tactic
AICiteKit editorial framework	Layered policy, delivery, answer, citation, and outcome method	How to structure an auditable workflow	Editorial

Sources and verification

IETF RFC 9309: Robots Exclusion Protocol — checked September 29, 2026; protocol boundaries for robots.txt.
Google Search Central: Introduction to robots.txt — checked September 29, 2026; Google-specific interpretation and implementation context.
Google Search Central: Sitemaps overview — checked September 29, 2026; sitemap purpose and limits.
The llms.txt proposal — checked September 29, 2026; proposal scope and format.
Google Search Central: AI features and your website — checked September 29, 2026; AI-feature links and foundational Search guidance.
Aggarwal et al., “GEO: Generative Engine Optimization” — checked September 29, 2026; independent research context, not a commercial or causal claim.