Can AI Assistants Read Your Website? A Technical Checklist for AI Search
A technical checklist for making your website readable by ChatGPT, Google AI Overviews, Perplexity, Claude, and Copilot, covering robots.txt, AI crawlers, server rendering, structured data, sitemaps, and page performance.
By Cinnamon Digital ·
AI assistants can only recommend what they can read. Before investing in content or authority building for AI search, confirm that the crawlers behind ChatGPT, Google AI Overviews, Perplexity, Claude, and Copilot can reach your pages, read the text, and understand who you are. This checklist covers the technical foundation.
1. Check robots.txt for AI crawlers
Open yourdomain.com/robots.txt and look for rules that block AI user agents, either by name or through a broad Disallow: /. Common crawlers to allow if you want AI visibility:
| Crawler | Operator | Used for |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search results |
| GPTBot | OpenAI | Collecting content for model training |
| ChatGPT-User | OpenAI | Fetching pages when a user's request needs them |
| ClaudeBot | Anthropic | Crawling for Claude |
| PerplexityBot | Perplexity | Indexing for Perplexity answers |
| Google-Extended | Controls use of content in Google's AI models (uses Googlebot's crawl) | |
| Bingbot | Microsoft | Bing search, which powers Copilot answers |
Operators update crawler names and behavior, so verify against each provider's current documentation. Some businesses choose to allow search and retrieval crawlers while blocking training crawlers; that is a valid decision as long as it is deliberate.
2. Confirm your CDN and firewall are not blocking them
Robots.txt is a request, not an enforcement layer. The opposite problem is also common: a CDN, web application firewall, or hosting provider's bot protection may block AI crawlers even though robots.txt allows them.
- Review bot management settings in your CDN or host.
- Search server or CDN logs for the user agents above and confirm they receive
200responses, not403errors or challenge pages.
3. Serve real HTML
Request a key page without JavaScript, for example with curl, or view the page source rather than the rendered page. If your main content, headings, and prices are missing from the raw HTML, many AI crawlers will not see them either.
Fixes include server-side rendering (SSR), static site generation (SSG), or prerendering for important pages. Frameworks like Nuxt and Next.js support these modes out of the box.
4. Publish structured data
Schema.org markup in JSON-LD format gives machines unambiguous facts about your business. Prioritize:
- Organization or LocalBusiness on your homepage or About page, with name, logo, URL, contact details, and
sameAslinks to official profiles. - Service and Offer on service and pricing pages.
- Article on blog posts and guides, with author and dates.
- FAQPage where you answer common questions on the page.
- BreadcrumbList to clarify your site structure.
Validate your markup with Google's Rich Results Test or the Schema.org validator, and make sure the facts in markup match the visible page.
5. Keep your sitemap complete and current
An XML sitemap helps crawlers find every important page and know when it changed.
- Include every page you want cited, and exclude redirects, duplicates, and
noindexpages. - Keep
lastmoddates accurate, so crawlers know which pages are worth revisiting. - Reference the sitemap in robots.txt and submit it in Google Search Console and Bing Webmaster Tools.
6. Make pages easy to quote
Technical access gets you read; structure gets you quoted.
- Use one clear
<h1>and logical<h2>/<h3>headings. - Open important pages with a direct, one-sentence answer or definition.
- Use real HTML tables and lists for comparisons and specifications instead of images.
- State facts plainly: what you do, who you serve, where, and what it costs if you publish pricing.
7. Avoid content that crawlers cannot reach
- Key information inside images, PDFs, or videos without a text equivalent.
- Content behind logins, cookie walls, or interstitials that block the first page load.
- Infinite scroll or "load more" buttons with no crawlable pagination.
- Tabs or accordions whose content is fetched only on click rather than present in the HTML.
8. Keep pages fast and stable
Crawlers have time and resource limits. Slow responses, frequent timeouts, and server errors reduce how much of your site gets crawled. Monitor uptime, fix 5xx errors, and keep response times low on your most important pages.
9. Consider an llms.txt file
llms.txt is a proposed standard: a Markdown file at your site's root that summarizes your business and links to key pages. Adoption by AI providers is still uneven, so it should not be your priority, but it is cheap to add once the items above are done.
10. Verify the result
After making changes, check that it worked:
- Fetch your pages as different user agents and confirm full HTML is returned.
- Watch server logs for AI crawler visits over the following weeks.
- Ask the major assistants questions about your business and note whether they cite your pages.
Where to start
Most sites can complete this checklist in a few days, and every later improvement in content and authority depends on it. If you want an outside review, the technical check is included in our AI Visibility Snapshot, and our AI search visibility team can implement the fixes.