[{"data":1,"prerenderedAt":412},["ShallowReactive",2],{"insight-\u002Finsights\u002Fai-crawler-technical-checklist":3,"insight-service-\u002Finsights\u002Fai-crawler-technical-checklist":409},{"id":4,"title":5,"author":6,"body":7,"category":378,"date":379,"description":380,"extension":381,"faq":382,"meta":397,"navigation":398,"path":399,"seo":400,"service":360,"sitemap":401,"stem":402,"tags":403,"updated":407,"__hash__":408},"insights\u002Finsights\u002Fai-crawler-technical-checklist.md","Can AI Assistants Read Your Website? A Technical Checklist for AI Search","Cinnamon Digital",{"type":8,"value":9,"toc":363},"minimark",[10,18,23,35,133,136,140,143,161,165,172,175,179,182,226,229,233,236,255,259,262,288,292,306,310,317,321,327,331,334,345,349],[11,12,13,17],"p",{},[14,15,16],"strong",{},"AI assistants can only recommend what they can read."," Before investing in content or authority building for AI search, confirm that the crawlers behind ChatGPT, Google AI Overviews, Perplexity, Claude, and Copilot can reach your pages, read the text, and understand who you are. This checklist covers the technical foundation.",[19,20,22],"h2",{"id":21},"_1-check-robotstxt-for-ai-crawlers","1. Check robots.txt for AI crawlers",[11,24,25,26,30,31,34],{},"Open ",[27,28,29],"code",{},"yourdomain.com\u002Frobots.txt"," and look for rules that block AI user agents, either by name or through a broad ",[27,32,33],{},"Disallow: \u002F",". Common crawlers to allow if you want AI visibility:",[36,37,38,54],"table",{},[39,40,41],"thead",{},[42,43,44,48,51],"tr",{},[45,46,47],"th",{},"Crawler",[45,49,50],{},"Operator",[45,52,53],{},"Used for",[55,56,57,69,79,89,100,111,122],"tbody",{},[42,58,59,63,66],{},[60,61,62],"td",{},"OAI-SearchBot",[60,64,65],{},"OpenAI",[60,67,68],{},"Surfacing sites in ChatGPT search results",[42,70,71,74,76],{},[60,72,73],{},"GPTBot",[60,75,65],{},[60,77,78],{},"Collecting content for model training",[42,80,81,84,86],{},[60,82,83],{},"ChatGPT-User",[60,85,65],{},[60,87,88],{},"Fetching pages when a user's request needs them",[42,90,91,94,97],{},[60,92,93],{},"ClaudeBot",[60,95,96],{},"Anthropic",[60,98,99],{},"Crawling for Claude",[42,101,102,105,108],{},[60,103,104],{},"PerplexityBot",[60,106,107],{},"Perplexity",[60,109,110],{},"Indexing for Perplexity answers",[42,112,113,116,119],{},[60,114,115],{},"Google-Extended",[60,117,118],{},"Google",[60,120,121],{},"Controls use of content in Google's AI models (uses Googlebot's crawl)",[42,123,124,127,130],{},[60,125,126],{},"Bingbot",[60,128,129],{},"Microsoft",[60,131,132],{},"Bing search, which powers Copilot answers",[11,134,135],{},"Operators update crawler names and behavior, so verify against each provider's current documentation. Some businesses choose to allow search and retrieval crawlers while blocking training crawlers; that is a valid decision as long as it is deliberate.",[19,137,139],{"id":138},"_2-confirm-your-cdn-and-firewall-are-not-blocking-them","2. Confirm your CDN and firewall are not blocking them",[11,141,142],{},"Robots.txt is a request, not an enforcement layer. The opposite problem is also common: a CDN, web application firewall, or hosting provider's bot protection may block AI crawlers even though robots.txt allows them.",[144,145,146,150],"ul",{},[147,148,149],"li",{},"Review bot management settings in your CDN or host.",[147,151,152,153,156,157,160],{},"Search server or CDN logs for the user agents above and confirm they receive ",[27,154,155],{},"200"," responses, not ",[27,158,159],{},"403"," errors or challenge pages.",[19,162,164],{"id":163},"_3-serve-real-html","3. Serve real HTML",[11,166,167,168,171],{},"Request a key page without JavaScript, for example with ",[27,169,170],{},"curl",", or view the page source rather than the rendered page. If your main content, headings, and prices are missing from the raw HTML, many AI crawlers will not see them either.",[11,173,174],{},"Fixes include server-side rendering (SSR), static site generation (SSG), or prerendering for important pages. Frameworks like Nuxt and Next.js support these modes out of the box.",[19,176,178],{"id":177},"_4-publish-structured-data","4. Publish structured data",[11,180,181],{},"Schema.org markup in JSON-LD format gives machines unambiguous facts about your business. Prioritize:",[144,183,184,198,208,214,220],{},[147,185,186,189,190,193,194,197],{},[14,187,188],{},"Organization"," or ",[14,191,192],{},"LocalBusiness"," on your homepage or About page, with name, logo, URL, contact details, and ",[27,195,196],{},"sameAs"," links to official profiles.",[147,199,200,203,204,207],{},[14,201,202],{},"Service"," and ",[14,205,206],{},"Offer"," on service and pricing pages.",[147,209,210,213],{},[14,211,212],{},"Article"," on blog posts and guides, with author and dates.",[147,215,216,219],{},[14,217,218],{},"FAQPage"," where you answer common questions on the page.",[147,221,222,225],{},[14,223,224],{},"BreadcrumbList"," to clarify your site structure.",[11,227,228],{},"Validate your markup with Google's Rich Results Test or the Schema.org validator, and make sure the facts in markup match the visible page.",[19,230,232],{"id":231},"_5-keep-your-sitemap-complete-and-current","5. Keep your sitemap complete and current",[11,234,235],{},"An XML sitemap helps crawlers find every important page and know when it changed.",[144,237,238,245,252],{},[147,239,240,241,244],{},"Include every page you want cited, and exclude redirects, duplicates, and ",[27,242,243],{},"noindex"," pages.",[147,246,247,248,251],{},"Keep ",[27,249,250],{},"lastmod"," dates accurate, so crawlers know which pages are worth revisiting.",[147,253,254],{},"Reference the sitemap in robots.txt and submit it in Google Search Console and Bing Webmaster Tools.",[19,256,258],{"id":257},"_6-make-pages-easy-to-quote","6. Make pages easy to quote",[11,260,261],{},"Technical access gets you read; structure gets you quoted.",[144,263,264,279,282,285],{},[147,265,266,267,270,271,274,275,278],{},"Use one clear ",[27,268,269],{},"\u003Ch1>"," and logical ",[27,272,273],{},"\u003Ch2>","\u002F",[27,276,277],{},"\u003Ch3>"," headings.",[147,280,281],{},"Open important pages with a direct, one-sentence answer or definition.",[147,283,284],{},"Use real HTML tables and lists for comparisons and specifications instead of images.",[147,286,287],{},"State facts plainly: what you do, who you serve, where, and what it costs if you publish pricing.",[19,289,291],{"id":290},"_7-avoid-content-that-crawlers-cannot-reach","7. Avoid content that crawlers cannot reach",[144,293,294,297,300,303],{},[147,295,296],{},"Key information inside images, PDFs, or videos without a text equivalent.",[147,298,299],{},"Content behind logins, cookie walls, or interstitials that block the first page load.",[147,301,302],{},"Infinite scroll or \"load more\" buttons with no crawlable pagination.",[147,304,305],{},"Tabs or accordions whose content is fetched only on click rather than present in the HTML.",[19,307,309],{"id":308},"_8-keep-pages-fast-and-stable","8. Keep pages fast and stable",[11,311,312,313,316],{},"Crawlers have time and resource limits. Slow responses, frequent timeouts, and server errors reduce how much of your site gets crawled. Monitor uptime, fix ",[27,314,315],{},"5xx"," errors, and keep response times low on your most important pages.",[19,318,320],{"id":319},"_9-consider-an-llmstxt-file","9. Consider an llms.txt file",[11,322,323,326],{},[27,324,325],{},"llms.txt"," is a proposed standard: a Markdown file at your site's root that summarizes your business and links to key pages. Adoption by AI providers is still uneven, so it should not be your priority, but it is cheap to add once the items above are done.",[19,328,330],{"id":329},"_10-verify-the-result","10. Verify the result",[11,332,333],{},"After making changes, check that it worked:",[144,335,336,339,342],{},[147,337,338],{},"Fetch your pages as different user agents and confirm full HTML is returned.",[147,340,341],{},"Watch server logs for AI crawler visits over the following weeks.",[147,343,344],{},"Ask the major assistants questions about your business and note whether they cite your pages.",[19,346,348],{"id":347},"where-to-start","Where to start",[11,350,351,352,357,358,362],{},"Most sites can complete this checklist in a few days, and every later improvement in content and authority depends on it. If you want an outside review, the technical check is included in our ",[353,354,356],"a",{"href":355},"\u002Fai-visibility-snapshot","AI Visibility Snapshot",", and our ",[353,359,361],{"href":360},"\u002Fservices\u002Fai-search-visibility","AI search visibility"," team can implement the fixes.",{"title":364,"searchDepth":365,"depth":365,"links":366},"",2,[367,368,369,370,371,372,373,374,375,376,377],{"id":21,"depth":365,"text":22},{"id":138,"depth":365,"text":139},{"id":163,"depth":365,"text":164},{"id":177,"depth":365,"text":178},{"id":231,"depth":365,"text":232},{"id":257,"depth":365,"text":258},{"id":290,"depth":365,"text":291},{"id":308,"depth":365,"text":309},{"id":319,"depth":365,"text":320},{"id":329,"depth":365,"text":330},{"id":347,"depth":365,"text":348},"AI Search","2026-11-10","A technical checklist for making your website readable by ChatGPT, Google AI Overviews, Perplexity, Claude, and Copilot, covering robots.txt, AI crawlers, server rendering, structured data, sitemaps, and page performance.","md",{"title":383,"items":384},"Technical AI search questions",[385,388,391,394],{"label":386,"content":387},"Which AI crawlers should I allow in robots.txt?","If you want to be found and cited in AI answers, allow the crawlers used by the major assistants, including OAI-SearchBot and GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Google's AI products), and Bingbot (Microsoft Copilot and Bing). Check each provider's documentation, because crawler names and purposes change over time.",{"label":389,"content":390},"Do AI crawlers run JavaScript?","Many do not, or do so inconsistently. Content that only appears after client-side JavaScript runs may be invisible to them. Server-side rendering or static generation makes sure your text is in the initial HTML response.",{"label":392,"content":393},"What is llms.txt and do I need one?","llms.txt is a proposed convention for a plain-text file at the root of a site that summarizes the site and links to its key pages for language models. Support across AI providers is limited and inconsistent, so treat it as a low-cost extra, not a replacement for crawlable HTML and structured data.",{"label":395,"content":396},"Can a firewall or CDN block AI crawlers without me knowing?","Yes. Bot protection settings at CDNs and hosting providers can block or challenge AI crawlers even when robots.txt allows them. Check your CDN's bot settings and server logs to confirm crawlers receive successful responses.",{},true,"\u002Finsights\u002Fai-crawler-technical-checklist",{"title":5,"description":380},{"loc":399,"lastmod":379},"insights\u002Fai-crawler-technical-checklist",[404,405,406],"AI search","Technical SEO","Generative engine optimization",null,"zEdPP2cN22QDpBVDJSB6FFlX-J2v-carvcs7XaHa5Zk",{"path":360,"title":410,"description":411},"AI Search Visibility","Generative engine optimization (GEO) and AI brand presence.",1794506504720]