Make Your Site Readable by AI: A Checklist for AI Crawlers, robots.txt and Structured Data

Making your site readable by AI requires three things. First, robots.txt must not block AI crawlers used for search. Second, your article text must be in the HTML itself, not injected later by JavaScript. Third, structured data should mark up the author, dates and FAQ. If AI crawlers cannot get in or cannot read the text, AI cannot cite you, however good the content is.

This is step four of the five GEO steps. Unlike the other four, it checks the whole site and usually only needs to be done once. The sections below cover what each AI crawler is for, how to set up robots.txt, how to confirm your text is readable, and what structured data, sitemaps and llms.txt do.

Why check that AI can read your site first?

AI tools search the web before answering, then pick passages from the pages they read. Google states that to appear as a supporting link in AI features, "a page must be indexed and eligible to be shown in Google Search with a snippet," and that there are no additional technical requirements (Google Search Central: AI features and your website).

In other words, AI can only cite you after a search system has read your page. The GEO paper's experiments rest on the same premise: the generative engine chooses sources only from the pages it retrieved (GEO paper). So the first time you do GEO, finish this step before you start writing.

What does each AI crawler do?

Most AI companies split their crawlers in two: one collects data to train models, the other searches in real time when answering. You can control them separately, so you can keep your content out of training and still be found by AI search.

Details worth knowing:

• OpenAI: the search crawler OAI-SearchBot and the training crawler GPTBot are controlled separately, so blocking GPTBot does not affect ChatGPT search (OpenAI crawlers).
• Anthropic: blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's search results (Anthropic Help Center).
• Perplexity: PerplexityBot surfaces and links websites in search results; Perplexity-User fetches a page only when a user asks, and Perplexity says it generally ignores robots.txt (Perplexity crawlers).
• Google: Google-Extended does not affect whether a site is included in Google Search and is not a ranking signal (Google's common crawlers). AI Overviews rely on regular Search indexing; only blocking Googlebot removes you from them.

How should you set up robots.txt?

robots.txt is a text file at the root of your site that tells crawlers which paths they may fetch. The most common problem is not forgetting to allow AI crawlers; it is an old "block everything" rule written to stop spam bots that also blocks AI search crawlers.

Steps to check:

1. Open your-domain/robots.txt in a browser.
2. Look for User-agent: * followed by Disallow: /. If it is there, every crawler is blocked.
3. Search for the crawler names in the table and confirm the search crawlers are not under Disallow: /.
4. To block training but not search, write Disallow: / only for GPTBot, ClaudeBot and Google-Extended, and leave the rest allowed.
5. For private pages such as admin or member areas, add Disallow rules for those paths only.

Note that robots.txt only asks crawlers not to fetch a page. It is not a way to keep a page out of search results; for Google, use noindex or password protection instead (Google robots.txt introduction).

How do you confirm your text is in the HTML?

On some sites, article text only appears after the browser runs JavaScript. Google runs JavaScript, but not every crawler does; Google itself says server-side rendering or pre-rendering is still a great idea because not all bots can run JavaScript (Google JavaScript SEO basics).

The simplest check needs no tools:

1. Open one of your articles, right-click and choose "View page source."
2. Press Ctrl + F and search for a sentence from the middle of the article.
3. If you find it, the text is in the HTML. If not, it depends on JavaScript, and some AI crawlers will miss it.

A common gap is an FAQ section that expands on click. If the answers are loaded by script after the click, the source has no answers. Use the HTML details and summary elements instead: the answer stays in the source and is only collapsed on screen.

What should structured data mark up?

Structured data is a standardized format for describing a page and classifying its content. Google recommends JSON-LD in most cases (Google intro to structured data).

The three most useful types for GEO:

The key rule: marked-up content must match what readers can see on the page, and you should not mark up content that is not visible (Google general structured data guidelines). Keep expectations realistic: since August 2023, Google shows FAQ rich results only for well-known, authoritative government and health websites (Google announcement). Add FAQPage to make the structure clearer, not to get extra space in search results.

What do sitemaps and llms.txt do?

A sitemap (sitemap.xml) lists the pages on your site so search engines can crawl it more efficiently and are less likely to miss new articles (Google sitemap overview). It does not guarantee indexing, but it shortens the time before a new article is found. Add the sitemap location as the last line of robots.txt.

llms.txt is a Markdown file at the root of your site written for AI: a one-sentence definition of the site plus its key pages and articles (llms.txt proposal). It is currently a community proposal, and no AI company has formally committed to reading it. It is cheap to make, so do it, but do not treat it as the core of GEO.

A one-time checklist

Each of these 8 items can be checked in under 10 minutes:

Item 5 is easy to miss. Google notes that controls such as nosnippet and max-snippet also limit how your content appears in AI features (Google AI features).

What comes next?

Once this step is done, go back to the flow every topic follows: choose a topic, write, publish, then measure. After publishing a new article, use items 2, 4 and 7 of the checklist to confirm it is actually readable, then wait about 4 weeks before comparing measurements.

If measurements do not improve, check this step again before blaming the content. Related articles: How to do GEO, How AI chooses which sources to cite, How to measure AI visibility.

FAQ

Q: How do I make my site readable by AI?

Allow AI search crawlers in robots.txt, put your text directly in the HTML, and mark up author and dates with structured data.

Once those three are done, confirm the page is indexed by search engines. Only then can AI cite you in its answers.

Q: Will blocking GPTBot stop ChatGPT search from finding me?

No. GPTBot is the training crawler; ChatGPT search uses OAI-SearchBot.

OpenAI controls them separately. If you only want to keep content out of training, block GPTBot and keep OAI-SearchBot allowed.

Q: Will blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect whether a site is included in Google Search.

AI Overviews use regular Search indexing. Only blocking Googlebot or adding noindex stops a page from being a supporting link.

Q: Can robots.txt keep a page out of search results?

No. robots.txt only asks crawlers not to fetch a page; it is not a way to exclude a page from search.

To keep a page out of Google Search, use noindex or password protection.

Q: How do I know whether AI crawlers can read my article text?

Right-click the article page, choose "View page source," and search for a sentence from the middle of the article.

If you find it, the text is in the HTML. If not, it appears only after JavaScript runs, and some AI crawlers will miss it.

Q: Do I have to add structured data?

It is recommended. At minimum, mark up the Article author and dates.

Structured data helps AI judge who wrote the content and how current it is. The markup must match what is visible on the page.

Q: Is llms.txt useful?

It helps a little but is not the core. llms.txt is a site description file for AI and is currently a community proposal.

No AI company has formally committed to reading it. It is cheap to make, but robots.txt, HTML text and structured data matter more.

Q: How often should I do this step?

The site-level setup only needs to be done once; check again after a redesign or a platform change.

After each new article, just confirm the text is in the source, the page is indexed and it appears in the sitemap.