Three different AI crawlers visit dental websites and they want different things. One puts your pages into an answer, one fetches a page because a patient asked for it, and one may collect your content for training. Allow the first two on purpose, decide about the third deliberately, and make sure no blanket rule blocks the first two by accident.

Three Kinds Of AI Crawler

The same vendor runs more than one crawler, and one decision covering all of them is the wrong decision. OpenAI documents three of its own in its crawler documentation: OAI-SearchBot for search indexing, ChatGPT-User for pages a person explicitly asks ChatGPT to open, and GPTBot for potential model training. Perplexity splits its crawlers along the same lines with PerplexityBot and Perplexity-User . Anthropic's crawler documentation covers ClaudeBot, Claude-User and Claude-SearchBot, and it is worth reading in full because those three names are easy to mix up.

BEFORE
robots.txt
User-agent: *
Disallow: /
AFTER
robots.txt
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /   # your decision
Illustrative example.

Grouped by what they are actually for, that gives three families.

Search Discovery

OAI-SearchBot, PerplexityBot and Claude-SearchBot. These fetch your pages into a search index that can then be quoted inside an answer. If a patient asks for a dentist in your city and the engine has never fetched your site, you are not in the running.

User-triggered Fetch

ChatGPT-User, Perplexity-User and Claude-User. These arrive because somebody asked the assistant to look at your page right now, and they carry a referral from a human. They do not index anything. Blocking them is the most expensive mistake in this article: you turn away the visit a patient asked for.

Potential Training

GPTBot, ClaudeBot and meta-externalagent. These fetch pages for possible use in training data. What happens to the content afterwards is the vendor's decision, which is why the sensible answer is a deliberate choice rather than a panic.

What Blocking Each One Costs

This assumes you want to be found and quoted, and have no feelings about training crawlers. Read the last column before you copy any file.

Swipe the table sideways to see every column.

CrawlerWhat It DoesIf You Block ItWhat You Lose
OAI-SearchBotIndexes your pages so ChatGPT can search themYour pages are never indexed for ChatGPT searchEvery question answered through ChatGPT search, in every city you serve
PerplexityBotIndexes your pages for Perplexity searchPerplexity cannot cite you from its indexAnswers that carry a sources list, and the click-through after it
Claude-SearchBotIndexes your pages for Claude searchClaude search cannot find youVisibility in the one engine many patients now open first
ChatGPT-UserFetches a page a person asked ChatGPT to openThe request fails and the assistant cannot read the pageThe exact page a patient asked about, at the moment they asked
Claude-UserFetches a page a person asked Claude to openThe request fails in the same wayThe same moment of patient intent, in a different assistant
Perplexity-UserFetches a page a person asked Perplexity to openThe request failsNothing else, provided PerplexityBot is still allowed
GPTBotCollects pages for possible model trainingYour content stays out of that training setNothing you can measure today, and maybe future reach in new markets
ClaudeBotCollects pages for possible model trainingYour content stays out of that training setNothing you can measure today
meta-externalagentCollects pages for Meta AI trainingYour content stays out of that training setNothing you can measure today

The first six rows cost you traffic you can count. The last three cost you something you cannot count today. That asymmetry is the whole argument.

The Paste-ready robots.txt

Start from the file below, change the two paths you want to keep private, and delete nothing else. It allows search discovery and user-triggered fetches, and leaves the training crawlers commented out so the decision is visible to whoever reads the file next.

# robots.txt for a single-location dental practice

# Change the paths and the sitemap address before you publish this.

``

User-agent: *

Allow: /

Disallow: /preview/

Disallow: /thank-you.html

Crawl-delay: 1

``

# AI search discovery: the crawlers that put your pages into an answer

User-agent: OAI-SearchBot

Allow: /

``

User-agent: PerplexityBot

Allow: /

``

User-agent: Claude-SearchBot

Allow: /

``

# AI fetchers that run because a person asked for your page

User-agent: ChatGPT-User

Allow: /

``

User-agent: Perplexity-User

Allow: /

``

User-agent: Claude-User

Allow: /

``

# Training crawlers. Opt in by removing the # from the lines below.

# User-agent: GPTBot

# Allow: /

``

# User-agent: ClaudeBot

# Allow: /

``

# User-agent: meta-externalagent

# Allow: /

``

Sitemap: https://example.com/sitemap.xml

Two notes. Crawl-delay: 1 asks crawlers to wait a second between requests; it is a politeness setting not every crawler honours, so treat it as a request and not a promise. And the training block is left commented on purpose, because a decision written in the file is one your next office manager can see instead of one they have to guess at.

On the difference between llms.txt and this file, what llms.txt is and whether it helps covers the part people confuse most often.

The Mistakes That Quietly Exclude You

None of these produce an error message. Each one simply removes you from the index while your site looks perfectly healthy.

A Named Group Overrides The Wildcard

Rules for User-agent: * do not merge with the rules for a named crawler. If your wildcard group says Allow: / and a separate group says User-agent: GPTBot with Disallow: /, that crawler is blocked everywhere, and the wildcard does nothing for it. The same trap runs the other way: a plugin, a security tool or a copy of a template can leave behind a named group that blocks paths for one bot that you assumed were covered by the wildcard. Read the file top to bottom and look for every named group, not just the first one.

Blocking The Folder That Holds Your Assets

A rule such as Disallow: /assets/ or Disallow: /cdn-cgi/ looks harmless because the page still loads. It is not harmless. Without the stylesheet and the script, the page arrives at a crawler as an unstyled skeleton, and a crawler that cannot render may never extract the text that matters.

Blocking The Code That Draws The Page

Scripts and stylesheets are not decoration to a crawler that renders. If the text you want quoted is assembled by JavaScript, the crawler has to fetch that JavaScript and run it. A rule that stops it fetching the script removes the answer along with the styling.

Hiding The File Itself

A robots.txt that answers with a redirect, a 404, a login page, a firewall challenge or a server error is treated as a complete instruction to stay away. The most common cause is not a rule at all: it is a security product challenging what it thinks is a bot, which turns your entire site into a blocked site for that crawler.

Check It Actually Works

Ten minutes of checking beats a month of guessing. Run through this list after you publish the file.

  • Fetch the file directly and confirm it returns the text you wrote, not a redirect, a login page or a challenge screen.
  • Read it as a person would, top to bottom, and note every named group you did not write yourself.
  • Confirm each crawler you care about appears in the group you intended, and that no later group contradicts an earlier one.
  • Request one of your real treatment pages with a crawler user agent and check that the page comes back complete, with the text in the HTML rather than behind a script.
  • Look at your server logs for user-agent strings you do not recognise, and decide what to do with each one.

The last two checks matter because a perfect rules file cannot rescue a page the crawler cannot load. If the page is slow or half client-rendered, the problem is described in does website speed affect AI visibility. If there is no page for the question the crawler was sent, the fix is structural: how to structure a dental site one page per question.

Review It Twice a Year

The crawler list changes. Vendors add agents, rename them, split one bot into two, or publish one for a product that did not exist when your file was written. A file correct in January is quietly incomplete by July, and an unrecognised crawler is treated as untrusted by default.

Put a reminder in January and July. On each date, read the vendor documentation for the engines you care about, check your logs for new user agents, and update the file in one place. If you run several sites, keep the dated copies together, because the fastest way to understand a sudden loss of visibility is to compare the file you have now with the one you had last time.

If you want the wider picture of how a practice gets found by these engines, the drsmiles.ai snapshot measures ten questions across three engines in about sixty seconds, and the method is written up in Dental AEO. A finished report looks like our sample report.