For twenty years, robots.txt existed to keep bandwidth-hungry crawlers off your server. In 2026, for anyone who wants their work to actually be found , that instinct is backwards. If your goal is showing up when someone asks Claude, ChatGPT, or Perplexity a question — not just ranking on page one of Google — you need AI crawlers hitting your pages. That's AEO: Answer Engine Optimization , and it runs on a completely different mechanism than classic SEO. Here's what that actually means in practice, without the hand-waving. The bots aren't one bot The biggest mistake I see (and made myself, initially) is treating "AI crawlers" as a single monolith you can allow or block with one User-agent: * rule. They're not. Each provider runs multiple , functionally distinct bots, and blocking one does nothing to the others: OpenAI : GPTBot (training), OAI-SearchBot (powers ChatGPT Search results), ChatGPT-User (live fetch when a user clicks a citation) Anthropic : ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (fetches a page at a user's direct request) Perplexity : PerplexityBot (its search index crawler), Perplexity-User (user-triggered fetches) Google : Google-Extended (opts you in/out of Gemini training specifically — separate from Googlebot , which handles classic Search and isn't affected by this toggle) Blocking only ClaudeBot does not block Claude-SearchBot or Claude-User — each token needs its own directive . The same logic applies across every vendor: training, search-indexing, and live-fetch are three separate decisions, not one. A concrete consequence: GPTBot scrapes content to train OpenAI's models, while OAI-SearchBot indexes content specifically for ChatGPT Search results — so blocking GPTBot alone doesn't remove you from OpenAI's training pool if you left OAI-SearchBot open, and vice versa. You have to decide, per bot, which behavior you're actually opting in or out of. robots.txt is a request, not a lock Worth being blunt about: robots.txt was standardized as RFC 9309 in 2022, but it's a voluntary request that well-behaved crawlers honor — it carries no legal force and doesn't technically block access . Real enforcement, if you need it, happens at the server or CDN layer. Most of the major players do comply. GPTBot, ClaudeBot, and PerplexityBot each identify themselves with a distinct user-agent string and obey robots.txt rules . But not everyone plays fair — Perplexity has been caught running undeclared crawlers that rotate user-agents, IPs, and ASNs specifically to evade no-crawl directives , and Cloudflare's own investigation concluded that robots.txt alone isn't a reliable defense against that behavior if a bot chooses to ignore it. If you actually need a hard block against a non-compliant crawler, that's a WAF/CDN-level rule, not a text file. The canonical tag claim doesn't hold up I want to flag something I said elsewhere that I'd retract: the idea that a rel="canonical" tag is "the key" signal telling AI crawlers "this is the source of truth." That's not documented anywhere I could verify. Canonical tags are a Google/Bing convention for deduping search index entries — there's no confirmed spec showing GPTBot, ClaudeBot, or PerplexityBot parse and weight it the same way for training ingestion or answer attribution. Treat that one as an unverified guess, not a fact to build a strategy around. What actually moves the needle Based on current documentation and crawler behavior: Explicit per-bot robots.txt entries. List GPTBot , OAI-SearchBot , ChatGPT-User , ClaudeBot , Claude-SearchBot , Claude-User , PerplexityBot , Perplexity-User , and Google-Extended individually. A blanket Allow: / for * doesn't give you granular control over training vs. search vs. live-fetch. Server-rendered, crawlable HTML. Not every AI crawler executes JavaScript. If your content only appears after client-side rendering, some bots simply won't see it. Structured data (schema.org). This one has actual documented consumption by search and AI systems, unlike the canonical-tag claim above. llms.txt . An emerging, not yet standardized convention (proposed by Answer.AI's Jeremy Howard and collaborators) that gives LLMs a curated map of your site's most important content, similar in spirit to sitemap.xml . Adoption sits at around 10% of domains as of mid-2026, so it's early — but it's low-effort and zero-risk to add. Check your CDN layer separately from your robots.txt. This is the one people miss most. If you're on Cloudflare or a similar service, its bot-management dashboard can silently override what your actual robots.txt file says. Research on Cloudflare's network found a meaningful share of sites accidentally blocking major AI crawlers at the CDN while their robots.txt says "allow" — the two layers have to agree, or your intended policy does nothing. A sane starting point # Allow AI search / retrieval (what drives citations) User - agent : OAI - SearchBot Allow : / User - agent : ChatGPT - User Allow : / User - agent : Claude - SearchBot Allow : / User - agent : Claude - User Allow : / User - agent : PerplexityBot Allow : / User - agent : Perplexity - User Allow : / # Training crawlers — allow if you want long-term model familiarity User - agent : GPTBot Allow : / User - agent : ClaudeBot Allow : / User - agent : Google - Extended Allow : / # Standard search engines, unchanged User - agent : * Allow : / Sitemap : https :// yourdomain . com / sitemap . xml Adjust the training-bot section if you have genuinely proprietary content you don't want memorized — that's a legitimate reason to Disallow GPTBot / ClaudeBot / Google-Extended specifically while still keeping the search/retrieval bots open for citations. But for a personal site, a portfolio, or a blog whose entire point is being read — open the gate. If you're auditing your own robots.txt , check your server logs for 403s against OAI-SearchBot , Claude-User , and PerplexityBot specifically — that's the fastest way to catch an accidental block costing you AI-search visibility. Ben Santora - July 2026