Bots now outnumber humans on the internet, and three names show up in nearly every website's server logs: GPTBot, ClaudeBot, and PerplexityBot. AI-related crawlers made up 52% of all crawler requests in June 2026, up from 22% a year earlier, according to Cloudflare. On September 15, Cloudflare starts blocking mixed-use AI crawlers by default on many sites, reports TechCrunch's Sarah Perez. Most business owners have never decided whether to let these bots in. This article explains what each one actually does, what it takes versus what it sends back, and how to position your business so the decision works in your favor. First, they aren't three bots. They're three families. The names people know are the training crawlers. But OpenAI, Anthropic, and Perplexity each run a set of bots with different jobs, and treating them as one thing is the most common mistake in AI crawler policy. Each company separates three activities: collecting content to train models, indexing content so it can be cited in AI search answers, and fetching a page live because a real user just asked about it. Company Training crawler Search indexer Live user fetch OpenAI (ChatGPT) GPTBot OAI-SearchBot ChatGPT-User Anthropic (Claude) ClaudeBot Claude-SearchBot Claude-User Perplexity None declared PerplexityBot Perplexity-User GPTBot: OpenAI's training crawler GPTBot gathers public web content that may be used to train OpenAI's models. It is the most-blocked AI bot on the web, named in roughly 18% of the robots.txt files Cloudflare Radar samples. Blocking it keeps your future content out of OpenAI's training data. It does not remove you from ChatGPT search; that is OAI-SearchBot's job, controlled separately. ChatGPT-User fires when a person asks ChatGPT something that needs your live page, and OpenAI notes that robots.txt rules may not apply to it because a human initiated the request. OpenAI publishes IP ranges for all three so you can verify a visitor is real. ClaudeBot: Anthropic's training crawler ClaudeBot does for Anthropic what GPTBot does for OpenAI, and in 2026 it became the busier of the two. Cloudflare data shows ClaudeBot overtook GPTBot in June and held the top AI-crawler slot through July, at roughly 16 to 20% of all AI-bot traffic. Anthropic's documentation is unusually clear: it honors robots.txt, supports crawl-delay, won't try to bypass CAPTCHAs, and if you block ClaudeBot, your future content is excluded from training. Claude-SearchBot indexes for Claude's search answers; Claude-User fetches pages on a user's behalf and is now the second most active crawler on the entire web after Googlebot, per SEOmator's Nick Sawinyh. PerplexityBot: the answer engine's indexer Perplexity is different. It doesn't train a foundation model on your content; PerplexityBot exists to surface and link your pages in Perplexity's answers. That should make it the friendliest of the three. In practice it is the most contested. Perplexity-User, its live-fetch agent, generally ignores robots.txt, and Perplexity's position is that it is "an agent, not a bot" acting for a person. In August 2025, Cloudflare published evidence that Perplexity was also crawling through undeclared bots with rotating identities to get around blocks, and removed it from its verified-bot list. Perplexity disputes the characterization. Either way, if you block PerplexityBot, assume some Perplexity traffic will still arrive. What they take versus what they give back The metric that matters is the crawl-to-refer ratio: how many of your pages a company's bots fetch for every visitor its product sends you. Cloudflare Radar's numbers for the 28 days ending September 2, 2026, reported by SEOmator's Isabella Edwards, tell the story in one line. Operator Pages crawled per referral Trend Perplexity 885 : 1 Up 139% in one window Anthropic 823 : 1 Halved from 1,601 : 1 OpenAI 455 : 1 Doubled from 226 : 1 Google 5.2 : 1 Stable Google crawls five pages per visitor it sends you. The AI companies crawl hundreds. That is the case for blocking. The case against is the quality of the few visitors they do send: Adobe's analytics research finds AI-referred visitors generate 37% more revenue per visit and stay 48% longer than other traffic, and Similarweb counts 770 million AI referral visits a month, up 117% year over year. And a bot can shape how an assistant describes your business without ever sending a click. Blocking is not free. Why September 15 changes the default Until now the default everywhere was "allow unless you say otherwise." Cloudflare, which fronts a large share of the web, is flipping that for mixed-use crawlers on ad-supported pages: new customers, new sites, and every existing free-plan site will block by default from September 15, 2026. Cloudflare is also turning its Pay Per Crawl marketplace into Pay Per Use, where publishers get paid when their content appears in an AI answer. The practical effect for a small business: you may be blocking AI bots right now without having decided to, and the platforms are being pushed to separate search from training so you can allow one without the other. How to think about it: three decisions, not one Split the question the way the bots are split. Training crawlers (GPTBot, ClaudeBot) are a licensing decision. You are deciding whether your content becomes free raw material. If your content is your product, block or charge. If your content exists to get you found, allowing it helps models recognize your brand. Search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are a distribution decision. Blocking them is opting out of the fastest-growing referral channel on the web. Almost every business should allow them. User-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) are a customer decision. There is a person on the other end asking about you. Allow them, and make sure the page they land on answers the question. 6 moves to position your business 1. Find out what you're already doing Check your robots.txt, your CDN's bot settings, and your firewall. Many hosts and CDNs now block AI bots by default, and after September 15 more will. If you have never touched these settings, you don't have a policy; you have someone else's. 2. Write a robots.txt that matches the three decisions Name each bot and give it the answer you actually mean. A common small-business configuration allows search and user fetch and reserves the training decision: User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User User-agent: Claude-User Allow: / User-agent: GPTBot User-agent: ClaudeBot Disallow: /private/ Allow: / Swap the last block to Disallow: / if your content is proprietary. Add Crawl-delay for ClaudeBot if server load is the concern. Then test it; a stray line can block Google too. 3. Make the fetched page worth fetching Three-quarters of AI-bot requests are for plain HTML pages, and only about 7% of top sites serve agents a clean, structured version. Put your offer, price or price range, location, availability, and next step in plain text near the top of the page. Add structured data to products and services. Publish an llms.txt file that tells bots which pages matter. This is the same work that gets you cited, and almost nobody has done it. 4. Verify before you trust Anyone can put "GPTBot" in a user-agent string. OpenAI and Anthropic publish official IP ranges; Cloudflare maintains a verified-bot list. Fake bots wearing real names are now a common scraping tactic. If you allow a bot, allow the real one. 5. Measure referrals, not just crawls In your analytics, create a channel for chatgpt.com, claude.ai, perplexity.ai, and gemini.google.com. Track sessions and conversions from each. Then compare against the crawl volume in your logs. Your own crawl-to-refer ratio, per platform, is the only number that should drive your policy. 6. Revisit every quarter In one 28-day window Anthropic's ratio halved while OpenAI's doubled and Perplexity's rose 139%. The platform that is worst for you today can be best next quarter the moment it ships a product that links out. Set a calendar reminder and re-run steps 1 and 5. What this looks like by business type If you are a... Training (GPTBot, ClaudeBot) Search and user fetch Service business or agency Allow. Recognition is the asset. Allow and optimize the pages they fetch. E-commerce store Allow product pages, block accounts and checkout. Allow. Add product and price markup. Publisher or content-first business Block or enroll in Pay Per Use. Allow. Citations are the new distribution. SaaS with public docs Allow docs, block app and support portal. Allow. Serve markdown to agents. Getting it done: Giggrabbers Most of the moves above are an afternoon for someone who does this weekly and a lost week for someone who doesn't. Giggrabbers connects entrepreneurs and businesses with expert freelance SEO and LLMO specialists and web developers who can audit your bot access, write the robots.txt, add the structured data, and set up AI referral tracking. Or start with GiggrabbersAI: describe what you need and it creates the finished deliverable in minutes — a prioritized SEO playbook with keyword map and ranked fixes, or a working website you can preview live — free to start, saved to your dashboard, with a professional freelancer ready to take over whenever you want. The bottom line GPTBot, ClaudeBot, and PerplexityBot are not one threat to block or one channel to embrace. They are nine bots doing three jobs, and each job deserves its own answer. Decide the training question on your terms, keep the search and user-fetch doors open, make the pages behind those doors answer the customer's question, and measure what comes back. Businesses that do that will be the ones AI assistants describe accurately and send customers to. The rest will be blocked by default and never know what they missed. Sources TechCrunch, Sarah Perez — Cloudflare's new policy pushes AI companies to pay for publishers' content (July 1, 2026) SEOmator, Isabella Edwards — 30+ AI SEO Statistics for 2026 (Cloudflare Radar data, September 2, 2026) SEOmator, Nick Sawinyh — GEO Data Report 2026: Crawl-to-Refer Ratios (July 21, 2026) Cloudflare Radar — AI Insights OpenAI — Overview of OpenAI crawlers Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? Perplexity — Agents or Bots? Making Sense of AI on the Open Web Similarweb — AI Referral Traffic by Industry: 2026 Data Adobe Digital Insights — AI-referred traffic conversion and revenue per visit (2026)