Free crawler list · Updated July 10, 2026

AI Crawler Index

AI bots are visiting websites. See which ones matter.

Browse 64 known AI assistants, search crawlers, and training bots that may be visiting your site.

ChatGPT
Claude
Perplexity
Gemini
Copilot
Meta

Names, companies, purpose, and detection notes. No account required.

Live index preview

Choose a crawler group

2026-07-10

64

Known crawlers

AI assistants, search bots, training crawlers, and research bots.

OAI-SearchBotOpenAISearchHelps ChatGPT find and show websites in search answers.ChatGPT-UserOpenAIAnswersVisits pages when a ChatGPT user asks for web content.ClaudeBotAnthropicTrainingCrawls public web content that may help improve Claude models.PerplexityBotPerplexitySearchCrawls pages used by Perplexity answer and search experiences.

Know which AI crawlers exist. See which ones matter. Decide what to do next.

Last updated 2026-07-10

A few crawlers in the index

AI answers

Bots that fetch pages for answers and citations.

1 crawler

ChatGPT-UserOpenAIVisits pages when a ChatGPT user asks for web content.Look for: ChatGPT-User in user-agent strings.

Search indexes

Crawlers that discover and refresh pages for search.

2 crawlers

OAI-SearchBotOpenAIHelps ChatGPT find and show websites in search answers.Look for: OAI-SearchBot in user-agent strings.PerplexityBotPerplexityCrawls pages used by Perplexity answer and search experiences.Look for: PerplexityBot in user-agent strings.

Training crawlers

Crawlers that collect public pages to improve models.

2 crawlers

GPTBotOpenAICrawls pages that may help improve OpenAI models.Look for: GPTBot in user-agent strings.ClaudeBotAnthropicCrawls public web content that may help improve Claude models.Look for: ClaudeBot in user-agent strings.

Research and product bots

Research, preview, advertising, and product crawlers.

1 crawler

OAI-AdsBotOpenAIChecks pages submitted for ChatGPT ads.Look for: OAI-AdsBot in user-agent strings.

Unlock all 64 crawlers

Get the complete crawler directory.

Get the complete directory with company, purpose, source, and detection notes.

No account or credit card required.