Répertoire des robots IA
Consultez les user-agents documentés par opérateur et par usage.
GPTBot
GPTBotControls use for improving OpenAI generative AI foundation models.
Voir le robot →AI AssistantsChatGPT-User
ChatGPT-UserUser-triggered visits from ChatGPT.
Voir le robot →AI SearchOAI-SearchBot
OAI-SearchBotSurfaces websites in ChatGPT search products.
Voir le robot →AI TrainingClaudeBot
ClaudeBotAutomated web crawling by Anthropic.
Voir le robot →AI SearchClaude-SearchBot
Claude-SearchBotSupports search results for Claude users.
Voir le robot →AI AssistantsClaude-User
Claude-UserUser-initiated retrieval in Claude.
Voir le robot →AI TrainingGoogle-Extended
Google-ExtendedControls use of crawled content for Gemini models and grounding.
Voir le robot →Search EnginesGooglebot
GooglebotGoogle web search crawler.
Voir le robot →Search EnginesGooglebot Image
Googlebot-ImageGoogle image search crawler.
Voir le robot →Search EnginesBingbot
bingbotMicrosoft Bing search crawler.
Voir le robot →Search EnginesApplebot
ApplebotSupports Apple search products.
Voir le robot →AI TrainingApplebot-Extended
Applebot-ExtendedControls use of crawled content for Apple generative models.
Voir le robot →AI SearchPerplexityBot
PerplexityBotIndexes content for Perplexity search.
Voir le robot →AI AssistantsPerplexity-User
Perplexity-UserUser-requested retrieval by Perplexity.
Voir le robot →Archive/Data CrawlersCCBot
CCBotBuilds the Common Crawl public web corpus.
Voir le robot →AI TrainingBytespider
BytespiderByteDance web crawler; published purpose details are limited.
Voir le robot →AI SearchAmazonbot
AmazonbotSupports Amazon services including search and assistant features.
Voir le robot →Social CrawlersFacebookBot
FacebookBotFetches content for Meta link and AI features.
Voir le robot →Social CrawlersTwitterbot
TwitterbotGenerates link previews on X.
Voir le robot →Social CrawlersLinkedInBot
LinkedInBotGenerates LinkedIn link previews.
Voir le robot →Search EnginesDuckDuckBot
DuckDuckBotDuckDuckGo search crawler.
Voir le robot →Search EnginesYandexBot
YandexBotYandex search crawler.
Voir le robot →Search EnginesBaiduspider
BaiduspiderBaidu search crawler.
Voir le robot →SEO CrawlersAhrefsBot
AhrefsBotBuilds Ahrefs search and SEO datasets.
Voir le robot →SEO CrawlersSemrushBot
SemrushBotCollects data for Semrush SEO products.
Voir le robot →