Anthropic utilizes three distinct bots for Claude: one for model training, one for indexing web content, and one for fetching pages upon user request. While site owners often block all three to prevent data scraping for training, this practice inadvertently excludes their content from Claude's search results and direct page retrieval.
Data from October 2026 shows that 63 out of 569 analyzed websites with a robots.txt file blocked Claude-SearchBot. Research indicates that Claude uses web search in 36.6% of responses. When search is active, the model relies on external indices; evidence suggests a strong correlation between search results in Brave Search and Claude's citations, as 87.8% of Claude's sources appeared in the top 10 of Brave Search.
To ensure visibility, site owners should configure robots.txt to allow Claude-SearchBot and Claude-User while specifically blocking only ClaudeBot if training data usage is the primary concern. Blocking bots via IP or WAF settings can override robots.txt instructions, potentially preventing the model from accessing the site even if explicitly allowed in the configuration file.
Claude's source selection differs significantly from ChatGPT, with a higher preference for curated lists and lower engagement with forums. Optimization for Claude requires ensuring that pages are indexed by the underlying search provider and that the site's robots.txt and WAF settings do not restrict the specific search and user-facing bots used by the platform.
