Anthropic utilizes two distinct tools: the Claude application and the Claude Code agent. Research by Profound, based on 24 135 responses to 1 724 queries between July 13 and July 23, 2026, highlights significant operational differences. Claude uses web search in over 93% of responses in this sample, whereas Claude Code uses it in only 13%.
Claude Code primarily accesses documentation, while Claude frequently visits robots.txt, sitemaps, and homepages. Claude Code uses the WebFetch tool to retrieve content, which typically processes pages through a smaller AI model before passing a summary to the main model. This process is intentionally lossy, though exceptions exist for a specific list of approximately 80 technical documentation sites.
Anthropic employs three distinct crawlers: ClaudeBot for training, Claude-User for user-initiated queries, and Claude-SearchBot for search indexing. All three respect robots.txt rules. Blocking ClaudeBot prevents site content from being used for model training, while blocking Claude-User or Claude-SearchBot can impact site visibility and citation accuracy in AI responses.
The llms.txt file format, intended for machine-readable site summaries, is currently rarely accessed by Anthropic bots. Ahrefs data from May 2026 shows that among 38 000 domains with a valid llms.txt, 97% received no requests for the file over a month. ClaudeBot accounted for 0.8% of requests to this file, and Claude-User accounted for 2.5%.
To monitor Anthropic bot activity, site owners should analyze access logs for user-agent strings. While Anthropic publishes a list of IP addresses in bots.json, identifying bots via user-agent signatures in server logs is considered more reliable. Official signatures for the Claude Code WebFetch tool remain unpublished by Anthropic.
