Screaming Frog SEO Spider 25.0, released on November 3, 2026, introduced a JSON configuration format, updated MCP tools, and new issue priority settings. The crawler now features automatic rate limiting for 429 and 503 responses.
Google announced plans to support a general robots.txt signal for AI training based on IETF AIPREF, expected in early 2027 with the token __usage_ai-training. This signal will act as a default for Google-Extended if no specific provider settings are present.
As of October 21, 2026, Google integrated Search generative AI control into its documentation, allowing site owners to manage content inclusion in AI Overviews and AI Mode via Search Console. This setting does not affect site ranking or AI training.
Bing Webmaster Tools launched the Topics Portfolio beta feature on October 13, 2026. It provides thematic clustering of grounding queries and citation data, though the metrics are AI-generated estimates.
Google updated its generative AI content guidance on October 1, 2026, emphasizing manual verification of metadata, structured data, and image alt text. Mass generation without added value may violate scaled content abuse policies.
A September 2026 study analyzing 1,536 AI product recommendations found a domain overlap of 12.0% for ChatGPT and 14.8% for Gemini when comparing user interfaces to API results.
