The developer tested 30 documentation websites for popular Python and JavaScript libraries to understand why AI coding assistants often overlook specific projects. The study examined potential technical barriers, including the presence of robots.txt files, the availability of llms.txt, and general crawler accessibility.
The findings revealed that none of the 30 domains blocked AI bots via robots.txt, neither at the documentation path nor the root level. While 11 projects implemented the llms.txt file, independent logs indicated that this file is rarely requested by AI systems.
The research also identified that only one project remained inaccessible to crawlers after verifying internal pages, despite an initial finding of three potentially problematic sites. The author provided the methodology, code, and a data table for further community verification.
