The Deckard team has released an open-source tool called agent-sitemap, designed to optimize how AI agents navigate websites. The generator converts website content into a structured format of Markdown notes linked together, allowing agents to access specific sections rather than processing entire pages.
The tool processes local HTML and Markdown files or crawls sites using sitemap.xml. It segments content by level-two headers and builds a link graph using TF-IDF and cosine similarity, with an optional LLM-based pass to refine connections and provide brief descriptions for each note.
Initial benchmarks using Claude Code showed that using the map reduced the number of requests from nine to six and decreased the volume of downloaded text by approximately 58%. Despite this, total input tokens increased by 3.5% due to the specific way the test environment accounts for cache reads and context management.
The generator is available under the MIT license. Users can deploy the generated catalog alongside their website and provide the index URL to an AI agent. The project includes a test stand for evaluating performance metrics, such as the number of requests and the accuracy of information retrieval.


