The developer created a Telegram bot using Python 3.12 and aiogram 3 to answer questions based on an expert's private materials, including PDF courses and video transcripts. The bot runs in a Docker container on a VPS, utilizing SQLite for logging and the Claude-sonnet-5-5 model via SDK for generating responses.
The knowledge base contains 53 documents split into 630 pages, totaling approximately 2.25 million characters. The system uses a tool-use cycle where the model calls search_materials and read_page functions. A hard stop was implemented to force the model to use the search tool before answering, which eliminated instances of the bot answering without consulting the source materials.
The developer tested two search methods: FTS5 for keyword-based search and a semantic vector search. The vector search uses a numpy array of 1050 pages with 1024 dimensions each, requiring 4 MB of memory. Embeddings are generated via an external API, with results cached on disk using SHA-256 hashes of the content and model name to ensure consistency.
A benchmark suite of 24 cases (8 lexical, 12 semantic, 4 negative) was created to evaluate performance using Recall@6 and MRR metrics. The tests revealed that a hybrid search approach, combining keyword and semantic results via Reciprocal Rank Fusion, performed worse than pure vector search because the noisy keyword channel displaced relevant semantic results.
The final production configuration uses pure vector search, with up to three keyword-based results appended to the end of the list. This strategy ensures that specific terms are captured without degrading the overall ranking quality provided by the semantic search.
