Companies often struggle with fragmented knowledge across wikis, file storage, email, 1C, and Bitrix24. To address this, developers are building local AI assistants using Retrieval-Augmented Generation (RAG) to ensure data privacy by keeping models and storage on internal hardware.
The architecture utilizes a data pipeline where documents are indexed in MongoDB and Qdrant. For dynamic data like 1C records or Bitrix24 deals, agents use Model Context Protocol (MCP) servers to fetch information in real-time. This approach avoids the issues of indexing rapidly changing transactional data.
The system employs a unified gateway with an OpenAI-compatible API to manage model routing and statistics. For the core engine, the developer selected vLLM with a Qwen MoE 35B model (3B active parameters), which balances response speed and performance for multiple concurrent users.
Key implementation strategies include generating semantic questions for each document to improve search accuracy and using incremental updates to avoid long re-indexing times. The system also provides real-time search status and source citations to build user trust.
Lessons learned emphasize that the quality of the data pipeline is more critical than the model itself. Future-proofing involves using REST APIs instead of direct database access and maintaining a dedicated test set of 50–100 real employee queries to validate system changes.
