Building a Self-Hosted RAG System with Local LLMs

The developer built a self-hosted RAG system using Python 3.12, FastAPI, PostgreSQL with pgvector, Celery, LangGraph, Ollama, and Streamlit. The architecture utilizes Qwen 2.5 14B and BGE-M3 models, with Langfuse for monitoring agent steps and token usage.

The system processes over 12,000 articles. To resolve semantic ambiguity, the team implemented a hybrid search combining vector search with BM25 via the Reciprocal Rank Fusion algorithm, which increased Recall@5 from 0.2 to 0.53.

LLM-based classification accuracy reached 85% after refining prompts to remove ambiguous categories. The system uses a LangGraph state machine with tools for SQL aggregation and entity-based search to perform analytics beyond simple retrieval.

Technical optimizations included setting OLLAMA_MAX_LOADED_MODELS to 2 to reduce latency, replacing LangChain with ollama.AsyncClient to eliminate streaming overhead, and adjusting OLLAMA_NUM_PARALLEL to 2 to improve processing speed by 1.5 times.

The project is currently in the Public MVP stage with source code available on GitHub. Future plans involve transitioning to personalized stock market analytics by linking companies to Moscow Exchange tickers.