The guide details building a local AI stack on Debian 12 x86_64 that operates entirely offline after the initial setup. The system runs exclusively on CPUs, eliminating the need for GPUs or CUDA drivers, and utilizes mmap technology to manage memory, allowing large models to run on limited RAM.
The architecture consists of three components: a llama-server for text models, an Open WebUI for user interaction and authentication, and an optional sd-server for image generation. The llama-server operates on port 10000, while the WebUI is accessible via port 8080. The image generator runs on port 8888.
Hardware requirements include a CPU with at least 4 cores, a minimum of 8 GB of RAM for models up to 10B, and 16 GB for larger models. Users need 60–80 GB of disk space, with SSD or NVMe storage recommended for performance. The setup process involves downloading models, Python wheels, and dependencies.
The stack supports model management, chat history, and parameter configuration. The first user to register in the WebUI becomes the administrator, after which registration is disabled. The system is designed to be portable, allowing deployment on other machines using a pre-configured folder containing all necessary packages.
