The developer has upgraded the 'Alpha' voice assistant project by integrating a neural processing unit (NPU) to enable local large language model (LLM) execution. The project aims to create a private, offline voice interface for smart home control, independent of external cloud services.
The hardware platform is based on the OrangePi 5 Ultra single-board computer, which features a Rockchip RK3588 SoC. This chip includes an NPU capable of 6 TOPS performance and supports INT4, INT8, INT16, and FP16 hybrid computing. The system uses a 256 GB eMMC module for storage.
The software environment runs on Debian 13. To facilitate the integration, the developer utilized the official Rockchip RKLLM software stack. The setup process involves installing the librkllmrt.so runtime library and compiling the llm_demo application, which was modified to include performance metrics.
The developer provided an automated installation script to handle dependency management, system configuration, and compilation of the demonstration program. The system requires an NPU driver version of at least 0.9.8 to function correctly with the RKLLM runtime.
For testing, the developer used the Qwen3 model with 1.7 billion parameters, which requires conversion and quantization (typically W8A8) to run on the RK3588 chip. Pre-quantized models can be obtained from repositories like Hugging Face.
