Converting Hugging Face models to GGUF format is necessary when pre-quantized versions are unavailable or specific quantization levels like Q4_K_M, Q5_K_M, or Q8_0 are required. The process involves converting raw .safetensors weights into an intermediate F16 GGUF file, followed by quantization using the llama-quantize tool.
A bash script, convert_to_gguf.sh, automates the pipeline. It handles system dependency checks, builds llama.cpp, sets up a Python virtual environment, and manages model downloads via the Hugging Face CLI. The script ensures that only CPU-based versions of PyTorch are installed to avoid unnecessary GPU driver dependencies.
The workflow includes safety checks to prevent errors. It verifies that downloaded files are actual weights rather than Git LFS pointers and performs a size check on the final Q4 file to ensure it is not corrupted. The script also implements an automatic cleanup mechanism to remove large intermediate files upon completion or failure.
Not all model architectures are supported for conversion. Users should check the llama.cpp/conversion/ directory for specific architecture support. If a model is not supported, conversion may fail, requiring either updates to llama.cpp or direct use of the transformers library.
