Model formats define how tensors are stored, while quantization methods compress weights. Standard unquantized models typically use 16-bit weights in .safetensors or .bin formats. Safetensors, a PyTorch Foundation project, replaces the older pickle-based .bin files to mitigate security risks by excluding executable code.
GGUF, introduced on August 21, 2023, by Georgi Gerganov, serves as the native format for llama.cpp. It supports metadata, tokenizers, and chat templates within a single file. It utilizes quantization schemes like Q4_K_M and supports calibration via llama-imatrix. The format is compatible with tools like Ollama, LM Studio, and GPT4All.
GPTQ, published at ICLR 2023, uses second-order Hessian information for post-training quantization. AWQ, developed at MIT and awarded at MLSys 2024, protects salient weights by analyzing activation magnitudes. Both methods require small calibration datasets but avoid full retraining.
EXL2 is the native format for ExLlamaV2, designed for consumer GPUs, while its successor, EXL3, incorporates QTIP technology published at NeurIPS 2024. ExLlamaV3 adds features like KV-cache quantization and requires CUDA 12.4 or later. Apple Silicon users utilize MLX-LM, which stores quantized weights in safetensors.
