The study analyzes post-training quantization of the RWKV-7 model family, ranging from 0.1B to 13.3B parameters. Two presets were tested: REDUCTION, providing 2.1x compression, and COMPRESSION, offering 3.1x compression with 4-bit projections.
Research revealed that the 2.9B model degrades more than the 1.5B model under 4-bit quantization due to a specific output projection matrix in the zero block. Retaining this matrix in bf16 format, which accounts for 0.5% of the file size, reduces KL divergence by 39%.
The author developed an automated bit-depth selection rule based on the 'damage removed per byte' metric. This approach, combined with GPTQ, results in perplexity losses between 0.70% and 2.76% for files 31.6% to 35.2% the size of the original bf16 models.
Performance comparisons show that the 6-bit files with GPTQ achieve 2.0 to 3.3 times lower KL divergence than GGUF Q6_K with imatrix, and 3.5 to 7.4 times lower than MLX int6. The 4-bit files are 2.1 times more accurate than Q4_K_M on the 1.5B model.
