The experiment involved fine-tuning the lumi-mini model, based on Gemma 4 E2B, using a MacBook with an M5 Pro chip and 48 GB of memory. The author utilized mlx-lm version 0.32 and a 4-bit quantized version of the model. The initial baseline performance on a 23-question test set was 6 correct answers out of 23.
The first fine-tuning attempt used a LoRA rank of 8 across 16 layers with a learning rate of 5e-5 over 75 steps. This resulted in a score of 4 out of 23, indicating performance degradation. The model exhibited catastrophic forgetting, prioritizing the imitation of response styles over factual accuracy.
A second attempt involved lowering the learning rate to 1e-5, reducing the layers to 8, and implementing a replay strategy with 117 original model responses. This brought the score back to 6 out of 23, matching the baseline without improvement. The model successfully learned formatting but failed to acquire new knowledge.
Technical hurdles prevented the integration of the fine-tuned model into the Ollama-based Lumi launcher. Issues included a lack of support for LoRA adapters and errors during the merging of 4-bit weights. Consequently, the author decided to continue using the standard lumi-mini model.
