An independent developer has created a custom vision-language model (VLM) with 628 million parameters, leveraging the Baguettotron language model and the InternViT-300M-448px-V2.5 visual encoder. The project aims to provide a transparent, reproducible recipe for building small-scale vision-language systems using open-source components.
The training process was conducted on a single H100 SXM GPU rented via RunPod, with total costs amounting to approximately $200. The architecture follows a standard ViT-MLP-LLM design, utilizing a pixel unshuffle operation to compress 1024 visual tokens into 256, which are then projected into the language model's embedding space via a two-layer MLP.
The training pipeline consisted of three stages. The first stage involved training only the 2.7 million parameters of the MLP projector on 595,000 image-caption pairs. In the second stage, the language model was unfrozen and trained on the Cauldron dataset. A third stage attempted to introduce reasoning capabilities using R1-Vision-Reasoning-Instructions, though this resulted in hallucinations.
The developer utilized Claude to assist with brainstorming, project configuration, and model evaluation. The final model is capable of generating descriptive text for images, such as identifying objects and their positions, though it struggles with complex reasoning tasks and detailed object identification in specific contexts.
All project code, model weights, and documentation are publicly available on GitHub and Hugging Face. The developer emphasizes that this experiment serves as a practical guide for those looking to build and study vision-language models with limited budgets and open-source resources.
