OpenAI has introduced MultiLangBench, a new benchmark designed to evaluate how AI models handle low-resource programming languages such as COBOL, Fortran, Visual Basic, and OCaml. The benchmark requires models to build a database engine compatible with SQLite 3.40.1 from scratch using only documentation.
In testing, the GPT-6 Astra model achieved an 87.4% success rate across 20 languages, compared to 26.8% for the previous GPT-5.6 Sol. While COBOL remains the most challenging language, the success rate for GPT-6 Astra reached 72.1%, up from 10.9% for its predecessor.
A separate test focused on migrating COBOL code to modern languages like C++, Java, and Rust. The GPT-6.1 Sol model successfully completed 86.6% of tasks, rising to 97.9% with extended processing time, while the older GPT-5.6 Sol managed only 5.4%.
OpenAI notes that while models have improved at writing code, verifying business logic remains a critical human task. Differences in arithmetic handling between COBOL and modern languages can lead to calculation errors, necessitating expert oversight during system migrations.
