What happened
OpenAI has introduced MultiLangBench, a new evaluation framework designed to test AI model proficiency in software development and modernization across 20 programming languages, including legacy systems like COBOL, Fortran, and Visual Basic. The company reported that its GPT-6 Astra model achieved an 87.4% average test pass rate on the , which requires models to build a SQLite-compatible database engine from scratch. This performance marks a significant increase over the 26.8% pass rate recorded by the previous GPT-5.6 Sol model.
OpenAI's MultiLangBench moves beyond isolated coding exercises by requiring models to build a functional, SQLite-compatible database engine. The evaluation assesses the model's ability to organize code, manage information, handle errors, and introduce new functionality while maintaining compatibility with reference implementations.
The company also conducted a separate assessment focused on migrating legacy COBOL programs into modern languages such as C++, Java, Kotlin, Python, and Rust. This test required byte-for-byte identical observable outputs to ensure that rounding rules and data handling remained consistent, a critical requirement for financial systems.
Performance results showed GPT-6.1 Sol achieving 86.6% accuracy at low reasoning effort for migration tasks, compared to 5.4% for the previous GPT-5.6 Sol. At higher reasoning effort, the model reached 97.9% accuracy.
OpenAI also introduced GPT-6 Luna as a lower-cost alternative, which achieved a 40.3% pass rate on the primary at an estimated API cost of $0.77 per sample, representing a 78% cost reduction compared to the previous generation's performance metrics.
Source details: edtechinnovationhub.com
Why it matters
The ability to modernize legacy codebases is a critical challenge for global infrastructure, as languages like COBOL underpin a vast majority of banking and ATM transactions. By focusing on languages with limited public documentation, OpenAI’s addresses the difficulty of migrating systems where original developers are unavailable. While these results demonstrate improved reasoning capabilities, OpenAI explicitly notes that these figures reflect performance on internal evaluations rather than proven success in production environments, emphasizing that expert human review remains essential for deploying migrated software.
Legacy systems remain the backbone of essential infrastructure, with COBOL supporting over 40% of online banking and 95% of ATM transactions. Modernizing these systems is notoriously difficult due to incomplete documentation and the loss of institutional knowledge.
The shift toward evaluating models on their ability to preserve existing business logic—rather than just generating new code—represents a practical step toward automating the maintenance of critical, aging software infrastructure.
OpenAI acknowledges that these benchmarks are internal and do not guarantee success in production. The requirement for expert review highlights the current limitations of AI in handling the nuanced dependencies and edge cases inherent in large-scale enterprise software.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Which component of an AI application is the machine-learning model itself?
What to watch next
OpenAI has released the MultiLangBench evaluation setup, test suite, and language-level results on GitHub for developer inspection. The company is currently soliciting feedback from enterprise users regarding specific languages and workflows that require further improvement. Observers should monitor how these models perform when applied to real-world, complex legacy systems where business rules are deeply embedded and undocumented, as well as whether the reported accuracy gains translate to reduced human oversight requirements in professional software engineering contexts.
Developers can now access the MultiLangBench suite on GitHub to verify the evaluation methodology and test the models against their own specific language requirements.
The industry will be watching to see if these performance gains hold up when applied to proprietary, non-public codebases that lack the clean structure of a test.
Future updates will likely focus on the integration of these models into enterprise workflows, specifically regarding how they handle the 'expert review' phase of code migration.