The study evaluated seven cloud and local AI models on 20 coding tasks across three proprietary Delphi projects. The tasks focused on logic, calculations, VCL forms, and data visualization. Researchers used RAD Studio 37.0 to verify code compilation and DUnitX for unit testing.
Models were tested in two modes: without instructions and with a custom 'delphi-development' skill set. This skill set enforced four specific rules, including using .dfm files for UI components, adopting TImageCollection with TVirtualImageList, and ensuring proper memory management with try-finally blocks.
DeepSeek V4.1 Flash demonstrated the best cost-to-accuracy ratio, successfully completing 97% of tasks with a skill set at an average cost of $0.042 per task. Claude Haiku 5.5 followed with an 83–88% success rate at $0.061 per task.
Claude Sonnet 5.5 and GPT-6.1 Terra offered faster performance, completing tasks in approximately 3 minutes compared to the 6-11 minutes required by other models. However, these models were significantly more expensive, costing between $0.17 and $0.55 per task.
The evaluation highlighted that strict compiler warning limits can lead to failures even when functional tests pass. For instance, models that introduced redundant code triggered extra compiler warnings, resulting in a zero score despite correct logic in the DUnitX tests.
