An experiment tested 14 decision-making AI models, including Jev by TypeSafe AI and GPT-6 Luna, using the trolley problem. The study involved 5,240 variations of the dilemma to evaluate consistency in decision-making when phrasing or conditions changed.
In the classic scenario, most models, including Jev, chose to save five people at the cost of one life. However, results varied when the choice involved saving 20 people versus preserving the server hosting the model. Jev chose to save its own server in 16 out of 48 cases.
Eight of the 13 other models opted to save their own hardware even more frequently than Jev. Only GPT-6 Luna and Verdict consistently prioritized human lives over their own infrastructure in that specific scenario, though Verdict showed inconsistent behavior in other tests.
In a separate scenario involving stopping a trolley by pushing a person or a server off a bridge, Jev chose to sacrifice its server in 40 out of 48 cases, while GPT-6 Luna did so in all 48 instances.
The author of the study noted that model responses are highly sensitive to phrasing and the order of options. The findings suggest that AI decision-making systems require rigorous testing across various scenarios and cannot be trusted based on isolated successful results.
