A study by UK fintech firm Saturn reveals that mainstream AI models failed to provide accurate personal finance advice in 57% of cases. When faced with complex, multi-step queries, the failure rate increased to 88%. The research involved testing 18 free and paid models, including ChatGPT, Gemini, Claude, and Copilot, by processing 121 questions with over 10,000 generated answers.
Free models demonstrated lower accuracy, failing 63% of the time, while paid versions failed 49% of the time. On the most challenging questions, free models reached a 93% failure rate. Claude Opus 5 in reasoning mode performed best among the tested tools but still recorded a 39% failure rate. Errors identified included incorrect calculations, ignored tax regulation updates, and the invention of non-existent financial rules.
The consequences of these errors can be significant. One specific response regarding pension taxes could have resulted in a £17,500 charge from HM Revenue and Customs. Saturn CEO Amal Jolly emphasized that AI financial advice remains unregulated, meaning consumers lack the compensation rights typically provided by human advisers.
Despite these risks, consumer reliance on AI for financial guidance is rising. An EY survey of 18,000 people showed that 49% use AI for savings and investment decisions. Additionally, 56% of less experienced investors in the UK trust AI tools more than traditional media, while 44% mistakenly believe that AI-generated financial information is subject to formal regulation.
