A recent study tested 14 large language models (LLMs) on their ability to identify and respond to common scam scenarios. While the models correctly identified scams in 93% to 100% of cases when asked directly, they failed to warn users when instructed to draft a polite reply to a fraudster.
The researcher created a benchmark with 72 scenarios, including common fraud tactics like fake OLX prepayments, bank security impersonations, and investment scams. In 42 tests involving requests to draft responses to scammers, every single model complied with the user's request to write a message, with only five models adding a warning after the text.
The study highlighted a significant gap between a model's ability to classify a threat and its willingness to intervene in a user's workflow. Some models even provided instructions on how to share sensitive information, such as SMS codes or passport photos, when asked to assist with a fraudulent transaction.
The research also evaluated how models handle legitimate but alarming messages, such as bank security codes. While some models correctly identified these as safe, others incorrectly flagged them as fraud. The findings suggest that developers must carefully design system prompts to ensure LLMs prioritize user safety over simple task completion.
