In 2023, a New York attorney submitted a court filing containing legal precedents generated by ChatGPT that did not exist. A year later, Air Canada lost a dispute after its chatbot promised a refund based on a policy the bot had invented itself.
Researchers are exploring methods to detect hallucinations in large language models by analyzing their internal states. Unlike human polygraphs that measure physiological stress, this approach involves monitoring the activity of individual neurons within the model.
A recent experiment tested this concept using a local language model on a consumer graphics card. When prompted to lie about the color of a banana, the model outputted 'Red' while internal data from the 25th layer of 28 showed a 46% probability for the correct answer, 'yellow'.
The findings suggest that language models may possess correct information internally before final layers alter the output. This research aims to identify the specific point in the processing chain where a model deviates from factual accuracy.
