Researchers evaluated nine open-source language models from the Apertus, DeepSeek, Gemma, Llama, Mistral, and Qwen families. The study compared AI-generated responses against real user messages from X, Bluesky, and Reddit. To measure authenticity, the team developed the Conversational Human Authenticity Test (CHAT), which assesses semantic similarity, style, vocabulary, and emotional tone.
Machine learning algorithms successfully identified AI-generated text with 70–80% accuracy, even after models underwent fine-tuning and calibration. While fine-tuning improved the imitation of formal characteristics like sentence length, it failed to replicate human emotional nuances. Sentiment and toxicity remained the most reliable indicators of artificial origin, regardless of the training strategy used.
Scaling parameters did not resolve the issue; the 70-billion-parameter Llama 3.1 model performed similarly to smaller alternatives. Interestingly, models without instruction-following training mimicked human speech more effectively than those specifically tuned for tasks. Performance also varied by platform, with X being the easiest to replicate and Reddit the most challenging.
A significant trade-off exists between naturalness and semantic accuracy. Methods that made AI responses harder for classifiers to detect simultaneously reduced their semantic alignment with human originals. Increasing generation parameters also lowered detection rates but further degraded the quality of the content. These findings are critical for social science research relying on AI simulations.
