OpenAI has introduced MentalHealthBench, an open-source benchmark designed to evaluate how AI systems handle mental health-related conversations. The dataset consists of 1,215 synthetic dialogues covering scenarios ranging from routine discussions to emergency situations, developed with input from over 80 licensed psychologists and psychiatrists from 22 countries.
The benchmark includes 5,262 expert-defined scoring criteria across 19 languages. Conversations are categorized by risk level: 53.5% are non-acute, 18.2% are high-risk, and 28.3% are emergency cases. User profiles include adults (68.1%), adolescents (21.2%), clinicians (5.8%), and caregivers (4.9%).
Performance testing shows GPT-6 Astra leading with a task-clipped score of 57.3%, followed by GPT-6 Sol at 53.9% and Claude Opus 5.5 at 52.4%. Other models, including GPT-4o and Gemini 2.5 Pro, scored 32.1% and 29.5% respectively. The evaluation uses an automated scorer based on the GPT-5.6 Sol model.
OpenAI emphasizes that the tool is intended for diagnostic auditing rather than ranking. The company explicitly states that ChatGPT is not a substitute for professional medical care. To prevent data contamination, researchers are prohibited from publishing dataset samples in plain text or image formats online.
