Dermatologist Magnus Lynch tested the diagnostic accuracy of GPT-6 Astra and Claude Fable 5.1 against a 2015-era convolutional neural network (CNN) using 200 skin lesions. All cases were biopsy-confirmed from the MILK10k dataset, released in 2025. The test set included 100 high-risk lesions and 100 benign cases.
The 2015 Inception v3 CNN, fine-tuned on 68,011 images from the ISIC Archive, correctly identified 125 out of 200 diagnoses (62%). In comparison, the two LLMs correctly identified 84 and 85 cases (42%). When considering the top two diagnostic options, the CNN correctly identified 173 cases, while the LLMs identified 134 and 138.
The CNN demonstrated higher accuracy in identifying specific conditions, such as melanoma, where it correctly diagnosed 13 out of 30 cases compared to 8 and 11 for the LLMs. The study also noted significant performance gaps in identifying benign nevi and seborrheic keratosis, where the CNN outperformed both language models.
The CNN used in the study is lightweight enough to run locally in a mobile browser via WebGPU. The researcher noted that while LLMs are advanced, they showed a tendency to favor specific diagnoses regardless of the actual pathology, whereas the specialized CNN maintained higher diagnostic precision across the test set.
