Researcher Nick Cammarata published an essay on goodfire.com proposing that LLM neuron polysemanticity is caused by correctable factors rather than superposition. The author suggests that most neurons in large models perform a single, identifiable task. This remains a working hypothesis, as a full technical report detailing the experiments has not yet been released.
The experiments conducted by Cammarata utilized the Qwen model, employing agents to audit neurons and verify hypotheses. The study involved an ablation test, specifically the 'weak75' method, which reduced average accuracy from 72% to 71%. Cammarata argues that the resulting changes in responses are minor and occur primarily between classes where the network already struggles.
Independent testing on an MLP-512 model using the same 'weak75' protocol showed an increase in standard test accuracy from 97.5% to 98.3%. However, when tested on edge cases, such as sorting sequences with two adjacent swapped elements, accuracy dropped from 93.1% to 72.4%. This indicates a 48% loss in capability beyond random guessing.
The findings suggest that weak activations contain critical class boundaries. While Cammarata's method indicates network stability, it may fail to account for the importance of these boundaries in complex tasks. Further evaluation of these claims awaits the publication of the complete technical report from the author.
