The study examined the existence of a negation direction in the embeddings of eight language models, including all-mpnet-base-v2, bge-base-en-v1.5, and rubert-tiny2. Researchers used the DiffMean method on 240 pairs of statements per language to identify a stable vector axis representing negation.
Results show that this negation axis is consistent across all 11 model-language pairs tested. While the axis overlaps with antonym replacement in English models, a stable component remains when controlling for sentiment. This component identifies negation in STS-B phrases with an AUC between 0.89 and 0.99.
The study found that subtracting the negation axis from embeddings worsens performance. However, incorporating it as a feature in cosine similarity calculations improves results, reducing the frequency of errors where negated phrases are incorrectly identified as paraphrases.
The proposed method shows limitations, as it penalizes correct answers that naturally contain negation. The research was limited to short template phrases with explicit particles like 'not' or 'не', excluding morphological negation and long documents.
