The study explores methods for creating isotropic embeddings using a 29M parameter model. The author evaluates three models: cointegrated/rubert-tiny2, sergeyzh/rubert-tiny-turbo, and the DeepPavlov/rubert-base-cased-sentence teacher model.
The evaluation uses a test set of 10 pairs, including paraphrases, hard negatives, and standard negatives. The metric for comparison is the gap, defined as the difference between the mean cosine similarity of similar pairs and the mean cosine similarity of dissimilar pairs.
The analysis reveals that the turbo model outperforms the tiny2 model by a factor of two in semantic understanding, despite sharing the same architecture of 3 layers and 312 dimensions. The teacher model, with 12 layers and 768 dimensions, remains 2.5 times more effective than the turbo model.
The author identifies a limitation in the teacher model regarding hard negatives, where it frequently produces high similarity scores for semantically opposite phrases. This is attributed to the mean pooling strategy, which dilutes the signal of specific tokens like negation.
Future steps involve using contrastive pre-training on the ru-HNP dataset to improve the student model's performance. The goal is to adapt the student's semantic representations while replacing the standard mean pooling with an attention-based approach.
