Linkup Research has introduced SPARSEUP, an open-source sparse embedding model featuring 149 million parameters. The model is available on Hugging Face under the Apache 2.0 license and supports integration via Transformers or Sentence Transformers libraries.
The model was developed using the LateOn-unsupervised checkpoint as a base, incorporating the original ModernBERT MLM head. Training utilized contrastive learning with 7 hard negatives per query and was performed on a single H100 GPU without cross-encoder distillation.
In performance evaluations on the BEIR-13 benchmark, SPARSEUP achieved a score of 56.4, compared to 58.9 for LateOn and 57.9 for DenseOn. The model utilizes the Seismic inverted index, which enables recall rates exceeding 97% with a single-threaded latency of approximately 380 microseconds per query.
SPARSEUP produces an average of 47 non-zero terms per query and 190 per document. The developers implemented specific query and document prefixes, set evaluation lengths at 128 and 512 tokens respectively, and opted to maintain a sparse structure rather than increasing vector size.
