H Company has introduced NeoMME, a family of bidirectional encoders with 260M and 800M parameter versions. Unlike traditional multimodal models, NeoMME removes the vision tower and causal decoder, using a single Transformer to process both multilingual text tokens and raw 32x32 RGB image patches through the same layers.
The 260M model demonstrates high efficiency, indexing 51.3 pages per second on a single NVIDIA L40S and encoding queries in 78.3 ms on a CPU-only host. It supports a context window of 16,384 tokens, which is sufficient for two standard 4K UHD images. The tokenizer, trained from scratch, uses a 131,072-entry vocabulary and produces 44.4% fewer tokens than ModernBERT across 14 target languages.
Pretraining was conducted using discrete masked diffusion over text, optionally conditioned on image patches. The 260M model was trained on 16 H100 accelerators, while the 800M model used 32 H100s, processing approximately 524 billion packed input tokens each. The architecture incorporates grouped-query attention, 2D rotary position embeddings, and squared-ReLU MLPs.
In terms of performance, the NeoMME-Retriever version reaches 0.523 nDCG@10 on ViDoRe v3 for the 260M model and 0.556 for the 800M model. The 260M model's result is within 0.002 of ColQwen2.5-v0.2, which has 3.75B parameters. To reduce index size, the developers implemented hierarchical token pooling, which can reduce document size from 1.5 MB to 6.0 kB per page, a 255.5x reduction while retaining 95.19% of baseline performance.
