AI NEWS
NeoMME: an efficient Multimodal-native and Multilingual Encoder
Hugging Face introduces NeoMME, a new family of efficient multimodal encoders (260M and 800M parameters) designed for visual document retrieval. Unlike standard Visual Language Models that use separate vision towers and causal decoders, NeoMME uses a single bidirectional Transformer to process both text and raw image patches simultaneously. The model is trained from scratch using a masked discrete-diffusion objective on multilingual data. A fine-tuned variant, NeoMME-Retriever, offers dual-head outputs for dense and late-interaction embeddings in one forward pass. It achieves state-of-the-art performance on the ViDoRe v3 benchmark while significantly reducing storage requirements through hierarchical token pooling and asymmetric quantization, shrinking index sizes from 1.5 MB to just 6 kB per page with minimal quality loss.
THE NEWS
What happened
Hugging Face introduces NeoMME, a new family of efficient multimodal encoders (260M and 800M parameters) designed for visual document retrieval. Unlike standard Visual Language Models that use separate vision towers and causal decoders, NeoMME uses a single bidirectional Transformer to process both text and raw image patches simultaneously. The model is trained from scratch using a masked discrete-diffusion objective on multilingual data. A fine-tuned variant, NeoMME-Retriever, offers dual-head outputs for dense and late-interaction embeddings in one forward pass. It achieves state-of-the-art performance on the ViDoRe v3 benchmark while significantly reducing storage requirements through hierarchical token pooling and asymmetric quantization, shrinking index sizes from 1.5 MB to just 6 kB per page with minimal quality loss.
CONTEXT
Why it matters
Hugging Face introduces NeoMME: a groundbreaking multimodal encoder that unifies text and image processing into a single bidirectional Transformer. No separate vision tower needed. The new NeoMME-Retriever model sets new benchmarks for visual document retrieval on ViDoRe v3, achieving state-of-the-art nDCG@10 scores while drastically reducing storage needs via advanced compression. Read more: https://huggingface.co/blog/Hcompany/neomme
AT A GLANCE
Key facts
- NeoMME is a single bidirectional Transformer that processes text tokens and raw image patches together without a separate pretrained vision tower.
- The model comes in two sizes: 260M and 800M parameters, both sharing the same architecture.
- NeoMME-Retriever achieves an nDCG@10 score of 0.523 with the 260M variant, outperforming models like ColModernVBERT while using fewer parameters.
- Advanced compression techniques reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page while retaining over 95% of baseline retrieval quality.
- The 260M model encodes images at approximately 51 pages per second on an NVIDIA L40S GPU, nearly twice the speed of ColModernVBERT.
- All model checkpoints are released under the Apache 2.0 license and available via Hugging Face Transformers.
- The model supports dynamic image resolution and a context length of 16,384 tokens.
SOURCE
Original source
This article is based on information published by Hugging Face.
KEEP READING
