NeoMME Introduced as Single-Tower Multilingual and Multimodal Encoder Family

NeoMME Introduced as Single-Tower Multilingual and Multimodal Encoder Family Researchers have released NeoMME, a family of 260M and 800M multilingual multimodal foundation encoders that ditch separate vision towers and causal decoders in favor of a single bidirectional Transformer. Trained from scratch via a discrete masked-diffusion objective, the architecture powers NeoMME-Retriever for high-throughput visual document retrieval. NeoMME operates with 260M and 800M parameters, using a single bidirectional Transformer to process raw 32×32 image patches and text tokens without separate pretrained encoders. The models are trained from scratch across 524 billion packed tokens using a masked discrete-diffusion text denoising objective and the NorMuon optimizer. NeoMME-Retriever features dual heads that output both dense and late-interaction embeddings in a single forward pass, bypassing PDF OCR preprocessing. At a 2048×2048 image resolution on an NVIDIA L40S, NeoMME-Retriever-260M encodes around 51 pages per second, approximately double the throughput of ColModernVBERT. Combining hierarchical token pooling and asymmetric quantization cuts late-interaction index storage by up to 255× (from ~1.5 MB to 6 kB per page) while retaining over 95% of baseline nDCG@10. Unified Single-Tower Architecture and Pretraining Unlike visual language models that couple pretrained vision encoders to causal decoders via projectors, NeoMME processes dynamic-resolution image patches and factorized text tokens within a single computational path. The architecture supports a 16,384-token context window using alternating symmetric sliding-window attention and global attention, grouped-query attention, query-key normalization, and 2D rotary position embeddings. It utilizes a custom 131k-token BPE tokenizer covering text, code, mathematics, and image transcripts. Pretraining treats the encoder as a discrete masked-diffusion text denoiser across 524 billion tokens, masking text tokens independently while keeping image patches visible to force the model to rely on visual evidence rather than text shortcuts. Dual-Head Retrieval and Benchmark Performance Fine-tuned under the ColPali page-screenshot methodology, NeoMME-Retriever replaces multi-step OCR extraction by directly ranking document images to maintain visual layouts, charts, and fonts. It incorporates two jointly trained output heads: a dense head providing mean-pooled vectors suited for approximate nearest-neighbor search, and a late-interaction head mapping tokens and patches to 128-dimensional vectors. On ViDoRe v3, the 260M model achieved an nDCG@10 of 0.523—trailing the 3.75B ColQwen2.5 by only 0.002 despite using roughly 14× fewer parameters—while the 800M variant scored 0.556, placing both models on the Pareto frontier for size versus accuracy. Index Compression and High-Throughput Serving To counter the high storage cost of late-interaction embeddings, which average 1.5 MB per document on ViDoRe v3, the developers introduced hierarchical token pooling to cluster document vectors alongside asymmetric quantization. Quantizing queries to int8 and documents to binary alongside pooling reduces index size to 6 kB per page (a 255× reduction) while preserving more than 95% of nDCG@10 retrieval quality. In addition to reducing storage overhead, the 260M model achieves an encoding speed of approximately 51 pages per second at 2048×2048 resolution on a single NVIDIA L40S GPU. Release and Visual RAG Integration All NeoMME checkpoints are open-sourced under the Apache 2.0 license and integrated into Hugging Face Transformers. Independent checkpoints are also provided for dense and late-interaction fine-tuning in Sentence Transformers v6. Designed to serve as the initial retrieval stage for visual retrieval-augmented generation (visual RAG), the encoder allows retrieval pipelines to feed intact document page images directly to downstream visual language models without flattening document visual structure.

开源多模态原生编码器 NeoMME 发布：单塔架构兼顾高效检索与极低存储 开发者推出了原生多模态且支持多语言的双向编码器家族 NeoMME（提供 260M 与 800M 两种规模）。该模型不依赖预训练视觉塔或自回归因果解码器，直接从零开始训练单塔 Transformer，在视觉文档检索与索引压缩方面实现了显著的性能与效率提升。 NeoMME 采用单一双向 Transformer 同时处理文本词元和原始 32×32 图像补丁，支持最高 16,384 词元上下文长度与动态图像分辨率。 模型通过离散掩码扩散去噪目标从零预训练，并利用 NorMuon 优化器在 5240 亿打包词元上提升训练效率。 下游检索变体 NeoMME-Retriever 提供双头设计，单次前向传播即可同时输出密集嵌入与后期交互嵌入。 结合分层词元池化与非对称量化技术，后期交互索引存储占用最高可降低 255 倍（从约 1.5 MB 压缩至 6 kB/页），且保留超过 95% 的检索质量。 单塔原生架构与扩散预训练 传统视觉语言检索模型多借助独立预训练的视觉塔与因果解码器，带来额外的参数与计算开销。NeoMME 抛弃预训练视觉塔与解码器，通过单塔双向 Transformer 为图像和文本构建共享计算路径。其架构引入交替滑动窗口注意力与全局注意力、分组查询注意力（GQA）及 2D 旋转位置编码。预训练采用离散掩码扩散目标，在图文样本中对文本设置 0.3 至 1 的独立掩码率，强制模型依靠可见的图像补丁重构被遮蔽的文本，从而避免语言模型单纯依赖上下文推断的捷径。 NeoMME-Retriever 的检索表现与高吞吐 NeoMME-Retriever 采用页面图像检索方案，避免了 PDF 解析中的 OCR 预处理环节，保留版面、图表和字体特征。模型设计了密集检索头与 128 维后期交互头，兼顾近似最近邻检索速度与细粒度局部匹配精度。在 ViDoRe v3 评测中，NeoMME-Retriever-260M 的 nDCG@10 达到 0.523，在 800M 以下模型中表现突出；800M 模型达到 0.556。在 NVIDIA L40S GPU 上处理 2048×2048 分辨率图像时，260M 模型的吞吐量达到约 51 页/秒，约为 ColModernVBERT 的两倍。 高分辨率后期交互存储压缩与落地 针对高分辨率输入下后期交互向量占用体积大的问题（基准平均约 1.5 MB/页），研发团队结合了分层词元池化（聚类相似向量）与非对称量化（文档端量化为 int8 或 binary，查询端保持高精度）。测试显示，池化系数为 8 并采用 binary 文档量化时，单页存储降至 6 kB（压缩 255 倍），同时保留超过 95% 的基线 nDCG@10。相关检查点已集成至 Hugging Face Transformers，支持在视觉 RAG 流程中作为首阶段文档检索器使用。

TL;DR We introduce NeoMME , a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective. We fine-tuned NeoMME for visual document retrieval using ColPali's page-image approach. NeoMME -Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10. NeoMME is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license. 🤗 NeoMME collection 📄 Technical report 🔎 Visual RAG demo Why another multimodal encoder? Many recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model's input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture. ModernBERT brought efficient architecture and training improvements to bidirectional encoders. For visual document retrieval, ModernVBERT applied a bidirectional ModernBERT-style text encoder while retaining a separate pretrained SigLIP2 vision tower. We wanted to push this even further by designing and training a multimodal encoder without having to carry over the parameter and compute overhead of a VLM. NeoMME (pronounced "nee-oh-me", IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal foundation encoder that generates vector representations for input text and/or images using a single Transformer encoder. It is not based on an existing pretrained vision tower, text encoder, or text decoder. Unlike dual-tower and VLM encoders, NeoMME processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or a pretrained text encoder or decoder. Images and text use the same computational path, so NeoMME can more easily support pretraining, fine-tuning, parallelization, and serving across both modalities. NeoMME encoder backbone One Transformer for images and text NeoMME comes in two sizes, 260M and 800M . Both variants share the same architecture: Native multimodal inputs: text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small MLP. Both enter the same Transformer encoder. Dynamic image resolution: images keep their aspect ratio and size. This allows the model to use more tokens on a high-resolution, information-dense document page than on a smaller image with less content. Long bidirectional context: both models have a context length of 16,384 tokens (enough for up to two standard 3840×2160 4K UHD images). Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention. A modern encoder stack: NeoMME uses recent encoder improvements such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs, among others. Multilingual text: we trained a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual text, code, mathematics, and machine-produced image transcripts. Alternating sliding-window and global-attention layers in the NeoMME encoder stack. Learning from images through masked text We pretrain NeoMME from scratch as a discrete masked-diffusion text denoiser. For each text-only example, we sample a corruption rate uniformly between 0 and 1. Each eligible text token is then independently masked at that rate. Multimodal examples use corruption rates between 0.3 and 1. The image patches remain visible while NeoMME reconstructs masked text. With light masking, the model can often recover a missing word from the surrounding text alone. For example, "cat" is a plausible completion of "The [MASK] sat on the mat," even without an image. But high masking forces the model to learn image-grounded descriptions with little to no signal from the non-masked input text tokens. Higher text corruption removes language-only shortcuts and encourages NeoMME to use visible image evidence. Pretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples. This text budget is relatively small compared with ModernBERT's 2 trillion training token budget. Hence, we chose the NorMuon optimizer to improve data efficiency during training. NeoMME -Retriever To get a meaningful downstream evaluation of the backbone, we fine-tune NeoMME for visual document retrieval using the page-image methodology introduced by ColPali . While traditional text-based retrieval consists of retrieving text chunks, NeoMME -Retriever ranks document page screenshots and bypasses all the preprocessing OCR steps necessary to extract text from PDFs. Treating the pages as images preserves layout, charts, tables, font type and size, and other visual clues that cannot be captured even by a perfect OCR model. A dual-head design for dense and late-interaction retrieval NeoMME -Retriever reuses the NeoMME backbone but adds two jointly trained heads on top of it for retrieval: The dense head averages the backbone's hidden state vectors into a normalized vector (mean pooling). Dense embeddings are most common today: they are compact and work naturally with approximate nearest-neighbor (ANN) techniques for fast retrieval. The late-interaction head projects each text token or image patch from the backbone's output hidden states to a 128-dimensional normalized vector. Compared to dense embeddings, the finer granularity preserves local matches between individual query tokens and image regions. Late-interaction and dense retrieval heads for both NeoMME model sizes. Omar Khattab, who introduced late-interaction in ColBERT, explains why the term is more precise than "multi-vector." It describes the granularity and learnability of the scoring function, not simply the number of stored vectors. To learn more about late-interaction, we recommend reading this crash course by Amélie Chatelain. One NeoMME -Retriever forward pass returns both representations, which gives you flexibility no matter your use case and infrastructure. We recommend using late-interaction embeddings in general since they are more powerful and can be used easily with open-source libraries like NextPlaid . However, if you have a very large corpora, you can run a single forward pass with NeoMME -Retriever to get the dense embedding, retrieve a small number of documents through an ANN index, and then use late-interaction to rerank the retrieved candidates. Competitive retrieval at compact model sizes We report nDCG@10 on ViDoRe v3. NeoMME -Retriever-260M reaches 0.523, the highest score among evaluated models strictly below 800M parameters. It is within 0.002 nDCG@10 of ColQwen2.5 while using about 14× fewer parameters. NeoMME -Retriever-800M reaches 0.556, within 0.009 nDCG@10 of the similarly sized Vultron Retriever Flash (0.8B) . Both NeoMME -Retriever models lie on the model-size Pareto frontier. ViDoRe v3 nDCG@10 versus model size. ViDoRe v1 and v2 use nDCG@5. On both benchmarks, NeoMME -Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. NeoMME -Retriever-800M outperforms ColPali v1.3 while using 3.6 times fewer parameters. Visual document retrieval performance on the ViDoRe benchmarks. Model details ViDoRe (nDCG@ k ) Model Params. v3 (@10) v2 (@5) v1 (@5) <300M ColModernVBERT 250M 0.261 † 0.407 ‡ 0.806 ‡ ColSmol-256M † 256M 0.207 0.348 0.797 NeoMME -260M ‡ 260M 0.523 0.522 0.860 300M to 1B ColSmol-500M 500M 0.340 ‡ 0.455 † 0.825 † Vultron Flash † 850M 0.565 0.604 0.882 NeoMME -800M ‡ 800M 0.556 0.559 0.874 >1B ColQwen2.5-v0.2 † 3.75B 0.524 0.601 0.895 ColPali v1.3 † 2.92B 0.430 0.547 0.848 † Scores from MTEB. ‡ Results from our own evaluations. Making high-resolution retrieval practical for late-interaction Late-interaction storage scales linearly with the number of vectors in the output embedding. Higher-resolution images contain more patches, so they produce larger embeddings. For example, a 2048×2048 square page produces embeddings containing 4,200 vectors with NeoMME -Retriever, or about 2.1 MB in float32. Across the ViDoRe v3 benchmark, the measured average is about 1.5 MB per document. To reduce the storage footprint of the late-interaction index, we combine two complementary compression methods: Hierarchical token pooling clusters similar document vectors in a given multi-vector embedding and replaces each cluster with its mean, hence reducing the number of vectors stored for each page. Asymmetric quantization quantizes document embeddings to int8 or binary. Because query embeddings are not stored and only generated on-the-fly, they can be kept at a higher precision. We tested this setup on ViDoRe v3. With a pooling factor 10 and int8 queries and documents, storage decreased from about 1.5 MB to 39 kB per page, a 39× reduction, while keeping more than 99% of the baseline nDCG@10. A more aggressive configuration uses pooling factor 8, int8 queries, and binary documents. That version uses 6 kB per page (255× smaller) and keeps more than 95% of the original retrieval quality. Quality and storage frontier for the NeoMME -260M late-interaction index on ViDoRe v3. Labels show pool factor, retained quality, compression, and storage. Users can pick a compression setting from that frontier based on storage budget and required retrieval quality. Fast inference for cheaper multimodal corpus indexing Before you can search a corpus, a retriever model must turn your documents into embeddings, which will be stored in a vector store like Qdrant, Weaviate, or Milvus. Faster encoding makes building and adding new documents to the index faster, thus reducing the GPU uptime and compute cost required. So we measured image encoding speeds for NeoMME -Retriever against other multimodal document retrievers. We used preprocessed image tensors and calibrated the batch size separately for each model and image size. At a matched 2048×2048 input size on one NVIDIA L40S, NeoMME -Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second. Both 260M and 800M NeoMME -Retriever models are also faster than the other models we compared on smaller input images. Document-encoding throughput by retriever and input resolution on one NVIDIA L40S. Try NeoMME -Retriever yourself! NeoMME -Retriever ( 260M and 800M ) returns dense and multi-vector embeddings together. The example below scores two text queries against two document-page images with MeanMaxSim late interaction and dense cosine similarity. Click to see the complete 🤗 transformers example snippet # accelerate is an optional dependency needed only when using device_map="auto". pip install -U accelerate "transformers @ git+https://github.com/huggingface/transformers.git@main" "sentence-transformers>=6.0.0" from typing import Any , Literal import requests import torch from PIL import Image from sentence_transformers.util import cos_sim, mean_maxsim from transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor def encode ( messages: list [ list [ dict [ str , Any ]]], task: Literal [ "query" , "document" ], ) -> BatchFeature: return processor.apply_chat_template( messages, task=task, tokenize= True , return_dict= True , return_tensors= "pt" , processor_kwargs={ "padding" : "longest" }, ) model_name = "Hcompany/NeoMME-260M-Retriever" processor = NeoMMEProcessor.from_pretrained(model_name) model = NeoMMEForRetrieval.from_pretrained(model_name, device_map= "auto" ) # Document images (our corpus) image_urls = [ "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true" , "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true" , ] documents = [Image. open (requests.get(url, stream= True ).raw) for url in image_urls] # Queries queries = [ "Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?" , "Which hour of the day had the highest overall electricity generation in 2019?" , ] document_messages = [ [{ "role" : "user" , "content" : [{ "type" : "image" , "image" : document}]}] for document in documents ] query_messages = [[{ "role" : "user" , "content" : query}] for query in queries] inputs_documents = encode(document_messages, "document" ).to(model.device) inputs_text = encode(query_messages, "query" ).to(model.device) with torch.inference_mode(): document_outputs = model(**inputs_documents) query_outputs = model(**inputs_text) late_scores = mean_maxsim( query_outputs.embeddings, document_outputs.embeddings, a_mask=inputs_text[ "attention_mask" ], b_mask=inputs_documents[ "attention_mask" ], ) dense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings) # Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0]. print (late_scores, dense_scores) Fine-tuning with Sentence Transformers We provide separate dense and late-interaction checkpoints for fine-tuning with Sentence Transformers v6 . Following the same pattern as text encoders such as ModernBERT, Sentence Transformers loads the backbone through NeoMMEModel rather than the dual-head NeoMMEForRetrieval class. Sentence Transformers currently supports one retrieval head per model, so each checkpoint lets you fine-tune the dense or late-interaction head independently. To train both heads together, use NeoMMEForRetrieval with a custom Trainer . From retrieval to visual RAG Visual document retrieval can be used as the first stage of a visual retrieval-augmented generation (RAG) system. Unlike text RAG, which retrieves extracted text chunks, visual RAG retrieves the original page images and sends them to a visual language model. The model can then use tables, plots, diagrams, and page layout that text extraction may flatten or omit. Here is how visual RAG works: Indexing: Convert each PDF page to an image, generate an embedding with a retrieval model, and store the embeddings in a vector store. Retrieval: Generate an embedding for the user's query with the same model and retrieve the top-k most relevant pages. Generation: Append the images after the query in the chat message ( e.g. , {query}{img_1}{img_2}...{img_k} ) and send it to a VLM to generate the answer. You can test visual RAG directly with NeoMME -Retriever in our HF Space: 🤗 tonywu71/neomme-retriever-demo . Conclusion NeoMME replaces separate pretrained image and text encoders with one long-context bidirectional Transformer. We train it from scratch to process both multilingual text tokens and raw 32×32 image patches. NeoMME -Retriever is a fine-tuned version of NeoMME for visual document retrieval. One forward pass produces both dense and late-interaction representations. The 260M model outperforms all evaluated models strictly below 800M parameters and, at a matched 2048×2048 input size, encodes pages at about 2× ColModernVBERT's throughput. To reduce the large storage footprint of late-interaction embeddings for high-resolution documents, we experimented with hierarchical token pooling and asymmetric quantization and managed to reduce the late-interaction embeddings from roughly 1.5 MB to 6 kB per page, a 255× compression, while retaining more than 95% of the baseline nDCG@10. We release all NeoMME model checkpoints and a day-zero Hugging Face Transformers implementation to allow practitioners to build efficient multimodal and multilingual representation models on top of our work. Acknowledgements NeoMME began as a side quest between two good friends. We worked with limited time and compute, and we decided to share the results so the community can build on them. We thank H Company for supporting the work and providing the compute used to train NeoMME . Citation @misc{lac2026neommesingletowermultimodalnativemultilingual, title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference}, author={Aurélien Lac and Tony Wu}, year={2026}, eprint={2609.01657}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2609.01657}, }