·
AI & ML interests
None yet
Recent Activity
reacted to FredyRivera-dev's post with 👍 10 days ago I've written a technical blog post about how we create a multimodal model:
Kairos: a multimodal model built with LFM2.5-2.6B as the LLM, MoonViT-3D (the vision tower of Kimi-K2.6) as the vision encoder, and a custom projector.
The original plan was LLaVA's approach, two stages: first align the projector with the LLM frozen, and then train the projector + LLM together. The first stage worked in terms of loss (ablation with +3.7 nats in favor of the image), but in free generation the image shifted the logits without changing the argmax: the model received the image and ignored it. That's why we jumped directly to early fusion, with a reasoning dataset.
For that, we created Kairos-Multimodal-Reasoning: 116,357 examples with explicit reasoning traces, generated through distillation (60,041 from LLaVA-CC3M, 2,295 from WebSight, and 54,021 from Zebra-CoT), with GPT 5.6 Luna, Inkling, Qwen 3.6 27B, and Qwen 3.7 Plus as teachers.
The training, in two phases:
1. Projector through backbone with 80k image-text pairs (Kairos-Proj-80k).
2. Projector + LoRA (r=16) with 30k examples from the reasoning dataset (Kairos-Alig-30k).
Everything is open source:
- Full blog post with the process: https://aquiles-ai.vercel.app/blog/kairos-a-multimodal-model
- Implementation: https://github.com/Aquiles-ai/Kairos
To be honest: the checkpoints are not a competent model, they are experimental artifacts. But they validated the approach and precisely defined what the next iteration needs.
https://huggingface.co/collections/Aquiles-ai/kairos
https://huggingface.co/Aquiles-ai/MoonViT-3D
https://huggingface.co/LiquidAI/LFM2.5-2.6B reacted to tomaarsen's post with 👍 12 days ago 🚨 I've just published Sentence Transformers v6.0, introducing MultiVectorEncoder: ColBERT-style late interaction models are now a fourth model type, for training, inference, and interpretation, alongside the dense, sparse, and reranker models! Details:
Where a regular embedding model compresses a whole text into one vector, a multi-vector model keeps one vector per token and scores query against document with the MaxSim operator. That preserves token-level matching information that a single vector has to average away. It is also the state of the art for visual document retrieval, where a text query is matched against page images directly, charts and tables included, with no OCR step in between.
Any PyLate, Stanford ColBERT, or ColPali checkpoint loads straight into the same familiar API: model.encode_query(), model.encode_document(), and model.similarity() just work, whether the documents are texts or page images.
Does it help? LightOn trained LateOn (multi-vector) and DenseOn (dense) on the same data with the same 149M ModernBERT backbone, and the multi-vector model wins on 9 of the 13 NanoBEIR datasets: 0.6868 vs 0.6764 mean NDCG@10. The price is a bigger index, and the new HierarchicalTokenPooling module halves it at roughly no retrieval cost.
Antoine Chaffin, Raphaël Sourty, and I wrote a blog post walking through multi-vector models in practice: loading the various checkpoint formats, encoding and scoring, plugging them into a search stack, running them on page images, and keeping the index affordable. Check it out if you want to get started, or just point your Agent to the URL: https://huggingface.co/blog/multi-vector-encoder
pip install sentence-transformers==6.0.0
Release notes: https://github.com/huggingface/sentence-transformers/releases/tag/v6.0.0 Organizations
None yet
