| --- |
| license: apache-2.0 |
| library_name: sentence-transformers |
| pipeline_tag: feature-extraction |
| base_model: Qwen/Qwen3-0.6B |
| language: |
| - multilingual |
| - en |
| tags: |
| - sentence-transformers |
| - feature-extraction |
| - sentence-similarity |
| - embeddings |
| - mteb |
| - retrieval |
| - bidirectional |
| --- |
| |
| # Minnow-Em1-0.6B |
|
|
| **Minnow-Em1-0.6B** is a compact (0.6B-parameter) multilingual text-embedding model from |
| **KiteFish AI**, adapted from `Qwen/Qwen3-0.6B` into a fully **bidirectional** encoder and |
| fine-tuned for general-purpose embeddings: retrieval, semantic textual similarity (STS), |
| classification, clustering, reranking, and bitext mining. |
|
|
| > **Version:** v1 — the first public release in the Minnow-Em line. |
|
|
| --- |
|
|
| ## ⚠️ Important: this model must be loaded with bidirectional attention |
|
|
| This model was trained with the causal attention mask **removed** (every token attends to every |
| other token). That change is applied at load time and is **not** baked into the saved weights, so |
| loading the model the ordinary way leaves it in causal mode and produces poor embeddings. Always |
| apply the patch below after loading. |
|
|
| ```python |
| import types, torch |
| from sentence_transformers import SentenceTransformer |
| from transformers import PreTrainedModel |
| |
| def load_minnow(name="KiteFishAI/Minnow-Em1-0.6B", device="cuda"): |
| model = SentenceTransformer( |
| name, |
| model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "sdpa"}, |
| device=device, |
| ) |
| # --- make the backbone bidirectional (must match training) --- |
| hf = None |
| first = model[0] |
| for attr in ("auto_model", "model"): |
| c = getattr(first, attr, None) |
| if isinstance(c, PreTrainedModel): |
| hf = c; break |
| if hf is None: |
| hf = next(m for m in first.modules() if isinstance(m, PreTrainedModel)) |
| for _, m in hf.named_modules(): |
| if hasattr(m, "is_causal"): |
| m.is_causal = False |
| base = getattr(hf, "model", hf) |
| if hasattr(base, "_update_causal_mask"): |
| def _no_mask(self, attn_mask, inp, *a, **kw): |
| if attn_mask is None: |
| return None |
| if attn_mask.dim() == 2: |
| dt = inp.dtype |
| return (1.0 - attn_mask[:, None, None, :].to(dt)) * torch.finfo(dt).min |
| return attn_mask |
| base._update_causal_mask = types.MethodType(_no_mask, base) |
| hf.config.is_decoder = False |
| |
| # sanity check: token-0 state must change when a later token changes |
| tok = first.tokenizer |
| with torch.no_grad(): |
| a = tok(["The quick brown fox"], return_tensors="pt").to(hf.device) |
| b = tok(["The quick brown cat"], return_tensors="pt").to(hf.device) |
| d = (hf(**a).last_hidden_state[0, 0] - hf(**b).last_hidden_state[0, 0]).abs().max() |
| assert d > 1e-4, "Model is still causal — patch did not take effect." |
| return model |
| ``` |
|
|
| --- |
|
|
| ## Usage |
|
|
| The model is **instruction-aware**. Prepend a task instruction to each query using the format: |
|
|
| ``` |
| Instruct: {task instruction}\nQuery: {text} |
| ``` |
|
|
| - **Retrieval / reranking (asymmetric):** instruct the **query only**; leave documents raw. |
| - **STS / classification / clustering / bitext (symmetric):** instruct **all** texts. |
|
|
| ```python |
| model = load_minnow() |
| |
| def with_instruction(instruction, texts): |
| return [f"Instruct: {instruction}\nQuery: {t}" for t in texts] |
| |
| # --- retrieval example --- |
| queries = with_instruction( |
| "Given a query, retrieve documents that answer the query", |
| ["What causes the northern lights?"], |
| ) |
| docs = ["Auroras are produced when charged particles from the sun excite atoms in the upper atmosphere."] |
| |
| q = model.encode(queries, normalize_embeddings=True) |
| d = model.encode(docs, normalize_embeddings=True) # documents: no instruction |
| print((q @ d.T)) |
| ``` |
|
|
| --- |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Base model | `Qwen/Qwen3-0.6B` | |
| | Parameters | ~0.6B | |
| | Attention | Bidirectional (causal mask removed) | |
| | Pooling | Mean pooling | |
| | Embedding dim | 1024 | |
| | Max sequence length | 512 | |
| | Instruction-aware | Yes (`Instruct: … \nQuery: …`) | |
| | Similarity | Cosine | |
|
|
| ## Training |
|
|
| Minnow-Em1 follows the now-standard multi-stage recipe for compact LLM-based embedders |
| (cf. KaLM-Embedding-V2, Qwen3-Embedding, Llama-Embed-Nemotron): |
|
|
| 1. **Stage 1 — weakly-supervised contrastive pre-training.** Large-scale query/passage pairs, |
| in-batch negatives only, to adapt the bidirectional backbone to representation learning. |
| 2. **Stage 2 — supervised contrastive fine-tuning.** Task-homogeneous batches with mined hard |
| negatives, InfoNCE (temperature 0.02) with **focal reweighting** (γ = 0.5) to emphasize hard |
| examples, false-negative masking, and symmetric/asymmetric instruction routing by task type. |
|
|
| Training data spans retrieval, STS, classification, clustering, reranking, pair classification, and |
| bitext-mining sources across multiple languages. |
|
|
| ## Evaluation |
|
|
| Evaluation on the MMTEB / MTEB task suite is being finalized with the official `mteb` harness; a |
| full results table will be added to this card in a subsequent revision. The model is optimized for |
| the multilingual MMTEB task mix. |
|
|
| > Numbers will only be published once produced by the official `mteb` package on the complete |
| > benchmark task set (not a partial or custom run). |
|
|
| ## Limitations and intended use |
|
|
| - **Bidirectional load required** (see above) — without the patch the model is effectively causal |
| and underperforms badly. |
| - **In-domain training data.** The training mix includes the train splits of several public |
| benchmark datasets (e.g. MS MARCO, HotpotQA, Natural Questions, NFCorpus, MIRACL). Scores on the |
| corresponding evaluation tasks should be read as **in-domain, not zero-shot**. |
| - **Language balance.** v1's fine-tuning mix is weighted toward English question-answering |
| retrieval; performance on some low-resource and cross-lingual tasks is correspondingly weaker. |
| Rebalancing is planned for a future version. |
| - Intended for embedding/retrieval research and applications; not a generative model. |
|
|
| ## Acknowledgements |
|
|
| Built on `Qwen/Qwen3-0.6B`. Methodology informed by KaLM-Embedding-V2, Qwen3-Embedding, and |
| Llama-Embed-Nemotron-8B. Evaluated with the MTEB / MMTEB benchmark suite. |
|
|
| ## License |
|
|
| Released under Apache-2.0, consistent with the `Qwen/Qwen3-0.6B` base model. Verify license |
| compatibility for your use case before redistribution. |
|
|
