README.md

---
license: apache-2.0
language:
- en
library_name: sentence-transformers
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- text-embedding
pipeline_tag: feature-extraction
base_model: Qwen/Qwen3-0.6B
---

# Nano-Em1-0.6B-v2

Nano-Em1-0.6B-v2 is a 0.6B-parameter text embedding model built on a
bidirectional-attention variant of Qwen3-0.6B, developed by KiteFish AI. It
produces general-purpose embeddings for retrieval, classification, clustering,
semantic similarity, and pair classification, using task-specific instruction
prefixes.

## Model description

- **Base model:** Qwen3-0.6B
- **Attention:** bidirectional (the causal mask is replaced with full
  bidirectional attention at every layer, implemented directly in the model's
  code so it holds under any standard load — see [Architecture](#architecture))
- **Pooling:** mean pooling over token embeddings
- **Embedding dimension:** 1024
- **Max sequence length:** 512 tokens

## Architecture

Decoder-only language models use causal (left-to-right) attention by default,
which limits their quality as fixed-representation encoders. Nano-Em1 removes
the causal mask so every token attends to the full input in both directions.
This is implemented in the model's own code (`modeling_nano.py`, loaded via
`trust_remote_code=True`) rather than applied by the caller at inference time,
so it's preserved by any standard `AutoModel.from_pretrained` or
`SentenceTransformer` load.

## Usage

### Sentence-Transformers

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "KiteFishAI/Nano-Em1-0.6B-v2",
    trust_remote_code=True,
)

query_instruction = "Instruct: Given a query, retrieve documents that answer the query\nQuery: "

queries = [query_instruction + "How do I reset my password?"]
docs = ["To reset your password, go to Settings > Security and click 'Reset Password'."]

query_emb = model.encode(queries, normalize_embeddings=True)
doc_emb = model.encode(docs, normalize_embeddings=True)

similarity = query_emb @ doc_emb.T
print(similarity)
```

### Transformers

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_id = "KiteFishAI/Nano-Em1-0.6B-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()

def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs).last_hidden_state
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    pooled = (out * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    return torch.nn.functional.normalize(pooled, dim=-1)

embs = embed(["example sentence one", "example sentence two"])
```

### Instructions

Queries — and passages, for symmetric tasks — should be prefixed:

```
Instruct: {task instruction}
Query: {text}
```

| Task type | Example instruction |
|---|---|
| Retrieval / Reranking | `Given a query, retrieve documents that answer the query` |
| Semantic similarity / Pair classification | `Retrieve semantically similar text` |
| Clustering | `Identify the topic or theme of the given texts` |
| Classification | `Classify the given text` |

For asymmetric tasks (retrieval, reranking), only the query takes the
instruction prefix — passages/documents are encoded as-is.

## Limitations

- English-focused; not evaluated on non-English tasks.
- Requires `trust_remote_code=True` to load the custom bidirectional
  architecture.

## Citation

```bibtex
@misc{nano-em1-2026,
  title  = {Nano-Em1-0.6B-v2},
  author = {KiteFish AI},
  year   = {2026},
  url    = {https://huggingface.co/KiteFishAI/Nano-Em1-0.6B-v2}
}
```