README.md

---
license: cc-by-nc-4.0
base_model: cl-nagoya/ruri-v3-310m
base_model_relation: finetune
language:
- ja
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
- japanese
- retrieval
---

<p align="center">
  <img src="assets/sionic_ai.png" alt="Sionic AI" width="480"/>
</p>

# comsat-embed-ja-0.3b-preview

**comsat-embed-ja-0.3b-preview** is an encoder-based embedding model developed by **Sionic AI**, optimized for Japanese semantic retrieval tasks. Trained on **over 1.5M Japanese examples**, it encodes queries and documents into vectors so that the most relevant documents can be found by similarity. The model is designed to provide high-quality text representations for real-world information retrieval scenarios, including document search, question answering, knowledge base retrieval, and enterprise semantic search. At only **0.3B parameters**, it delivers robust performance across Japanese search environments where accurate semantic matching is essential.

## Highlights

- **Japanese-specialized** — trained on 1.5M+ Japanese examples; achieves **state-of-the-art average NDCG@10 (0.7785)** on the 11-task JMTEB(v2) retrieval benchmark among the compared models with **≤4B parameters**.
- **Compact & efficient** — 0.3B (310M) parameters, well suited to cost-efficient, low-latency deployment.
- **Long context** — handles inputs up to 8,192 tokens.
- **Asymmetric encoding** — queries and documents are encoded with their respective prefixes (`検索クエリ: ` / `検索文書: `).
- **Embeddings** — 768-dimensional, mean-pooled and L2-normalized, compared with cosine similarity.

## Usage

First install the Sentence Transformers library

```bash
pip install -U sentence-transformers
```

### Sentence Transformers Usage

> ⚠️ Encode queries with the **query** prompt and documents with the **document** prompt. (Both use their own prefix; skipping the prompt slightly degrades retrieval quality.)

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sionic-ai/comsat-embed-ja-0.3b-preview")

queries   = ["日本の首都はどこですか？"]
documents = ["日本の首都は東京です。"]

# Prefixes ("検索クエリ: " / "検索文書: ") are applied automatically by prompt_name
q_emb = model.encode(queries,   prompt_name="query",    normalize_embeddings=True)
d_emb = model.encode(documents, prompt_name="document", normalize_embeddings=True)

# Option: sentence-transformers 5.x helper API (equivalent)
# q_emb = model.encode_query(queries)
# d_emb = model.encode_document(documents)

scores = q_emb @ d_emb.T   # cosine similarity
print(scores)
```

### Transformers Usage

```python
import torch
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def mean_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
    mask = attention_mask.unsqueeze(-1).to(last_hidden_states.dtype)
    return (last_hidden_states * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)


# Prepend the prefixes manually when using plain Transformers
queries   = ["検索クエリ: 日本の首都はどこですか？"]
documents = ["検索文書: 日本の首都は東京です。"]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained("sionic-ai/comsat-embed-ja-0.3b-preview")
model = AutoModel.from_pretrained("sionic-ai/comsat-embed-ja-0.3b-preview")

batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=8192,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**batch_dict)

embeddings = mean_pool(outputs.last_hidden_state, batch_dict["attention_mask"])
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = embeddings[:1] @ embeddings[1:].T   # cosine similarity
print(scores.tolist())
```

### JMTEB Retrieval Benchmark
- [NLPJournalTitleAbsRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalTitleAbsRetrieval.V2): Japanese **academic paper retrieval** — retrieve the abstract from the paper title.
- [NLPJournalTitleIntroRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalTitleIntroRetrieval.V2): Japanese **academic paper retrieval** — retrieve the introduction from the title.
- [NLPJournalAbsIntroRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalAbsIntroRetrieval.V2): Japanese **academic paper retrieval** — retrieve the introduction from the abstract.
- [NLPJournalAbsArticleRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalAbsArticleRetrieval.V2): Japanese **academic paper retrieval** — retrieve the article body from the abstract.
- [MintakaRetrieval](https://huggingface.co/datasets/mteb/MintakaRetrieval): A **multilingual open-domain QA retrieval dataset** (Japanese subset).
- [JaGovFaqsRetrieval](https://huggingface.co/datasets/mteb/JaGovFaqsRetrieval): A **Japanese government FAQ retrieval dataset**.
- [JaqketRetrieval](https://huggingface.co/datasets/mteb/jaqket): A **Japanese open-domain quiz QA retrieval dataset**.
- [MultiLongDocRetrieval](https://huggingface.co/datasets/mteb/MultiLongDocRetrieval): A **long-document retrieval dataset** (Japanese subset).
- [JaCWIRRetrieval](https://huggingface.co/datasets/mteb/JaCWIRRetrieval): A **Japanese casual web information retrieval dataset**.
- [MIRACLRetrieval](https://huggingface.co/datasets/mteb/MIRACLRetrieval): A **Wikipedia-based retrieval dataset** (Japanese subset).
- [MrTidyRetrieval](https://huggingface.co/datasets/mteb/mrtidy): A **Wikipedia-based Japanese retrieval dataset**.

## Performance (JMTEB v2 Retrieval, NDCG@10)

Only models with **≤4B parameters** are shown. All scores are NDCG@10; for multilingual tasks the Japanese subset is used (Mintaka/MultiLongDoc/MIRACL=`ja`, MrTidy=`japanese`).

| Model | Avg | NLPJ-TitleAbs | NLPJ-TitleIntro | NLPJ-AbsIntro | NLPJ-AbsArticle | Mintaka | JaGovFaqs | Jaqket | MultiLongDoc | JaCWIR | MIRACL | MrTidy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **comsat-embed-ja-0.3b-preview** | **0.7785** | 0.9807 | 0.9772 | 0.9945 | 0.9951 | 0.3686 | 0.7902 | 0.7617 | 0.4689 | 0.8721 | 0.7143 | 0.6402 |
| Qwen/Qwen3-Embedding-4B | 0.7779 | 0.9753 | 0.9589 | 0.9881 | 0.9959 | 0.5201 | 0.7179 | 0.6136 | 0.5659 | 0.8560 | 0.7244 | 0.6406 |
| codefuse-ai/F2LLM-v2-4B | 0.7705 | 0.9853 | 0.9803 | 0.9937 | 0.9966 | 0.4767 | 0.8064 | 0.6528 | 0.4701 | 0.8166 | 0.6527 | 0.6442 |
| sbintuitions/sarashina-embedding-v2-1b | 0.7659 | 0.9804 | 0.9782 | 0.9954 | 0.9858 | 0.4365 | 0.7561 | 0.7371 | 0.4529 | 0.8552 | 0.6552 | 0.5916 |
| cl-nagoya/ruri-v3-130m | 0.7641 | 0.9807 | 0.9643 | 0.9894 | 0.9959 | 0.3283 | 0.7729 | 0.7514 | 0.4565 | 0.8349 | 0.7157 | 0.6149 |
| cl-nagoya/ruri-v3-310m | 0.7630 | 0.9785 | 0.9653 | 0.9908 | 0.9959 | 0.3353 | 0.7726 | 0.7342 | 0.4393 | 0.8405 | 0.7233 | 0.6168 |
| cl-nagoya/ruri-v3-70m | 0.7473 | 0.9705 | 0.9620 | 0.9862 | 0.9896 | 0.2974 | 0.7461 | 0.7093 | 0.4392 | 0.8201 | 0.7050 | 0.5947 |
| nvidia/llama-nemotron-embed-vl-1b-v2 | 0.7464 | 0.9765 | 0.9669 | 0.9898 | 0.9966 | 0.2949 | 0.7076 | 0.6495 | 0.4257 | 0.8605 | 0.7143 | 0.6277 |
| codefuse-ai/F2LLM-v2-1.7B | 0.7426 | 0.9790 | 0.9699 | 0.9932 | 0.9980 | 0.3584 | 0.7857 | 0.6012 | 0.4597 | 0.8181 | 0.6153 | 0.5897 |
| cl-nagoya/ruri-v3-30m | 0.7330 | 0.9748 | 0.9540 | 0.9910 | 0.9893 | 0.2836 | 0.7236 | 0.6530 | 0.4626 | 0.8193 | 0.6635 | 0.5481 |
| cl-nagoya/ruri-large-v2 | 0.7260 | 0.9750 | 0.8184 | 0.9145 | 0.9083 | 0.3377 | 0.7744 | 0.7336 | 0.3933 | 0.8021 | 0.7136 | 0.6152 |
| Qwen/Qwen3-VL-Embedding-2B | 0.7253 | 0.9644 | 0.9454 | 0.9833 | 0.9946 | 0.3026 | 0.6915 | 0.5605 | 0.4638 | 0.8510 | 0.6402 | 0.5815 |
| BAAI/bge-m3 | 0.7245 | 0.9592 | 0.9164 | 0.9710 | 0.9528 | 0.2145 | 0.7066 | 0.5122 | 0.5034 | 0.8509 | 0.7285 | 0.6545 |
| google/embeddinggemma-300m | 0.7230 | 0.9627 | 0.9231 | 0.9757 | 0.9866 | 0.2683 | 0.7209 | 0.6749 | 0.3852 | 0.8524 | 0.6542 | 0.5490 |
| Snowflake/snowflake-arctic-embed-l-v2.0 | 0.7111 | 0.9727 | 0.9444 | 0.9873 | 0.9643 | 0.2344 | 0.7203 | 0.4328 | 0.4648 | 0.8549 | 0.6608 | 0.5856 |
| sbintuitions/sarashina-embedding-v1-1b | 0.7078 | 0.9688 | 0.9661 | 0.9916 | 0.9920 | 0.4025 | 0.7223 | 0.6412 | 0.3420 | 0.8254 | 0.5124 | 0.4219 |
| codefuse-ai/F2LLM-v2-0.6B | 0.7078 | 0.9671 | 0.9590 | 0.9921 | 0.9966 | 0.2734 | 0.7609 | 0.4896 | 0.4185 | 0.8018 | 0.5723 | 0.5548 |
| cl-nagoya/ruri-base-v2 | 0.7043 | 0.9658 | 0.7821 | 0.8982 | 0.9045 | 0.2997 | 0.7532 | 0.6947 | 0.3675 | 0.8044 | 0.6840 | 0.5928 |

> Avg is the mean over the 11 JMTEB(v2) retrieval tasks (higher is better).
> Reproduction: evaluated with the MTEB/JMTEB retrieval pipeline (NDCG@10, full corpus); the query prompt (`検索クエリ: `) is applied to queries and the document prompt (`検索文書: `) to documents.

## License

- Model weights: **cc-by-nc-4.0** (non-commercial use).