README.md

---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3-Embedding-8B
base_model_relation: finetune
language:
- ja
- en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
- japanese
- retrieval
---

<p align="center">
  <img src="assets/sionic_ai.png" alt="Sionic AI" width="480"/>
</p>

# comsat-embed-ja-8b-preview

**comsat-embed-ja-8b-preview** is a decoder-based embedding model developed by **Sionic AI**, optimized for Japanese semantic retrieval tasks. Trained on **over 1.5M Japanese examples**, it encodes queries and documents into vectors so that the most relevant documents can be found by similarity. The model is designed to provide high-quality text representations for real-world information retrieval scenarios, including document search, question answering, knowledge base retrieval, and enterprise semantic search. By leveraging Japanese retrieval-oriented training data, comsat-embed-ja-8b-preview delivers robust performance across Japanese search environments where accurate semantic matching is essential.

## Highlights

- **Japanese-specialized** — trained on **1.5M+ Japanese examples** and tuned for Japanese search; achieves **state-of-the-art average NDCG@10 (0.8133)** on the 11-task JMTEB(v2) retrieval benchmark among the compared models.
- **Long context** — handles inputs up to 8,192 tokens, well suited to long-document retrieval.
- **Instruction-aware queries** — queries are encoded with a task-instruction prompt to improve retrieval quality; documents need no prefix.
- **High-dimensional embeddings** — 4096-dimensional, last-token pooled and L2-normalized, compared with cosine similarity.

## Usage

First install the Sentence Transformers library

```bash
pip install -U sentence-transformers
```

### Sentence Transformers Usage

> ⚠️ Queries **must** be encoded with the query prompt; documents are encoded **without** any prefix. (Skipping the query prompt slightly degrades retrieval quality.)

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sionic-ai/comsat-embed-ja-8b-preview")

queries  = ["日本の首都はどこですか？"]
passages = ["日本の首都は東京都です。"]

# Option 1) pass the query prompt explicitly (query only; documents get no prefix)
q_emb = model.encode(queries,  prompt_name="query", normalize_embeddings=True)
d_emb = model.encode(passages,                      normalize_embeddings=True)

# Option 2) sentence-transformers 5.x helper API (equivalent result)
# q_emb = model.encode_query(queries)
# d_emb = model.encode_document(passages)

scores = q_emb @ d_emb.T   # cosine similarity
print(scores)
```

### Transformers Usage

```python
# Requires transformers>=4.51.0

import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def last_token_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        sequence_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]


def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery:{query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'

queries = [
    get_detailed_instruct(task, '日本の首都はどこですか？'),
    get_detailed_instruct(task, '光合成はどのように起こりますか？')
]
# No need to add instruction for retrieval documents
documents = [
    "日本の首都は東京都です。",
    "光合成は、植物が光エネルギーを利用して二酸化炭素と水からブドウ糖を合成する過程です。"
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained('sionic-ai/comsat-embed-ja-8b-preview', padding_side='left')
model = AutoModel.from_pretrained('sionic-ai/comsat-embed-ja-8b-preview')

# We recommend enabling flash_attention_2 for better acceleration and memory saving.
# model = AutoModel.from_pretrained('sionic-ai/comsat-embed-ja-8b-preview', attn_implementation="flash_attention_2", torch_dtype=torch.bfloat16).cuda()

max_length = 8192

# Tokenize the input texts
batch_dict = tokenizer(
    input_texts,
    padding=True,
    truncation=True,
    max_length=max_length,
    return_tensors="pt",
)
batch_dict.to(model.device)
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T)
print(scores.tolist())
```

### Japanese Retrieval Benchmark (JMTEB v2)
- [NLPJournalTitleAbsRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalTitleAbsRetrieval.V2): Japanese **academic paper retrieval** — retrieve the abstract from the paper title.
- [NLPJournalTitleIntroRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalTitleIntroRetrieval.V2): Japanese **academic paper retrieval** — retrieve the introduction from the title.
- [NLPJournalAbsIntroRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalAbsIntroRetrieval.V2): Japanese **academic paper retrieval** — retrieve the introduction from the abstract.
- [NLPJournalAbsArticleRetrieval.V2](https://huggingface.co/datasets/mteb/NLPJournalAbsArticleRetrieval.V2): Japanese **academic paper retrieval** — retrieve the article body from the abstract.
- [MintakaRetrieval](https://huggingface.co/datasets/mteb/MintakaRetrieval): A **multilingual open-domain QA retrieval dataset** (Japanese subset).
- [JaGovFaqsRetrieval](https://huggingface.co/datasets/mteb/JaGovFaqsRetrieval): A **Japanese government FAQ retrieval dataset**.
- [JaqketRetrieval](https://huggingface.co/datasets/mteb/jaqket): A **Japanese open-domain quiz QA retrieval dataset**.
- [MultiLongDocRetrieval](https://huggingface.co/datasets/mteb/MultiLongDocRetrieval): A **long-document retrieval dataset** (Japanese subset).
- [JaCWIRRetrieval](https://huggingface.co/datasets/mteb/JaCWIRRetrieval): A **Japanese casual web information retrieval dataset**.
- [MIRACLRetrieval](https://huggingface.co/datasets/mteb/MIRACLRetrieval): A **Wikipedia-based retrieval dataset** (Japanese subset).
- [MrTidyRetrieval](https://huggingface.co/datasets/mteb/mrtidy): A **Wikipedia-based Japanese retrieval dataset**.

## Performance (JMTEB v2 Retrieval, NDCG@10)

All scores are NDCG@10, measured with the standard MTEB/JMTEB retrieval pipeline. For multilingual tasks the Japanese subset is used (Mintaka/MultiLongDoc/MIRACL=`ja`, MrTidy=`japanese`).

| Model | Avg | NLPJ-TitleAbs | NLPJ-TitleIntro | NLPJ-AbsIntro | NLPJ-AbsArticle | Mintaka | JaGovFaqs | Jaqket | MultiLongDoc | JaCWIR | MIRACL | MrTidy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **comsat-embed-ja-8b-preview** | **0.8133** | 0.9779 | 0.9781 | 0.9922 | 0.9973 | 0.6087 | 0.7634 | 0.7809 | 0.5655 | 0.8948 | 0.7479 | 0.6400 |
| codefuse-ai/F2LLM-v2-14B | 0.7965 | 0.9782 | 0.9823 | 0.9938 | 0.9966 | 0.5894 | 0.8244 | 0.7471 | 0.4854 | 0.8142 | 0.6941 | 0.6565 |
| Qwen/Qwen3-Embedding-8B | 0.7924 | 0.9649 | 0.9517 | 0.9900 | 0.9973 | 0.6023 | 0.7313 | 0.6642 | 0.5649 | 0.8590 | 0.7388 | 0.6524 |
| codefuse-ai/F2LLM-v2-8B | 0.7855 | 0.9808 | 0.9882 | 0.9932 | 0.9966 | 0.5438 | 0.8149 | 0.7050 | 0.4824 | 0.8121 | 0.6745 | 0.6485 |
| Qwen/Qwen3-Embedding-4B | 0.7779 | 0.9753 | 0.9589 | 0.9881 | 0.9959 | 0.5201 | 0.7179 | 0.6136 | 0.5659 | 0.8560 | 0.7244 | 0.6406 |
| codefuse-ai/F2LLM-v2-4B | 0.7705 | 0.9853 | 0.9803 | 0.9937 | 0.9966 | 0.4767 | 0.8064 | 0.6528 | 0.4701 | 0.8166 | 0.6527 | 0.6442 |
| Qwen/Qwen3-VL-Embedding-8B | 0.7702 | 0.9764 | 0.9648 | 0.9896 | 0.9973 | 0.4995 | 0.7051 | 0.6830 | 0.4936 | 0.8491 | 0.6934 | 0.6201 |
| sbintuitions/sarashina-embedding-v2-1b | 0.7659 | 0.9804 | 0.9782 | 0.9954 | 0.9858 | 0.4365 | 0.7561 | 0.7371 | 0.4529 | 0.8552 | 0.6552 | 0.5916 |
| cl-nagoya/ruri-v3-130m | 0.7641 | 0.9807 | 0.9643 | 0.9894 | 0.9959 | 0.3283 | 0.7729 | 0.7514 | 0.4565 | 0.8349 | 0.7157 | 0.6149 |
| cl-nagoya/ruri-v3-310m | 0.7630 | 0.9785 | 0.9653 | 0.9908 | 0.9959 | 0.3353 | 0.7726 | 0.7342 | 0.4393 | 0.8405 | 0.7233 | 0.6168 |

> Avg is the mean over the 11 JMTEB(v2) retrieval tasks (higher is better).
> Reproduction: evaluated with the MTEB/JMTEB retrieval pipeline (NDCG@10, full corpus); the query prompt is applied to queries only (documents get no prefix).

## License

- Model weights: **cc-by-nc-4.0** (non-commercial use).