docs/model_specific_benchmarking_notes.md

# Model-Specific Benchmarking Notes

When measuring specific models, follow the notes in this document. These notes
override generic benchmarking defaults when they are more specific, because some
models require exact prompts or runtime choices for comparable retrieval scores.
Start from the model author's officially recommended attention implementation
when it is documented. If this document records a different verified runtime,
use the verified note and preserve the attention implementation in result
metadata so slow default-attention runs are not mistaken for intentional
baselines.

## NanoMIRACL/en Runtime Matrix

On 2026-05-09, these models were revalidated on NanoMIRACL/en with a separate
output root for each runtime:

- logs: `tmp/runtime_matrix_nanomiracl_en_20260509_1935/`
- results: `output/runtime_matrix_nanomiracl_en_20260509_1935/`

The runtime order was:

1. Transformers 4.x + Flash Attention 2 (`tf4-fa2`)
2. Transformers 5.x + SDPA (`tf5-sdpa`)
3. Transformers 4.x + SDPA (`tf4-sdpa`)
4. Transformers 4.x default attention (`tf4-default`)

Use the first successful runtime below unless a later note for a specific model
has been superseded by a newer validation.

| model | method | first successful runtime |
| --- | --- | --- |
| `BAAI/bge-m3` | dense | `tf5-sdpa` |
| `Qwen/Qwen3-Embedding-0.6B` | dense | `tf4-fa2` |
| `google/embeddinggemma-300m` | dense | `tf4-fa2` |
| `hotchpotch/bekko-embedding-pico-beta-unir-v7` | dense | `tf4-fa2` |
| `hotchpotch/bekko-embedding-small-beta-unir-v8` | dense | `tf4-fa2` |
| `intfloat/multilingual-e5-large` | dense | `tf5-sdpa` |
| `intfloat/multilingual-e5-small` | dense | `tf5-sdpa` |
| `jinaai/jina-embeddings-v5-text-nano` | dense | `tf4-fa2` |
| `jinaai/jina-embeddings-v5-text-small` | dense | `tf4-fa2` |
| `cl-nagoya/ruri-v3-30m` | dense | `tf4-fa2` |
| `cl-nagoya/ruri-v3-310m` | dense | `tf4-fa2` |
| `perplexity-ai/pplx-embed-v1-0.6b` | dense | `tf4-fa2` |
| `ibm-granite/granite-embedding-311m-multilingual-r2` | dense | `tf4-fa2` |
| `Snowflake/snowflake-arctic-embed-l-v2.0` | dense | `tf5-sdpa` |
| `Alibaba-NLP/gte-multilingual-base` | dense | `tf4-sdpa` |
| `codefuse-ai/F2LLM-v2-330M` | dense | `tf4-fa2` |
| `jinaai/jina-embeddings-v3` | dense | `tf4-default` |
| `Snowflake/snowflake-arctic-embed-m-v2.0` | dense | failed in all four requested runtimes |
| `HIT-TMG/KaLM-embedding-multilingual-mini-v1` | dense | `tf4-fa2` |
| `codefuse-ai/F2LLM-v2-160M` | dense | `tf4-fa2` |
| `Lajavaness/bilingual-embedding-base` | dense | `tf4-default` |
| `intfloat/multilingual-e5-base` | dense | `tf5-sdpa` |
| `ibm-granite/granite-embedding-278m-multilingual` | dense | `tf5-sdpa` |
| `codefuse-ai/F2LLM-v2-80M` | dense | `tf4-fa2` |
| `Lajavaness/bilingual-embedding-small` | dense | `tf4-default` |
| `ibm-granite/granite-embedding-107m-multilingual` | dense | `tf5-sdpa` |
| `sentence-transformers/static-similarity-mrl-multilingual-v1` | dense | `tf4-fa2` |
| `sentence-transformers/all-MiniLM-L6-v2` | dense | `tf5-sdpa` |
| `naver/splade-v3` | sparse | `tf5-sdpa` |
| `lightonai/ColBERT-Zero` | late-interaction | `tf5-sdpa` with `uv run --group pylate` |
| `hotchpotch/bekko-embedding-pico-beta-unir-v9-QAT-ftQAT` | dense | `tf4-fa2` |
| `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` | dense | `tf5-sdpa` |

## cl-nagoya/ruri-v3

Applies to:

- `cl-nagoya/ruri-v3-30m`
- `cl-nagoya/ruri-v3-310m`

Use the retrieval prompts documented by the model card:

- query prompt: `検索クエリ: `
- document/corpus prompt: `検索文書: `

Example:

```bash
uv run hakari-bench evaluate dense \
  --model cl-nagoya/ruri-v3-310m \
  --query-prompt '検索クエリ: ' \
  --document-prompt '検索文書: '
```

Runtime notes:

- Prefer Transformers 4.x with Flash Attention 2 for ruri-v3 unless a newer
  runtime has been revalidated for the exact model and task set.
- In this project, `cl-nagoya/ruri-v3-310m` on NanoJMTEB/NanoJaqket produced
  abnormally low scores with `transformers==5.7.0` despite correct prompts.
  Re-running with `transformers==4.57.6` and Flash Attention 2 restored the
  expected current-dataset score.
- Do not shorten the model max sequence length to work around memory pressure.
  Reduce batch size first.

Historical comparison note:

- Older NanoJMTEB/NanoJaqket results around `nDCG@10 = 0.9426` were measured on
  the earlier q50 NanoJaqket data. Current NanoJMTEB uses q200 data, where
  `cl-nagoya/ruri-v3-310m` reproduced around `nDCG@10 = 0.8975` with
  Transformers 4.x + Flash Attention 2. Do not compare q50 and q200 scores as
  the same benchmark state.

## intfloat E5 And Multilingual E5

Applies to non-instruct E5 retrieval models, including:

- `intfloat/multilingual-e5-small`
- `intfloat/multilingual-e5-base`
- `intfloat/multilingual-e5-large`
- other non-instruct `intfloat/e5-*` models unless their model card says
  otherwise

Use the standard E5 retrieval prefixes:

- query prompt: `query: `
- document/corpus prompt: `passage: `

Example:

```bash
uv run hakari-bench evaluate dense \
  --model intfloat/multilingual-e5-base \
  --query-prompt 'query: ' \
  --document-prompt 'passage: '
```

Notes:

- Do not treat E5 instruct models as covered by this section. Instruct variants
  may require task-specific query instructions from their own model cards.
- Keep the prompts explicit in result metadata when re-running or comparing
  with older results.
- For `intfloat/multilingual-e5-large`, prefer SDPA over Flash Attention 2 until
  the FA2 path is revalidated. With `transformers==5.3.0`, `torch==2.9.0`, and
  `flash-attn==2.8.3`, NanoMIRACL/en scored lower with FA2 than SDPA despite the
  same prompts and model:

| attention | base | int8 | binary | int8_rescore | binary_rescore |
| --- | ---: | ---: | ---: | ---: | ---: |
| FA2 2.8.3 | 0.735197 | 0.742057 | 0.617535 | 0.735197 | 0.728762 |
| SDPA | 0.747493 | 0.743811 | 0.670783 | 0.743362 | 0.744807 |

Use:

```bash
uv run hakari-bench evaluate dense \
  --model intfloat/multilingual-e5-large \
  --attn-implementation sdpa \
  --query-prompt 'query: ' \
  --document-prompt 'passage: '
```

## Qwen Qwen3 Embedding

Applies to:

- `Qwen/Qwen3-Embedding-0.6B`

Use the Sentence Transformers prompt configuration:

- query prompt name: `query`
- document prompt name: `document`

The model card describes query-side instruction use as recommended for
retrieval. The stored `query` prompt is:

```text
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:
```

Example:

```bash
uv run hakari-bench evaluate dense \
  --model Qwen/Qwen3-Embedding-0.6B \
  --query-prompt-name query \
  --document-prompt-name document
```

Truncation notes:

- The model supports user-defined embedding dimensions from 32 to 1024.
- Use `--embedding-variant truncate:768,512,256,128,64,32` when measuring
  dimensional trade-offs.

## Google EmbeddingGemma

Applies to:

- `google/embeddinggemma-300m`

Use the Sentence Transformers retrieval prompt names:

- query prompt name: `Retrieval-query`
- document prompt name: `Retrieval-document`

These map to the model card's retrieval prompt formats:

- query: `task: search result | query: `
- document: `title: none | text: `

Example:

```bash
uv run hakari-bench evaluate dense \
  --model google/embeddinggemma-300m \
  --query-prompt-name Retrieval-query \
  --document-prompt-name Retrieval-document
```

Truncation notes:

- The model has 768-dimensional base embeddings and documented Matryoshka
  dimensions of 512, 256, and 128.
- Use `--embedding-variant truncate:512,256,128` when measuring dimensional
  trade-offs.

## hotchpotch Bekko Embeddings

Applies to:

- `hotchpotch/bekko-embedding-pico-beta-unir-v7`
- `hotchpotch/bekko-embedding-small-beta-unir-v8`
- `hotchpotch/bekko-embedding-pico-beta-unir-v9-QAT-ftQAT`
- `hotchpotch/bekko-embedding-pico-beta-unir-v9-GOR`
- `hotchpotch/bekko-embedding-pico-beta-unir-v9-GOR-pt`

Use the stored Sentence Transformers retrieval prompt names:

- query prompt name: `query`
- document prompt name: `passage`

These map to:

- query: `query: `
- passage: `passage: `

The configs also define `document` and `corpus` aliases that map to
`passage: `, but use `passage` for benchmark runs so metadata follows the
model card recommendation. Do not manually add `query: ` or `passage: ` when
using prompt names; that would apply the prefix twice.

Example:

```bash
uv run hakari-bench evaluate dense \
  --model hotchpotch/bekko-embedding-small-beta-unir-v8 \
  --query-prompt-name query \
  --document-prompt-name passage \
  --embedding-variant truncate:256,128,64
```

Truncation notes:

- `hotchpotch/bekko-embedding-pico-beta-unir-v7` documents base dim 384 and
  Matryoshka dims `256,128,64`.
- `hotchpotch/bekko-embedding-small-beta-unir-v8` documents base dim 384 and
  Matryoshka dims `256,128,64`.
- `hotchpotch/bekko-embedding-pico-beta-unir-v9-QAT-ftQAT` documents supported
  `truncate_dim` values `384,256,128,64`.
- `hotchpotch/bekko-embedding-pico-beta-unir-v9-GOR` documents supported
  `truncate_dim` values `384,256,128,64`. Use `256,128,64` for compact
  comparisons matching the GOR-pt run unless explicitly measuring the
  standalone 384-dimensional truncation variant.
- `hotchpotch/bekko-embedding-pico-beta-unir-v9-GOR-pt` recommends
  `truncate_dim` values `256,128,64`.

## Jina Embeddings v5

Applies to:

- `jinaai/jina-embeddings-v5-text-nano`
- `jinaai/jina-embeddings-v5-text-small`

Use remote code, retrieval encode tasks, and the stored retrieval prompt names:

- `--trust-remote-code`
- query encode task: `retrieval`
- document encode task: `retrieval`
- query prompt name: `query`
- document prompt name: `document`

These prompt names map to:

- query: `Query: `
- document: `Document: `

Example:

```bash
uv run hakari-bench evaluate dense \
  --model jinaai/jina-embeddings-v5-text-small \
  --trust-remote-code \
  --query-encode-task retrieval \
  --document-encode-task retrieval \
  --query-prompt-name query \
  --document-prompt-name document
```

Without the explicit retrieval encode task, the custom module can reject
Sentence Transformers' default `query` task with:
`Invalid task: query. Must be one of ['retrieval', 'text-matching',
'clustering', 'classification']`.

Truncation notes:

- `jinaai/jina-embeddings-v5-text-nano` supports Matryoshka dimensions
  512, 256, 128, 64, and 32 in addition to its 768-dimensional base output.
- `jinaai/jina-embeddings-v5-text-small` supports Matryoshka dimensions
  768, 512, 256, 128, 64, and 32 in addition to its 1024-dimensional base
  output.
- Use the matching `--embedding-variant truncate:...` list when measuring
  dimensional trade-offs.

## Jina Embeddings v3

Applies to:

- `jinaai/jina-embeddings-v3`

The model card and Sentence Transformers config define retrieval-specific
tasks and prompt names:

- query encode task: `retrieval.query`
- document encode task: `retrieval.passage`
- query prompt name: `retrieval.query`
- document prompt name: `retrieval.passage`
- `--trust-remote-code`

The prompts map to:

- query: `Represent the query for retrieving evidence documents: `
- document: `Represent the document for retrieval: `

Example:

```bash
uv run hakari-bench evaluate dense \
  --model jinaai/jina-embeddings-v3 \
  --trust-remote-code \
  --query-encode-task retrieval.query \
  --document-encode-task retrieval.passage \
  --query-prompt-name retrieval.query \
  --document-prompt-name retrieval.passage
```

Truncation notes:

- The model card documents Matryoshka dimensions
  768, 512, 256, 128, 64, and 32 in addition to the 1024-dimensional base
  output.

Compatibility notes:

- On NanoMIRACL/en, this model succeeded with Transformers 4.x default
  attention after failing with `tf4-fa2`, `tf5-sdpa`, and `tf4-sdpa`.
- Do not pass an attention override for this model unless that runtime has been
  revalidated.

## Snowflake Arctic Embed v2

Applies to:

- `Snowflake/snowflake-arctic-embed-l-v2.0`
- `Snowflake/snowflake-arctic-embed-m-v2.0`

Use the stored query prompt name:

- query prompt name: `query`
- no document prompt

The query prompt maps to `query: `.

Example:

```bash
uv run hakari-bench evaluate dense \
  --model Snowflake/snowflake-arctic-embed-l-v2.0 \
  --query-prompt-name query
```

Truncation notes:

- The v2 model card documents 256-dimensional MRL.
- Use `--embedding-variant truncate:256` when measuring dimensional trade-offs.

Compatibility notes:

- `Snowflake/snowflake-arctic-embed-m-v2.0` requires `--trust-remote-code` in
  this environment.
- `Snowflake/snowflake-arctic-embed-l-v2.0` succeeded on NanoMIRACL/en with
  `tf5-sdpa` in the 2026-05-09 runtime matrix.
- `Snowflake/snowflake-arctic-embed-m-v2.0` failed in all four requested
  runtimes (`tf4-fa2`, `tf5-sdpa`, `tf4-sdpa`, `tf4-default`). The immediate
  failure was `please install xformers` after the remote code forced eager
  attention because `use_memory_efficient_attention=true`.
- A prior same-day check with `xformers` installed reached CUDA index
  assertions even with `--model-max-seq-length 8192`; do not treat the medium
  checkpoint as verified until a working runtime is revalidated.

## IBM Granite Embeddings

Applies to:

- `ibm-granite/granite-embedding-311m-multilingual-r2`
- `ibm-granite/granite-embedding-97m-multilingual-r2`
- `ibm-granite/granite-embedding-278m-multilingual`
- `ibm-granite/granite-embedding-107m-multilingual`

No non-empty retrieval prompts were found in the Sentence Transformers configs
used for these checkpoints. Preserve the model default behavior unless the model
card changes.

Truncation notes:

- `ibm-granite/granite-embedding-311m-multilingual-r2` documents Matryoshka
  dimensions of 512, 384, 256, and 128 in addition to its 768-dimensional base
  output.
- Use `--embedding-variant truncate:512,384,256,128` for that R2 checkpoint when
  measuring dimensional trade-offs.
- `ibm-granite/granite-embedding-97m-multilingual-r2` has a 384-dimensional
  base output and does not document Matryoshka dimensions. Do not add truncation
  variants unless a newer model card documents them.
- Do not assume Matryoshka support for the older 278M and 107M multilingual
  checkpoints.

## CodeFuse F2LLM v2

Applies to:

- `codefuse-ai/F2LLM-v2-80M`
- `codefuse-ai/F2LLM-v2-160M`
- `codefuse-ai/F2LLM-v2-330M`

Use the stored Sentence Transformers prompt names:

- query prompt name: `query`
- document prompt name: `document`

The query prompt maps to:

```text
Instruct: Given a question, retrieve passages that can help answer the question.
Query:
```

Example:

```bash
uv run hakari-bench evaluate dense \
  --model codefuse-ai/F2LLM-v2-330M \
  --query-prompt-name query \
  --document-prompt-name document
```

No Matryoshka/truncate dimensions were found in the model cards or Sentence
Transformers configs checked for these models.

## Perplexity pplx Embed

Applies to:

- `perplexity-ai/pplx-embed-v1-0.6b`

Use:

- `--trust-remote-code`

Example:

```bash
uv run hakari-bench evaluate dense \
  --model perplexity-ai/pplx-embed-v1-0.6b \
  --trust-remote-code
```

No model-specific prompt or Matryoshka/truncate dimensions were found in the
files checked for this model.

## Alibaba GTE Multilingual Base

Applies to:

- `Alibaba-NLP/gte-multilingual-base`

Use:

- `--trust-remote-code`

Compatibility notes:

- The Sentence Transformers config sets `max_seq_length` to 8192.
- On NanoMIRACL/en, this model succeeded with `tf4-sdpa` after failing with
  `tf4-fa2` and `tf5-sdpa`.

## Lajavaness Bilingual Embeddings

Applies to:

- `Lajavaness/bilingual-embedding-base`
- `Lajavaness/bilingual-embedding-small`

Use:

- `--trust-remote-code`

Compatibility notes:

- In this project environment with Transformers 5.x, loading failed because the
  custom code imports `transformers.onnx`, which is unavailable.
- On NanoMIRACL/en, both checkpoints succeeded with Transformers 4.x default
  attention after failing with `tf4-fa2`, `tf5-sdpa`, and `tf4-sdpa`.
- Do not pass an attention override for these models unless that runtime has
  been revalidated.

## naver SPLADE v3

Applies to:

- `naver/splade-v3`

Use the sparse evaluator:

```bash
uv run hakari-bench evaluate sparse \
  --model naver/splade-v3
```

No non-empty query/document prompts were found in the Sentence Transformers
SparseEncoder config.

## LightOn ColBERT Zero

Applies to:

- `lightonai/ColBERT-Zero`

Use the late-interaction evaluator and the ColBERT-specific settings stored in
the model config:

- query prompt name: `query`
- document prompt name: `document`
- query prefix: `[Q] `
- document prefix: `[D] `
- query length: `39`
- document length: `519`

Example:

```bash
uv run --group pylate hakari-bench evaluate late-interaction \
  --model lightonai/ColBERT-Zero \
  --query-prompt-name query \
  --document-prompt-name document \
  --late-interaction-query-prefix '[Q] ' \
  --late-interaction-document-prefix '[D] ' \
  --late-interaction-query-length 39 \
  --late-interaction-document-length 519
```

Compatibility notes:

- This repository keeps PyLate behind the `pylate` dependency group, so use
  `uv run --group pylate ...` for this model.
- PyLate currently requires Transformers 5.x in this project, so the
  Transformers 4.x runtime matrix entries were skipped for this model.
- The local evaluator aliases PyLate's renamed `_input_length` helper to
  `_text_length` before encoding. With that compatibility shim, NanoMIRACL/en
  succeeded with `tf5-sdpa`.

## Sentence Transformers Static Similarity MRL

Applies to:

- `sentence-transformers/static-similarity-mrl-multilingual-v1`

The model card says this model is not intended for retrieval use cases, even
though it can be evaluated as a dense Sentence Transformers model. Keep that
limitation visible when reporting retrieval scores.

Truncation notes:

- The model card documents Matryoshka support.
- Use `--embedding-variant truncate:512,256,128,64,32` when measuring
  dimensional trade-offs.