Publish measured GöktuğTR release

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,5 @@
+{
+    "embedding_dimension": 640,
+    "pooling_mode": "lasttoken",
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,79 @@
+---
+language:
+- tr
+license: mit
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: GoktugD/goktugtr-retrieval-270m-v1
+datasets:
+- GoktugD/goktugtr-hard-negatives-50k
+tags:
+- sentence-transformers
+- semantic-search
+- information-retrieval
+- turkish
+- hard-negatives
+---
+
+# GöktuğTR Retrieval 270M v2
+
+A 268.1M-parameter Turkish dense retriever continued from GöktuğTR v1 on
+50,000 model-mined difficult negatives. The release is designed as a transparent
+hard-negative experiment: it publishes positive, neutral and negative evidence.
+
+## Five-task measured results
+
+| Model | Params | Dim | TurHist | XQuAD | WebFAQ | MKQA | Belebele | Macro |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| multilingual E5 base | 278.0M | 768 | **0.49726** | **0.95335** | **0.65032** | 0.07213 | **0.92503** | **0.619618** |
+| **GöktuğTR 270M v2** | **268.1M** | **640** | 0.42198 | 0.86393 | 0.56886 | **0.10331** | 0.88493 | **0.568602** |
+| GöktuğTR 270M v1 | 268.1M | 640 | 0.42196 | 0.85832 | 0.56402 | 0.10296 | 0.88222 | **0.565896** |
+| GöktuğTR 118M v1 | 117.7M | 384 | 0.25299 | 0.81123 | 0.46307 | 0.04855 | 0.82451 | **0.480070** |
+
+v2 improves v1 on all five tasks, with a macro change of +0.002706 points
+(about +0.48% relative). The hard-negative triplet validation score itself was
+unchanged at 0.8935. The appropriate claim is a small, consistent held-out
+gain—not a major jump. E5 remains the overall suite leader; GöktuğTR v2 exceeds
+it only on MKQA in this matrix.
+
+## Use
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("GoktugD/goktugtr-retrieval-270m-v2")
+task = "Given a Turkish web search query, retrieve relevant passages that answer the query"
+query = f"Instruct: {task}\nQuery: Hard negative neden önemlidir?"
+query_vector = model.encode(query, normalize_embeddings=True)
+document_vectors = model.encode(
+    ["Zor negatifler karar sınırını güçlendirir.", "Ankara Türkiye'nin başkentidir."],
+    normalize_embeddings=True,
+)
+print(document_vectors @ query_vector)
+```
+
+Queries require the instruction format shown above. Documents are plain text.
+
+## Mining and training
+
+- Base: `GoktugD/goktugtr-retrieval-270m-v1`
+- Mined data: 50,000 train / 2,000 validation triplets
+- Candidate pool: 70,172 source-labeled negatives; search depth: 32
+- Successful mined rows: 50,000; fallbacks: 0; mean cosine: 0.550376
+- Exact normalized TurHistQuad overlap: 0
+- Objective: Cached Multiple Negatives Ranking Loss
+- Sequence length: 256; effective batch: 64; learning rate: `8e-6`
+- One epoch, BF16, seed 3407, one local RTX 5060 Laptop GPU
+- Training time: 2,528 seconds
+
+## Limitations
+
+- The measured improvement is small and may not transfer to a target corpus.
+- A source-labeled negative can still be semantically relevant to a query.
+- The upstream corpus is machine translated.
+- Similarity is not a probability or a factuality score.
+- Five retrieval tasks do not cover every Turkish domain, dialect or intent.
+
+The repository ships raw per-task MTEB objects, checksums, training state,
+environment metadata, the mining audit and the exact evaluation code.
+

config.json

@@ -0,0 +1,65 @@
+{
+  "_sliding_window_pattern": 1,
+  "architectures": [
+    "Gemma3TextModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "head_dim": 256,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 640,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "model_type": "gemma3_text",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 18,
+  "num_key_value_heads": 1,
+  "pad_token_id": 0,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 1000000.0,
+      "rope_type": "default"
+    },
+    "rope_theta": null,
+    "rope_type": "default",
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sliding_window": 512,
+  "sliding_window_pattern": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_bidirectional_attention": false,
+  "use_cache": false,
+  "vocab_size": 262144
+}

config_sentence_transformers.json

@@ -0,0 +1,17 @@
+{
+  "__version__": {
+    "pytorch": "2.8.0+cu129",
+    "sentence_transformers": "5.7.0",
+    "transformers": "5.14.1"
+  },
+  "default_prompt_name": null,
+  "model_type": "SentenceTransformer",
+  "prompts": {
+    "bitext_query": "Instruct: Retrieve parallel sentences\nQuery: ",
+    "document": "",
+    "query": "",
+    "sts_query": "Instruct: Retrieve semantically similar text\nQuery: ",
+    "web_search_query": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
+  },
+  "similarity_fn_name": "cosine"
+}

examples/goktugtr_end_to_end.ipynb

@@ -0,0 +1,145 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# GöktuğTR: retrieval → reranking → cited answer\n",
+    "\n",
+    "This free Colab notebook loads public artifacts only. It performs Turkish semantic retrieval, optionally reranks the candidates, and produces an explicitly extractive answer with source identifiers. No API key or paid service is required."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip -q install sentence-transformers==5.7.0 faiss-cpu==1.15.0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "import numpy as np\n",
+    "import faiss\n",
+    "from sentence_transformers import SentenceTransformer, CrossEncoder\n",
+    "\n",
+    "RETRIEVER_ID = \"GoktugD/goktugtr-retrieval-270m-v2\"\n",
+    "RERANKER_ID = \"GoktugD/goktugtr-reranker-118m-v1\"\n",
+    "TASK = \"Given a Turkish web search query, retrieve relevant passages that answer the query\"\n",
+    "retriever = SentenceTransformer(RETRIEVER_ID)\n",
+    "print(RETRIEVER_ID, retriever.get_sentence_embedding_dimension())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Add your own Turkish documents\n",
+    "Each document keeps a stable source identifier so every answer fragment can be traced back."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "documents = [\n",
+    "    {\"source\": \"training.md\", \"text\": \"Gradient accumulation küçük batch gradyanlarını optimizer adımından önce biriktirerek daha büyük bir etkili batch sağlar.\"},\n",
+    "    {\"source\": \"evaluation.md\", \"text\": \"MRR ilk doğru belgenin sırasını, nDCG ise sıralamanın genel kalitesini ölçer.\"},\n",
+    "    {\"source\": \"deployment.md\", \"text\": \"ONNX modeli tarayıcıda çalıştırmak sorgunun ücretli bir sunucu API'sine gönderilmesini önler.\"},\n",
+    "    {\"source\": \"reranking.md\", \"text\": \"Cross-encoder reranker sorgu ve belgeyi birlikte okuyarak retriever adaylarını daha hassas biçimde yeniden sıralar.\"},\n",
+    "]\n",
+    "doc_vectors = retriever.encode(\n",
+    "    [item[\"text\"] for item in documents], normalize_embeddings=True, convert_to_numpy=True\n",
+    ").astype(\"float32\")\n",
+    "index = faiss.IndexFlatIP(doc_vectors.shape[1])\n",
+    "index.add(doc_vectors)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"Küçük GPU'da etkili batch nasıl büyütülür?\"\n",
+    "formatted_query = f\"Instruct: {TASK}\\nQuery: {query}\"\n",
+    "query_vector = retriever.encode([formatted_query], normalize_embeddings=True, convert_to_numpy=True).astype(\"float32\")\n",
+    "scores, ids = index.search(query_vector, min(4, len(documents)))\n",
+    "candidates = [{**documents[i], \"retrieval_score\": float(score)} for i, score in zip(ids[0], scores[0])]\n",
+    "candidates"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Rerank the retrieved candidates\n",
+    "The cross-encoder is slower but reads each query-document pair jointly. It is therefore applied only to the small candidate set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "reranker = CrossEncoder(RERANKER_ID)\n",
+    "reranker_scores = reranker.predict([[query, item[\"text\"]] for item in candidates])\n",
+    "for item, score in zip(candidates, reranker_scores):\n",
+    "    item[\"reranker_score\"] = float(score)\n",
+    "candidates.sort(key=lambda item: item[\"reranker_score\"], reverse=True)\n",
+    "candidates"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Build a cited, extractive answer\n",
+    "This cell deliberately does not pretend to be generative. It selects a sentence from the highest-ranked evidence and prints the source identifier."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def first_sentence(text):\n",
+    "    parts = re.split(r\"(?<=[.!?])\\s+\", text.strip())\n",
+    "    return parts[0]\n",
+    "\n",
+    "for rank, item in enumerate(candidates[:2], 1):\n",
+    "    print(f\"{first_sentence(item['text'])} [{rank}]\")\n",
+    "    print(f\"  source: {item['source']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Reproducibility notes\n",
+    "- Inspect model and data cards before production use.\n",
+    "- Pin Hub revisions for controlled experiments.\n",
+    "- Retrieval and reranking scores are not calibrated probabilities.\n",
+    "- Evaluate on a held-out corpus from the intended domain."
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {"name": "GöktuğTR end-to-end retrieval.ipynb", "provenance": []},
+  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
+  "language_info": {"name": "python", "version": "3.x"}
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c77c47cacd6425c2b7b73676dfc8626a4eab2c35acad23758ac4776cb4ff9e1f
+size 536221640

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.sentence_transformer.modules.normalize.Normalize"
+  }
+]

release-manifest.json

@@ -0,0 +1,120 @@
+{
+  "schema": "goktugtr-release-manifest-v2",
+  "files": [
+    {
+      "path": "1_Pooling/config.json",
+      "bytes": 95,
+      "sha256": "ed3fb647b7df6cf6c4f9a3f829cf46aeaf32ca350022d7973d28d19d9f8a3eab"
+    },
+    {
+      "path": "README.md",
+      "bytes": 3170,
+      "sha256": "a8c723296a64d0b4a21ccc611a3ef35bb9b951159c25c1da89f5323a4651159d"
+    },
+    {
+      "path": "config.json",
+      "bytes": 1533,
+      "sha256": "3e03b19e47c37f5fd3531918618814ef21eaa958ee1ac23cfd6ea614899ac10c"
+    },
+    {
+      "path": "config_sentence_transformers.json",
+      "bytes": 548,
+      "sha256": "d5177f34a4f8ea422f97a456bedc33dca067e30ae8922ca487cb32064f0b74d1"
+    },
+    {
+      "path": "examples/goktugtr_end_to_end.ipynb",
+      "bytes": 5235,
+      "sha256": "9c539da642076c5451dbee73c529c4849fa551b4ac12ec888ef5b2048c7a1737"
+    },
+    {
+      "path": "model.safetensors",
+      "bytes": 536221640,
+      "sha256": "c77c47cacd6425c2b7b73676dfc8626a4eab2c35acad23758ac4776cb4ff9e1f"
+    },
+    {
+      "path": "modules.json",
+      "bytes": 429,
+      "sha256": "fa7d796a215e1b2416773062891923b191fc5a4849bc44e322997433c90f459c"
+    },
+    {
+      "path": "results/baseline-triplet.json",
+      "bytes": 64,
+      "sha256": "16157fb98652e277c6c2bfa6374d819081e98b27f910d0520510c0e8be9e0e30"
+    },
+    {
+      "path": "results/environment.json",
+      "bytes": 1332,
+      "sha256": "e78aec2d95f9c8c9d3d04b140a38eac7f608e2a2c49717d933229f31d19f72da"
+    },
+    {
+      "path": "results/final-triplet.json",
+      "bytes": 64,
+      "sha256": "16157fb98652e277c6c2bfa6374d819081e98b27f910d0520510c0e8be9e0e30"
+    },
+    {
+      "path": "results/hard-negatives-overlap-audit.json",
+      "bytes": 562,
+      "sha256": "6034bc5f95cf5426abfce67bbd38335bbdf3bf118e34741123b903d12d219980"
+    },
+    {
+      "path": "results/mining-report.json",
+      "bytes": 731,
+      "sha256": "ca2cf95c9e68bc737a67c91d9234ee69daa680aab834c7a81251e88b50dedbf6"
+    },
+    {
+      "path": "results/mteb-five-task.json",
+      "bytes": 48722,
+      "sha256": "e78c4b02781781c9f3fd47e72ef773bcd242c563cef95e4dabc3a2dd29928daa"
+    },
+    {
+      "path": "results/suite-comparison.json",
+      "bytes": 1220,
+      "sha256": "427ff4549dda41d37c35928032d027c343673fec956748a5ef585227aa3ea528"
+    },
+    {
+      "path": "results/suite-manifest.json",
+      "bytes": 1541,
+      "sha256": "e183e75aed887112981a3bce5c94135817209e4761c6cd603c2cf2eb609c0bbb"
+    },
+    {
+      "path": "results/trainer-state.json",
+      "bytes": 15475,
+      "sha256": "26ef0d49bf244ff5f8e98cf0efd0511626c8a6b12fdca0b20fa1f320adddc924"
+    },
+    {
+      "path": "sentence_bert_config.json",
+      "bytes": 241,
+      "sha256": "3084164002c0bca01b0259c5327123803fce32e660a57feb93184ffead186fc8"
+    },
+    {
+      "path": "tokenizer.json",
+      "bytes": 33385259,
+      "sha256": "bd5f1ae5d7b01aadec7b28edcb1b218ff7ac3b73345cd73b40f463c9c3bab1a4"
+    },
+    {
+      "path": "tokenizer_config.json",
+      "bytes": 866,
+      "sha256": "c5f6794371c098425f17e2651f42b7a1784a6f71ed63947eb7af9b6fdea13c33"
+    },
+    {
+      "path": "training/evaluate_suite.py",
+      "bytes": 5008,
+      "sha256": "cd6e4948cc62603c2ddfba0a8ec348c7b2a998143be6ddad9b87cc641e2c6092"
+    },
+    {
+      "path": "training/mine_hard_negatives.py",
+      "bytes": 6796,
+      "sha256": "0d35e9a698b8ecb988748db8c1c0b7f6e94a017147b33ad4c3b8a1f834e3b7f0"
+    },
+    {
+      "path": "training/train.py",
+      "bytes": 8044,
+      "sha256": "183a430cba5484558ff219aeb9115cb43d3fc158b901501e769fd1f90e67bc64"
+    },
+    {
+      "path": "training/train.yaml",
+      "bytes": 540,
+      "sha256": "723cd78b304131a5fae323101384adc1a8d8e983250ca8547ca4207edb79b2fe"
+    }
+  ]
+}

results/baseline-triplet.json

@@ -0,0 +1,3 @@
+{
+  "goktugtr-validation_cosine_accuracy": 0.8934999704360962
+}

results/environment.json

@@ -0,0 +1,47 @@
+{
+  "python": "3.13.12",
+  "torch": "2.8.0+cu129",
+  "transformers": "5.14.1",
+  "sentence_transformers": "5.7.0",
+  "datasets": "4.8.5",
+  "cuda": "12.9",
+  "gpu": "NVIDIA GeForce RTX 5060 Laptop GPU",
+  "gpu_total_memory_gb": 7.527,
+  "config": {
+    "project_name": "goktugtr-retrieval-270m-v2-hard-negatives",
+    "base_model": "outputs/goktugtr-270m/final",
+    "base_model_revision": null,
+    "query_style": "harrier",
+    "padding_side": "left",
+    "dataset_id": "local/goktugtr-hard-negatives-v1",
+    "seed": 3407,
+    "max_seq_length": 256,
+    "train_rows": 50000,
+    "validation_rows": 2000,
+    "epochs": 1,
+    "learning_rate": 8e-06,
+    "warmup_ratio": 0.05,
+    "per_device_batch_size": 16,
+    "gradient_accumulation_steps": 4,
+    "loss": "cached_multiple_negatives_ranking",
+    "loss_mini_batch_size": 8,
+    "gradient_checkpointing": true,
+    "bf16": true,
+    "eval_steps": 125,
+    "save_steps": 125,
+    "logging_steps": 10
+  },
+  "train_rows": 50000,
+  "validation_rows": 2000,
+  "training_seconds": 2528.5468545341864,
+  "resumed_from_checkpoint": null,
+  "training_metrics": {
+    "train_runtime": 2528.4185,
+    "train_samples_per_second": 19.775,
+    "train_steps_per_second": 0.309,
+    "total_flos": 0.0,
+    "train_loss": 0.8376862271057676,
+    "epoch": 1.0
+  },
+  "max_gpu_memory_gb": 2.329
+}

results/final-triplet.json

@@ -0,0 +1,3 @@
+{
+  "goktugtr-validation_cosine_accuracy": 0.8934999704360962
+}

results/hard-negatives-overlap-audit.json

@@ -0,0 +1,14 @@
+{
+  "method": "sha256_of_casefolded_project_normalization",
+  "scope": "exact_normalized_text_overlap_only_not_semantic_or_near_duplicate_detection",
+  "training_rows": 50000,
+  "training_unique_queries": 46050,
+  "training_unique_passages": 69644,
+  "benchmark": "mteb/TurHistQuadRetrieval",
+  "benchmark_revision": "b6e74379b7486da28ce81c3d459cd7bbd87d4987",
+  "benchmark_queries": 1024,
+  "benchmark_documents": 1213,
+  "query_to_query_exact_overlaps": 0,
+  "passage_to_document_exact_overlaps": 0,
+  "any_training_text_to_benchmark_text_exact_overlaps": 0
+}

results/mining-report.json

@@ -0,0 +1,25 @@
+{
+  "method": "dense_hnsw_search_over_original_labeled_negatives",
+  "miner_model": "outputs/goktugtr-270m/final",
+  "query_style": "harrier",
+  "benchmark_data_used": false,
+  "rows": 50000,
+  "candidate_documents": 70172,
+  "search_k": 32,
+  "max_positive_token_jaccard": 0.8,
+  "mined_rows": 50000,
+  "fallback_rows": 0,
+  "mean_selected_cosine": 0.5503763144233823,
+  "mean_selected_rank": 1.00822,
+  "seconds": 209.525791735854,
+  "files": {
+    "train.parquet": {
+      "bytes": 32819198,
+      "sha256": "549c27403e51b200f8fa0a22005069134e673f0f89f34077922894fdc5e2a457"
+    },
+    "validation.parquet": {
+      "bytes": 936162,
+      "sha256": "f4ab199d49039e8efbfdeb84da1a549bd0c61d7b929a96e9c02cd48afcc39061"
+    }
+  }
+}

results/mteb-five-task.json

@@ -0,0 +1,1094 @@
+{
+  "suite": "goktugtr-turkish-retrieval-suite-v1",
+  "language_filter": "turkish_only_exclusive",
+  "model": "GöktuğTR 270M v2 hard negatives",
+  "model_source": "outputs/goktugtr-270m-v2/final",
+  "model_revision": null,
+  "parameters": 268098176,
+  "embedding_dimension": 640,
+  "inference_dtype": "bfloat16",
+  "normalized_embeddings": true,
+  "prompt_style": "harrier",
+  "device": "NVIDIA GeForce RTX 5060 Laptop GPU",
+  "mteb_version": "2.18.16",
+  "task_main_scores": {
+    "TurHistQuadRetrieval": 0.42198,
+    "XQuADRetrieval": 0.86393,
+    "WebFAQRetrieval": 0.56886,
+    "MKQARetrieval": 0.10331,
+    "BelebeleRetrieval": 0.88493
+  },
+  "macro_average": 0.568602,
+  "tasks": [
+    {
+      "name": "TurHistQuadRetrieval",
+      "dataset": {
+        "path": "asparius/TurHistQuAD",
+        "revision": "2a2b8ddecf1189f530676244d0751e1d0a569e03"
+      },
+      "license": "mit",
+      "domains": [
+        "Encyclopaedic",
+        "Non-fiction",
+        "Academic",
+        "Written"
+      ],
+      "eval_splits": [
+        "test"
+      ],
+      "subsets": [
+        "default"
+      ]
+    },
+    {
+      "name": "XQuADRetrieval",
+      "dataset": {
+        "path": "google/xquad",
+        "revision": "51adfef1c1287aab1d2d91b5bead9bcfb9c68583"
+      },
+      "license": "cc-by-sa-4.0",
+      "domains": [
+        "Web",
+        "Written"
+      ],
+      "eval_splits": [
+        "validation"
+      ],
+      "subsets": [
+        "tr"
+      ]
+    },
+    {
+      "name": "WebFAQRetrieval",
+      "dataset": {
+        "path": "mteb/WebFAQRetrieval",
+        "revision": "f64f483ad0f31d2e78209d524c14a4a867965959"
+      },
+      "license": "cc-by-4.0",
+      "domains": [
+        "Web",
+        "Written"
+      ],
+      "eval_splits": [
+        "test"
+      ],
+      "subsets": [
+        "tur"
+      ]
+    },
+    {
+      "name": "MKQARetrieval",
+      "dataset": {
+        "path": "mteb/MKQARetrieval",
+        "revision": "3e069e7a30079d214a859741f5a0de75cf878867"
+      },
+      "license": "cc-by-3.0",
+      "domains": [
+        "Written"
+      ],
+      "eval_splits": [
+        "train"
+      ],
+      "subsets": [
+        "tr"
+      ]
+    },
+    {
+      "name": "BelebeleRetrieval",
+      "dataset": {
+        "path": "mteb/belebele",
+        "revision": "979a211276faa22f671e69d096634193567cfd05"
+      },
+      "license": "cc-by-sa-4.0",
+      "domains": [
+        "Web",
+        "News",
+        "Written"
+      ],
+      "eval_splits": [
+        "test"
+      ],
+      "subsets": [
+        "tur_Latn-tur_Latn"
+      ]
+    }
+  ],
+  "raw_mteb_result": {
+    "model_name": "no_model_name/available",
+    "model_revision": "no_revision_available",
+    "task_results": [
+      {
+        "dataset_revision": "2a2b8ddecf1189f530676244d0751e1d0a569e03",
+        "task_name": "TurHistQuadRetrieval",
+        "mteb_version": "2.18.16",
+        "scores": {
+          "test": [
+            {
+              "ndcg_at_1": 0.44727,
+              "ndcg_at_3": 0.36818,
+              "ndcg_at_5": 0.39742,
+              "ndcg_at_10": 0.42198,
+              "ndcg_at_20": 0.44086,
+              "ndcg_at_100": 0.47466,
+              "ndcg_at_1000": 0.51823,
+              "map_at_1": 0.22363,
+              "map_at_3": 0.29012,
+              "map_at_5": 0.30726,
+              "map_at_10": 0.31975,
+              "map_at_20": 0.3268,
+              "map_at_100": 0.33331,
+              "map_at_1000": 0.33552,
+              "recall_at_1": 0.22363,
+              "recall_at_3": 0.35498,
+              "recall_at_5": 0.41357,
+              "recall_at_10": 0.4751,
+              "recall_at_20": 0.53516,
+              "recall_at_100": 0.68604,
+              "recall_at_1000": 0.97266,
+              "accuracy": 0.22363,
+              "precision_at_1": 0.44727,
+              "precision_at_3": 0.23665,
+              "precision_at_5": 0.16543,
+              "precision_at_10": 0.09502,
+              "precision_at_20": 0.05352,
+              "precision_at_100": 0.01372,
+              "precision_at_1000": 0.00195,
+              "mrr_at_1": 0.447265625,
+              "mrr_at_3": 0.54541015625,
+              "mrr_at_5": 0.563134765625,
+              "mrr_at_10": 0.5715328156001984,
+              "mrr_at_20": 0.5747740260525898,
+              "mrr_at_100": 0.5764007840136673,
+              "mrr_at_1000": 0.5767683869085801,
+              "nauc_ndcg_at_1_max": 0.1947773993090055,
+              "nauc_ndcg_at_1_std": -0.0910494007784227,
+              "nauc_ndcg_at_1_diff1": 0.4073158485432013,
+              "nauc_ndcg_at_3_max": 0.15459659971936998,
+              "nauc_ndcg_at_3_std": -0.13572605317548517,
+              "nauc_ndcg_at_3_diff1": 0.3590182901499481,
+              "nauc_ndcg_at_5_max": 0.12351242622860217,
+              "nauc_ndcg_at_5_std": -0.13862840806665486,
+              "nauc_ndcg_at_5_diff1": 0.3411180215230726,
+              "nauc_ndcg_at_10_max": 0.10508451932152987,
+              "nauc_ndcg_at_10_std": -0.15917084194618117,
+              "nauc_ndcg_at_10_diff1": 0.3456925788408306,
+              "nauc_ndcg_at_20_max": 0.07934060994772754,
+              "nauc_ndcg_at_20_std": -0.17556108096439801,
+              "nauc_ndcg_at_20_diff1": 0.34005209604606523,
+              "nauc_ndcg_at_100_max": 0.06263250772213183,
+              "nauc_ndcg_at_100_std": -0.19644024854652758,
+              "nauc_ndcg_at_100_diff1": 0.34964007250841367,
+              "nauc_ndcg_at_1000_max": 0.06570815848059838,
+              "nauc_ndcg_at_1000_std": -0.1926137221140858,
+              "nauc_ndcg_at_1000_diff1": 0.36291691156660905,
+              "nauc_map_at_1_max": 0.1947773993090055,
+              "nauc_map_at_1_std": -0.0910494007784227,
+              "nauc_map_at_1_diff1": 0.4073158485432013,
+              "nauc_map_at_3_max": 0.15482569356800127,
+              "nauc_map_at_3_std": -0.12953054928321617,
+              "nauc_map_at_3_diff1": 0.3547572884337557,
+              "nauc_map_at_5_max": 0.13400725043834277,
+              "nauc_map_at_5_std": -0.13487559474480726,
+              "nauc_map_at_5_diff1": 0.3411052428046666,
+              "nauc_map_at_10_max": 0.12580634709979557,
+              "nauc_map_at_10_std": -0.14482991442291338,
+              "nauc_map_at_10_diff1": 0.3388736900349583,
+              "nauc_map_at_20_max": 0.1171912993522551,
+              "nauc_map_at_20_std": -0.15057501514134508,
+              "nauc_map_at_20_diff1": 0.33579860813933154,
+              "nauc_map_at_100_max": 0.11439780365587258,
+              "nauc_map_at_100_std": -0.15406029446515598,
+              "nauc_map_at_100_diff1": 0.3366239925867801,
+              "nauc_map_at_1000_max": 0.11378034070804263,
+              "nauc_map_at_1000_std": -0.15413650834976175,
+              "nauc_map_at_1000_diff1": 0.33743800581253264,
+              "nauc_recall_at_1_max": 0.1947773993090055,
+              "nauc_recall_at_1_std": -0.0910494007784227,
+              "nauc_recall_at_1_diff1": 0.4073158485432013,
+              "nauc_recall_at_3_max": 0.11721165014618391,
+              "nauc_recall_at_3_std": -0.16453793054408086,
+              "nauc_recall_at_3_diff1": 0.31826402362729905,
+              "nauc_recall_at_5_max": 0.032770031656111503,
+              "nauc_recall_at_5_std": -0.16108032882961745,
+              "nauc_recall_at_5_diff1": 0.24788015226976987,
+              "nauc_recall_at_10_max": -0.015009736950870631,
+              "nauc_recall_at_10_std": -0.2028271635328523,
+              "nauc_recall_at_10_diff1": 0.24498908438888148,
+              "nauc_recall_at_20_max": -0.09578925764875043,
+              "nauc_recall_at_20_std": -0.22857085957680415,
+              "nauc_recall_at_20_diff1": 0.19408264003786216,
+              "nauc_recall_at_100_max": -0.16490966442677502,
+              "nauc_recall_at_100_std": -0.3015537080276642,
+              "nauc_recall_at_100_diff1": 0.21955302141734043,
+              "nauc_recall_at_1000_max": -1.3782566762561899,
+              "nauc_recall_at_1000_std": -1.2793437787675817,
+              "nauc_recall_at_1000_diff1": 0.7032453889650138,
+              "nauc_precision_at_1_max": 0.1947773993090055,
+              "nauc_precision_at_1_std": -0.0910494007784227,
+              "nauc_precision_at_1_diff1": 0.4073158485432013,
+              "nauc_precision_at_3_max": 0.11721165014618445,
+              "nauc_precision_at_3_std": -0.16453793054408086,
+              "nauc_precision_at_3_diff1": 0.3182640236272993,
+              "nauc_precision_at_5_max": 0.03277003165611099,
+              "nauc_precision_at_5_std": -0.1610803288296178,
+              "nauc_precision_at_5_diff1": 0.2478801522697694,
+              "nauc_precision_at_10_max": -0.01500973695087072,
+              "nauc_precision_at_10_std": -0.20282716353285224,
+              "nauc_precision_at_10_diff1": 0.24498908438888103,
+              "nauc_precision_at_20_max": -0.0957892576487511,
+              "nauc_precision_at_20_std": -0.22857085957680465,
+              "nauc_precision_at_20_diff1": 0.1940826400378614,
+              "nauc_precision_at_100_max": -0.16490966442677507,
+              "nauc_precision_at_100_std": -0.30155370802766424,
+              "nauc_precision_at_100_diff1": 0.21955302141734045,
+              "nauc_precision_at_1000_max": -1.378256676256199,
+              "nauc_precision_at_1000_std": -1.279343778767596,
+              "nauc_precision_at_1000_diff1": 0.7032453889650114,
+              "nauc_mrr_at_1_max": 0.1947773993090055,
+              "nauc_mrr_at_1_std": -0.0910494007784227,
+              "nauc_mrr_at_1_diff1": 0.4073158485432013,
+              "nauc_mrr_at_3_max": 0.17951015442495216,
+              "nauc_mrr_at_3_std": -0.11954604918828698,
+              "nauc_mrr_at_3_diff1": 0.3878193805370859,
+              "nauc_mrr_at_5_max": 0.1690611363913176,
+              "nauc_mrr_at_5_std": -0.11456851241986926,
+              "nauc_mrr_at_5_diff1": 0.3854262100889673,
+              "nauc_mrr_at_10_max": 0.16279109738166908,
+              "nauc_mrr_at_10_std": -0.12140021284110425,
+              "nauc_mrr_at_10_diff1": 0.39431158801201477,
+              "nauc_mrr_at_20_max": 0.15854074365731471,
+              "nauc_mrr_at_20_std": -0.12493806418765903,
+              "nauc_mrr_at_20_diff1": 0.39652712372135157,
+              "nauc_mrr_at_100_max": 0.15656075467937358,
+              "nauc_mrr_at_100_std": -0.12717669984150143,
+              "nauc_mrr_at_100_diff1": 0.39764245610734594,
+              "nauc_mrr_at_1000_max": 0.15714128366655825,
+              "nauc_mrr_at_1000_std": -0.1268302978503602,
+              "nauc_mrr_at_1000_diff1": 0.3974845927958745,
+              "hit_rate_at_1": 0.44727,
+              "hit_rate_at_3": 0.66797,
+              "hit_rate_at_5": 0.74805,
+              "hit_rate_at_10": 0.80859,
+              "hit_rate_at_20": 0.85352,
+              "hit_rate_at_100": 0.91406,
+              "hit_rate_at_1000": 1.0,
+              "main_score": 0.42198,
+              "hf_subset": "default",
+              "languages": [
+                "tur-Latn"
+              ],
+              "mteb_version": "2.18.16"
+            }
+          ]
+        },
+        "evaluation_time": 6.209755585761741,
+        "kg_co2_emissions": null,
+        "date": "2026-08-11T08:37:55.009831Z",
+        "evaluation_phases": [
+          {
+            "name": "Data loading",
+            "start": 0.0,
+            "end": 1.6432985588908195,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Encoding queries and documents",
+            "start": 1.6578817549161613,
+            "end": 5.618854850064963,
+            "split": "test",
+            "subset": "default"
+          },
+          {
+            "name": "Scoring",
+            "start": 5.618875534972176,
+            "end": 6.201895288890228,
+            "split": "test",
+            "subset": "default"
+          }
+        ]
+      },
+      {
+        "dataset_revision": "51adfef1c1287aab1d2d91b5bead9bcfb9c68583",
+        "task_name": "XQuADRetrieval",
+        "mteb_version": "2.18.16",
+        "scores": {
+          "validation": [
+            {
+              "ndcg_at_1": 0.81588,
+              "ndcg_at_3": 0.85376,
+              "ndcg_at_5": 0.85972,
+              "ndcg_at_10": 0.86393,
+              "ndcg_at_20": 0.87064,
+              "ndcg_at_100": 0.88069,
+              "ndcg_at_1000": 0.88277,
+              "map_at_1": 0.81588,
+              "map_at_3": 0.84516,
+              "map_at_5": 0.84849,
+              "map_at_10": 0.85014,
+              "map_at_20": 0.85191,
+              "map_at_100": 0.8535,
+              "map_at_1000": 0.8536,
+              "recall_at_1": 0.81588,
+              "recall_at_3": 0.87838,
+              "recall_at_5": 0.89274,
+              "recall_at_10": 0.90625,
+              "recall_at_20": 0.93328,
+              "recall_at_100": 0.9848,
+              "recall_at_1000": 1.0,
+              "accuracy": 0.81588,
+              "precision_at_1": 0.81588,
+              "precision_at_3": 0.29279,
+              "precision_at_5": 0.17855,
+              "precision_at_10": 0.09063,
+              "precision_at_20": 0.04666,
+              "precision_at_100": 0.00985,
+              "precision_at_1000": 0.001,
+              "mrr_at_1": 0.8158783783783784,
+              "mrr_at_3": 0.845579954954955,
+              "mrr_at_5": 0.8487471846846847,
+              "mrr_at_10": 0.8497372372372372,
+              "mrr_at_20": 0.8519017309373831,
+              "mrr_at_100": 0.853515089167124,
+              "mrr_at_1000": 0.8536142784769822,
+              "nauc_ndcg_at_1_max": 0.3284540665242737,
+              "nauc_ndcg_at_1_std": -0.03350450468321942,
+              "nauc_ndcg_at_1_diff1": 0.7757754213683679,
+              "nauc_ndcg_at_3_max": 0.25818700964639063,
+              "nauc_ndcg_at_3_std": -0.055404890207890786,
+              "nauc_ndcg_at_3_diff1": 0.7357097193459545,
+              "nauc_ndcg_at_5_max": 0.23532978864261622,
+              "nauc_ndcg_at_5_std": -0.070773060535053,
+              "nauc_ndcg_at_5_diff1": 0.7296829159511898,
+              "nauc_ndcg_at_10_max": 0.2221519746395343,
+              "nauc_ndcg_at_10_std": -0.0875650487509888,
+              "nauc_ndcg_at_10_diff1": 0.732598352255786,
+              "nauc_ndcg_at_20_max": 0.20773138536758215,
+              "nauc_ndcg_at_20_std": -0.09989246220154115,
+              "nauc_ndcg_at_20_diff1": 0.7340091998299748,
+              "nauc_ndcg_at_100_max": 0.19515655222487252,
+              "nauc_ndcg_at_100_std": -0.11460185139630633,
+              "nauc_ndcg_at_100_diff1": 0.7352192500871594,
+              "nauc_ndcg_at_1000_max": 0.22818231071148642,
+              "nauc_ndcg_at_1000_std": -0.08733055810540781,
+              "nauc_ndcg_at_1000_diff1": 0.7390905883324356,
+              "nauc_map_at_1_max": 0.3284540665242737,
+              "nauc_map_at_1_std": -0.03350450468321942,
+              "nauc_map_at_1_diff1": 0.7757754213683679,
+              "nauc_map_at_3_max": 0.2777399037391668,
+              "nauc_map_at_3_std": -0.048568184058805995,
+              "nauc_map_at_3_diff1": 0.7462289334451868,
+              "nauc_map_at_5_max": 0.26622347518153544,
+              "nauc_map_at_5_std": -0.0563052173192779,
+              "nauc_map_at_5_diff1": 0.7433725065389168,
+              "nauc_map_at_10_max": 0.26177082716700206,
+              "nauc_map_at_10_std": -0.06208533201755686,
+              "nauc_map_at_10_diff1": 0.7443466727140162,
+              "nauc_map_at_20_max": 0.25872168437955684,
+              "nauc_map_at_20_std": -0.06463743789758596,
+              "nauc_map_at_20_diff1": 0.7448129266649355,
+              "nauc_map_at_100_max": 0.2559148722096524,
+              "nauc_map_at_100_std": -0.06773002736169367,
+              "nauc_map_at_100_diff1": 0.7449903003740266,
+              "nauc_map_at_1000_max": 0.2572009615963307,
+              "nauc_map_at_1000_std": -0.06667363214343108,
+              "nauc_map_at_1000_diff1": 0.7451457131512783,
+              "nauc_recall_at_1_max": 0.3284540665242737,
+              "nauc_recall_at_1_std": -0.03350450468321942,
+              "nauc_recall_at_1_diff1": 0.7757754213683679,
+              "nauc_recall_at_3_max": 0.18820039148395235,
+              "nauc_recall_at_3_std": -0.08026325128177393,
+              "nauc_recall_at_3_diff1": 0.6983862971238585,
+              "nauc_recall_at_5_max": 0.10995206042485102,
+              "nauc_recall_at_5_std": -0.13101453725155376,
+              "nauc_recall_at_5_diff1": 0.6751281440235256,
+              "nauc_recall_at_10_max": 0.03450281639611959,
+              "nauc_recall_at_10_std": -0.21514187982604624,
+              "nauc_recall_at_10_diff1": 0.6827023652581147,
+              "nauc_recall_at_20_max": -0.14669192998797975,
+              "nauc_recall_at_20_std": -0.35926833389834817,
+              "nauc_recall_at_20_diff1": 0.6732122472101519,
+              "nauc_recall_at_100_max": -1.5891802105217783,
+              "nauc_recall_at_100_std": -1.5891802105217783,
+              "nauc_recall_at_100_diff1": 0.5276547460163096,
+              "nauc_recall_at_1000_max": null,
+              "nauc_recall_at_1000_std": null,
+              "nauc_recall_at_1000_diff1": null,
+              "nauc_precision_at_1_max": 0.3284540665242737,
+              "nauc_precision_at_1_std": -0.03350450468321942,
+              "nauc_precision_at_1_diff1": 0.7757754213683679,
+              "nauc_precision_at_3_max": 0.18820039148395123,
+              "nauc_precision_at_3_std": -0.08026325128177557,
+              "nauc_precision_at_3_diff1": 0.6983862971238574,
+              "nauc_precision_at_5_max": 0.10995206042484995,
+              "nauc_precision_at_5_std": -0.13101453725155207,
+              "nauc_precision_at_5_diff1": 0.6751281440235273,
+              "nauc_precision_at_10_max": 0.03450281639612019,
+              "nauc_precision_at_10_std": -0.21514187982604183,
+              "nauc_precision_at_10_diff1": 0.6827023652581166,
+              "nauc_precision_at_20_max": -0.1466919299879782,
+              "nauc_precision_at_20_std": -0.35926833389834945,
+              "nauc_precision_at_20_diff1": 0.6732122472101516,
+              "nauc_precision_at_100_max": -1.5891802105218076,
+              "nauc_precision_at_100_std": -1.5891802105218076,
+              "nauc_precision_at_100_diff1": 0.5276547460163015,
+              "nauc_precision_at_1000_max": 1.0,
+              "nauc_precision_at_1000_std": 1.0,
+              "nauc_precision_at_1000_diff1": 1.0,
+              "nauc_mrr_at_1_max": 0.3284540665242737,
+              "nauc_mrr_at_1_std": -0.03350450468321942,
+              "nauc_mrr_at_1_diff1": 0.7757754213683679,
+              "nauc_mrr_at_3_max": 0.2755646757001353,
+              "nauc_mrr_at_3_std": -0.051182154244866135,
+              "nauc_mrr_at_3_diff1": 0.7452215992777843,
+              "nauc_mrr_at_5_max": 0.2670252442185693,
+              "nauc_mrr_at_5_std": -0.055120825642614404,
+              "nauc_mrr_at_5_diff1": 0.7409265069774154,
+              "nauc_mrr_at_10_max": 0.26289797765516126,
+              "nauc_mrr_at_10_std": -0.05924126003203253,
+              "nauc_mrr_at_10_diff1": 0.7415305795746361,
+              "nauc_mrr_at_20_max": 0.2590123096058126,
+              "nauc_mrr_at_20_std": -0.06499429407428235,
+              "nauc_mrr_at_20_diff1": 0.7426366640770949,
+              "nauc_mrr_at_100_max": 0.2560796971787312,
+              "nauc_mrr_at_100_std": -0.06747575427838415,
+              "nauc_mrr_at_100_diff1": 0.7427706567785499,
+              "nauc_mrr_at_1000_max": 0.2573657793463667,
+              "nauc_mrr_at_1000_std": -0.06641928062359208,
+              "nauc_mrr_at_1000_diff1": 0.7429244793968778,
+              "hit_rate_at_1": 0.81588,
+              "hit_rate_at_3": 0.87838,
+              "hit_rate_at_5": 0.89274,
+              "hit_rate_at_10": 0.90625,
+              "hit_rate_at_20": 0.93328,
+              "hit_rate_at_100": 0.9848,
+              "hit_rate_at_1000": 1.0,
+              "main_score": 0.86393,
+              "hf_subset": "tr",
+              "languages": [
+                "tur-Latn"
+              ],
+              "mteb_version": "2.18.16"
+            }
+          ]
+        },
+        "evaluation_time": 5.384912968846038,
+        "kg_co2_emissions": null,
+        "date": "2026-08-11T08:38:00.395989Z",
+        "evaluation_phases": [
+          {
+            "name": "Data loading",
+            "start": 0.0,
+            "end": 2.4216172359883785,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Encoding queries and documents",
+            "start": 2.435616716975346,
+            "end": 5.167067898903042,
+            "split": "validation",
+            "subset": "tr"
+          },
+          {
+            "name": "Scoring",
+            "start": 5.167083007982001,
+            "end": 5.382447937969118,
+            "split": "validation",
+            "subset": "tr"
+          }
+        ]
+      },
+      {
+        "dataset_revision": "f64f483ad0f31d2e78209d524c14a4a867965959",
+        "task_name": "WebFAQRetrieval",
+        "mteb_version": "2.18.16",
+        "scores": {
+          "test": [
+            {
+              "ndcg_at_1": 0.4783,
+              "ndcg_at_3": 0.53775,
+              "ndcg_at_5": 0.55368,
+              "ndcg_at_10": 0.56886,
+              "ndcg_at_20": 0.58072,
+              "ndcg_at_100": 0.59863,
+              "ndcg_at_1000": 0.60486,
+              "map_at_1": 0.4783,
+              "map_at_3": 0.52333,
+              "map_at_5": 0.53216,
+              "map_at_10": 0.53846,
+              "map_at_20": 0.5417,
+              "map_at_100": 0.54424,
+              "map_at_1000": 0.54448,
+              "recall_at_1": 0.4783,
+              "recall_at_3": 0.5794,
+              "recall_at_5": 0.6181,
+              "recall_at_10": 0.6649,
+              "recall_at_20": 0.7119,
+              "recall_at_100": 0.8074,
+              "recall_at_1000": 0.8563,
+              "accuracy": 0.4783,
+              "precision_at_1": 0.4783,
+              "precision_at_3": 0.19313,
+              "precision_at_5": 0.12362,
+              "precision_at_10": 0.06649,
+              "precision_at_20": 0.0356,
+              "precision_at_100": 0.00807,
+              "precision_at_1000": 0.00086,
+              "mrr_at_1": 0.4782,
+              "mrr_at_3": 0.5233166666666667,
+              "mrr_at_5": 0.5320866666666667,
+              "mrr_at_10": 0.5383844841269841,
+              "mrr_at_20": 0.5416332922284973,
+              "mrr_at_100": 0.544169475461704,
+              "mrr_at_1000": 0.5444125970973864,
+              "nauc_ndcg_at_1_max": 0.12378526993465135,
+              "nauc_ndcg_at_1_std": 0.15914555681902476,
+              "nauc_ndcg_at_1_diff1": 0.8371664323172846,
+              "nauc_ndcg_at_3_max": 0.059497603641565644,
+              "nauc_ndcg_at_3_std": 0.19280724617182773,
+              "nauc_ndcg_at_3_diff1": 0.8106765381725193,
+              "nauc_ndcg_at_5_max": 0.03830703968134529,
+              "nauc_ndcg_at_5_std": 0.20203097792669134,
+              "nauc_ndcg_at_5_diff1": 0.8069760545861215,
+              "nauc_ndcg_at_10_max": 0.01692778814755956,
+              "nauc_ndcg_at_10_std": 0.2100322549145765,
+              "nauc_ndcg_at_10_diff1": 0.805655549489814,
+              "nauc_ndcg_at_20_max": 0.0018299159845378435,
+              "nauc_ndcg_at_20_std": 0.21686590866569472,
+              "nauc_ndcg_at_20_diff1": 0.8035360516982142,
+              "nauc_ndcg_at_100_max": -0.023569682566652816,
+              "nauc_ndcg_at_100_std": 0.2278208198263667,
+              "nauc_ndcg_at_100_diff1": 0.8051237611879092,
+              "nauc_ndcg_at_1000_max": -0.03534215224230123,
+              "nauc_ndcg_at_1000_std": 0.23034163818382103,
+              "nauc_ndcg_at_1000_diff1": 0.8088858890505707,
+              "nauc_map_at_1_max": 0.12378526993465135,
+              "nauc_map_at_1_std": 0.15914555681902476,
+              "nauc_map_at_1_diff1": 0.8371664323172846,
+              "nauc_map_at_3_max": 0.07601436468829656,
+              "nauc_map_at_3_std": 0.1844036695986002,
+              "nauc_map_at_3_diff1": 0.8172367412045666,
+              "nauc_map_at_5_max": 0.06473499918618558,
+              "nauc_map_at_5_std": 0.1892017055047459,
+              "nauc_map_at_5_diff1": 0.8152117908664681,
+              "nauc_map_at_10_max": 0.05655810792894781,
+              "nauc_map_at_10_std": 0.1922956559395659,
+              "nauc_map_at_10_diff1": 0.8148003875474032,
+              "nauc_map_at_20_max": 0.052812545064260606,
+              "nauc_map_at_20_std": 0.1939204688449605,
+              "nauc_map_at_20_diff1": 0.814304031703465,
+              "nauc_map_at_100_max": 0.049870914558898784,
+              "nauc_map_at_100_std": 0.19525523675361117,
+              "nauc_map_at_100_diff1": 0.8145625414323227,
+              "nauc_map_at_1000_max": 0.04951462246634232,
+              "nauc_map_at_1000_std": 0.19535656764417045,
+              "nauc_map_at_1000_diff1": 0.814689665486308,
+              "nauc_recall_at_1_max": 0.12378526993465135,
+              "nauc_recall_at_1_std": 0.15914555681902476,
+              "nauc_recall_at_1_diff1": 0.8371664323172846,
+              "nauc_recall_at_3_max": 0.00906800781090249,
+              "nauc_recall_at_3_std": 0.21836130557966743,
+              "nauc_recall_at_3_diff1": 0.7907510757682494,
+              "nauc_recall_at_5_max": -0.049291960498243304,
+              "nauc_recall_at_5_std": 0.244415239072271,
+              "nauc_recall_at_5_diff1": 0.780370627517092,
+              "nauc_recall_at_10_max": -0.1334658927813473,
+              "nauc_recall_at_10_std": 0.2763889626664622,
+              "nauc_recall_at_10_diff1": 0.7730445065609719,
+              "nauc_recall_at_20_max": -0.22424014427676828,
+              "nauc_recall_at_20_std": 0.3179950761211255,
+              "nauc_recall_at_20_diff1": 0.7589578131314744,
+              "nauc_recall_at_100_max": -0.5378719532193261,
+              "nauc_recall_at_100_std": 0.4528372991444859,
+              "nauc_recall_at_100_diff1": 0.756077007675729,
+              "nauc_recall_at_1000_max": -0.8933631153969248,
+              "nauc_recall_at_1000_std": 0.5591965880094494,
+              "nauc_recall_at_1000_diff1": 0.8049804885079451,
+              "nauc_precision_at_1_max": 0.12378526993465135,
+              "nauc_precision_at_1_std": 0.15914555681902476,
+              "nauc_precision_at_1_diff1": 0.8371664323172846,
+              "nauc_precision_at_3_max": 0.009068007810902291,
+              "nauc_precision_at_3_std": 0.21836130557966757,
+              "nauc_precision_at_3_diff1": 0.7907510757682494,
+              "nauc_precision_at_5_max": -0.0492919604982433,
+              "nauc_precision_at_5_std": 0.24441523907227103,
+              "nauc_precision_at_5_diff1": 0.7803706275170921,
+              "nauc_precision_at_10_max": -0.13346589278134638,
+              "nauc_precision_at_10_std": 0.27638896266646273,
+              "nauc_precision_at_10_diff1": 0.773044506560973,
+              "nauc_precision_at_20_max": -0.22424014427676833,
+              "nauc_precision_at_20_std": 0.31799507612112565,
+              "nauc_precision_at_20_diff1": 0.758957813131475,
+              "nauc_precision_at_100_max": -0.5378719532193269,
+              "nauc_precision_at_100_std": 0.4528372991444852,
+              "nauc_precision_at_100_diff1": 0.7560770076757304,
+              "nauc_precision_at_1000_max": -0.8933631153969233,
+              "nauc_precision_at_1000_std": 0.5591965880094497,
+              "nauc_precision_at_1000_diff1": 0.8049804885079432,
+              "nauc_mrr_at_1_max": 0.1234547429404733,
+              "nauc_mrr_at_1_std": 0.1586471848506029,
+              "nauc_mrr_at_1_diff1": 0.8374205635455932,
+              "nauc_mrr_at_3_max": 0.0759591156474296,
+              "nauc_mrr_at_3_std": 0.18432139049652596,
+              "nauc_mrr_at_3_diff1": 0.8172997943423154,
+              "nauc_mrr_at_5_max": 0.06458798468556289,
+              "nauc_mrr_at_5_std": 0.18879150925263186,
+              "nauc_mrr_at_5_diff1": 0.8153588873994831,
+              "nauc_mrr_at_10_max": 0.05640819131273918,
+              "nauc_mrr_at_10_std": 0.19202416957177562,
+              "nauc_mrr_at_10_diff1": 0.8149415213237646,
+              "nauc_mrr_at_20_max": 0.05261900861603199,
+              "nauc_mrr_at_20_std": 0.19368254224028988,
+              "nauc_mrr_at_20_diff1": 0.8144435264474017,
+              "nauc_mrr_at_100_max": 0.04969749121247066,
+              "nauc_mrr_at_100_std": 0.19500516541332255,
+              "nauc_mrr_at_100_diff1": 0.8147021280654135,
+              "nauc_mrr_at_1000_max": 0.04934573213223441,
+              "nauc_mrr_at_1000_std": 0.1951105366077533,
+              "nauc_mrr_at_1000_diff1": 0.8148301808658873,
+              "hit_rate_at_1": 0.4783,
+              "hit_rate_at_3": 0.5794,
+              "hit_rate_at_5": 0.6181,
+              "hit_rate_at_10": 0.6649,
+              "hit_rate_at_20": 0.7119,
+              "hit_rate_at_100": 0.8074,
+              "hit_rate_at_1000": 0.8563,
+              "main_score": 0.56886,
+              "hf_subset": "tur",
+              "languages": [
+                "tur-Latn"
+              ],
+              "mteb_version": "2.18.16"
+            }
+          ]
+        },
+        "evaluation_time": 308.66258307010867,
+        "kg_co2_emissions": null,
+        "date": "2026-08-11T08:43:09.059761Z",
+        "evaluation_phases": [
+          {
+            "name": "Data loading",
+            "start": 4.1369348764419556e-06,
+            "end": 8.95585033413954,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Dataset transform",
+            "start": 8.955858692061156,
+            "end": 8.955860669957474,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Encoding queries and documents",
+            "start": 9.023173480993137,
+            "end": 301.69988178205676,
+            "split": "test",
+            "subset": "tur"
+          },
+          {
+            "name": "Scoring",
+            "start": 301.69990209210664,
+            "end": 308.4712615769822,
+            "split": "test",
+            "subset": "tur"
+          }
+        ]
+      },
+      {
+        "dataset_revision": "3e069e7a30079d214a859741f5a0de75cf878867",
+        "task_name": "MKQARetrieval",
+        "mteb_version": "2.18.16",
+        "scores": {
+          "train": [
+            {
+              "ndcg_at_1": 0.06139,
+              "ndcg_at_3": 0.07844,
+              "ndcg_at_5": 0.08876,
+              "ndcg_at_10": 0.10331,
+              "ndcg_at_20": 0.11739,
+              "ndcg_at_100": 0.14806,
+              "ndcg_at_1000": 0.19718,
+              "map_at_1": 0.04632,
+              "map_at_3": 0.06645,
+              "map_at_5": 0.07352,
+              "map_at_10": 0.07996,
+              "map_at_20": 0.08404,
+              "map_at_100": 0.08831,
+              "map_at_1000": 0.08992,
+              "recall_at_1": 0.04632,
+              "recall_at_3": 0.08999,
+              "recall_at_5": 0.11716,
+              "recall_at_10": 0.15855,
+              "recall_at_20": 0.20969,
+              "recall_at_100": 0.36572,
+              "recall_at_1000": 0.7445,
+              "accuracy": 0.04632,
+              "precision_at_1": 0.06139,
+              "precision_at_3": 0.04172,
+              "precision_at_5": 0.03307,
+              "precision_at_10": 0.02291,
+              "precision_at_20": 0.01526,
+              "precision_at_100": 0.00531,
+              "precision_at_1000": 0.00107,
+              "mrr_at_1": 0.06138790035587188,
+              "mrr_at_3": 0.08311091340450771,
+              "mrr_at_5": 0.09005782918149466,
+              "mrr_at_10": 0.09632983580937318,
+              "mrr_at_20": 0.10029345491156058,
+              "mrr_at_100": 0.10418328055808178,
+              "mrr_at_1000": 0.10551962334234978,
+              "nauc_ndcg_at_1_max": 0.19046090779456812,
+              "nauc_ndcg_at_1_std": 0.2005678293225855,
+              "nauc_ndcg_at_1_diff1": 0.2732613145926889,
+              "nauc_ndcg_at_3_max": 0.154650069438559,
+              "nauc_ndcg_at_3_std": 0.21864693259137372,
+              "nauc_ndcg_at_3_diff1": 0.19561940316458706,
+              "nauc_ndcg_at_5_max": 0.14091485629423525,
+              "nauc_ndcg_at_5_std": 0.23477553800788697,
+              "nauc_ndcg_at_5_diff1": 0.17968447890707392,
+              "nauc_ndcg_at_10_max": 0.12493779940679503,
+              "nauc_ndcg_at_10_std": 0.25141145550716937,
+              "nauc_ndcg_at_10_diff1": 0.16150290010603824,
+              "nauc_ndcg_at_20_max": 0.11065180702665893,
+              "nauc_ndcg_at_20_std": 0.27110351591711057,
+              "nauc_ndcg_at_20_diff1": 0.15703674738718967,
+              "nauc_ndcg_at_100_max": 0.08421783643347659,
+              "nauc_ndcg_at_100_std": 0.2946243071156509,
+              "nauc_ndcg_at_100_diff1": 0.1502281839569693,
+              "nauc_ndcg_at_1000_max": 0.060271168700708054,
+              "nauc_ndcg_at_1000_std": 0.30321027407391904,
+              "nauc_ndcg_at_1000_diff1": 0.17005046587139194,
+              "nauc_map_at_1_max": 0.16299753614775253,
+              "nauc_map_at_1_std": 0.16346220475254442,
+              "nauc_map_at_1_diff1": 0.2917363544554654,
+              "nauc_map_at_3_max": 0.15598673546990813,
+              "nauc_map_at_3_std": 0.19971704494790168,
+              "nauc_map_at_3_diff1": 0.21645522028299932,
+              "nauc_map_at_5_max": 0.15166101798657583,
+              "nauc_map_at_5_std": 0.21573359427337446,
+              "nauc_map_at_5_diff1": 0.20455508216082804,
+              "nauc_map_at_10_max": 0.14475635349933652,
+              "nauc_map_at_10_std": 0.22759247812792957,
+              "nauc_map_at_10_diff1": 0.19358022834017713,
+              "nauc_map_at_20_max": 0.13899000442931903,
+              "nauc_map_at_20_std": 0.23628409823009514,
+              "nauc_map_at_20_diff1": 0.1906825576599391,
+              "nauc_map_at_100_max": 0.13372591272884576,
+              "nauc_map_at_100_std": 0.2432631891975316,
+              "nauc_map_at_100_diff1": 0.188127691740388,
+              "nauc_map_at_1000_max": 0.13264497426057695,
+              "nauc_map_at_1000_std": 0.2439228359663457,
+              "nauc_map_at_1000_diff1": 0.18877656163071965,
+              "nauc_recall_at_1_max": 0.16299753614775253,
+              "nauc_recall_at_1_std": 0.16346220475254442,
+              "nauc_recall_at_1_diff1": 0.2917363544554654,
+              "nauc_recall_at_3_max": 0.1314709043759448,
+              "nauc_recall_at_3_std": 0.22219290324299643,
+              "nauc_recall_at_3_diff1": 0.15892511021348496,
+              "nauc_recall_at_5_max": 0.11041311908878675,
+              "nauc_recall_at_5_std": 0.2479865816737892,
+              "nauc_recall_at_5_diff1": 0.12973531101440325,
+              "nauc_recall_at_10_max": 0.08352674383767103,
+              "nauc_recall_at_10_std": 0.27801950065943004,
+              "nauc_recall_at_10_diff1": 0.10277047520590306,
+              "nauc_recall_at_20_max": 0.055451740458610316,
+              "nauc_recall_at_20_std": 0.30864764930100463,
+              "nauc_recall_at_20_diff1": 0.10009990485099715,
+              "nauc_recall_at_100_max": -0.010313300454429164,
+              "nauc_recall_at_100_std": 0.3507341463365123,
+              "nauc_recall_at_100_diff1": 0.09178275311751917,
+              "nauc_recall_at_1000_max": -0.20168627852829185,
+              "nauc_recall_at_1000_std": 0.42587214313005395,
+              "nauc_recall_at_1000_diff1": 0.18865741402834005,
+              "nauc_precision_at_1_max": 0.19046090779456812,
+              "nauc_precision_at_1_std": 0.2005678293225855,
+              "nauc_precision_at_1_diff1": 0.2732613145926889,
+              "nauc_precision_at_3_max": 0.17065346765787537,
+              "nauc_precision_at_3_std": 0.2580216755377953,
+              "nauc_precision_at_3_diff1": 0.15208690889778514,
+              "nauc_precision_at_5_max": 0.15966584831210248,
+              "nauc_precision_at_5_std": 0.29340982416875505,
+              "nauc_precision_at_5_diff1": 0.13560040963190081,
+              "nauc_precision_at_10_max": 0.13047182792970813,
+              "nauc_precision_at_10_std": 0.3126115268876884,
+              "nauc_precision_at_10_diff1": 0.11276306288171808,
+              "nauc_precision_at_20_max": 0.09951766493736443,
+              "nauc_precision_at_20_std": 0.35593195934746613,
+              "nauc_precision_at_20_diff1": 0.10597363396032576,
+              "nauc_precision_at_100_max": 0.04010877959852337,
+              "nauc_precision_at_100_std": 0.378379077826633,
+              "nauc_precision_at_100_diff1": 0.0922392067579821,
+              "nauc_precision_at_1000_max": -0.044271818009240606,
+              "nauc_precision_at_1000_std": 0.29888109273282804,
+              "nauc_precision_at_1000_diff1": 0.09766770636218437,
+              "nauc_mrr_at_1_max": 0.19046090779456812,
+              "nauc_mrr_at_1_std": 0.2005678293225855,
+              "nauc_mrr_at_1_diff1": 0.2732613145926889,
+              "nauc_mrr_at_3_max": 0.16211133977600667,
+              "nauc_mrr_at_3_std": 0.22287755102083415,
+              "nauc_mrr_at_3_diff1": 0.20847534169317564,
+              "nauc_mrr_at_5_max": 0.15334494533630705,
+              "nauc_mrr_at_5_std": 0.2342332780831409,
+              "nauc_mrr_at_5_diff1": 0.19804894502376114,
+              "nauc_mrr_at_10_max": 0.1440542234453121,
+              "nauc_mrr_at_10_std": 0.240522209249698,
+              "nauc_mrr_at_10_diff1": 0.1893144127381246,
+              "nauc_mrr_at_20_max": 0.13965220841407827,
+              "nauc_mrr_at_20_std": 0.246658048198982,
+              "nauc_mrr_at_20_diff1": 0.18779493799431327,
+              "nauc_mrr_at_100_max": 0.1352372575762705,
+              "nauc_mrr_at_100_std": 0.24993578620339646,
+              "nauc_mrr_at_100_diff1": 0.1864436719961944,
+              "nauc_mrr_at_1000_max": 0.13462182002208006,
+              "nauc_mrr_at_1000_std": 0.25008212756311865,
+              "nauc_mrr_at_1000_diff1": 0.18711915110093033,
+              "hit_rate_at_1": 0.06139,
+              "hit_rate_at_3": 0.11254,
+              "hit_rate_at_5": 0.14339,
+              "hit_rate_at_10": 0.19024,
+              "hit_rate_at_20": 0.24763,
+              "hit_rate_at_100": 0.41429,
+              "hit_rate_at_1000": 0.78455,
+              "main_score": 0.10331,
+              "hf_subset": "tr",
+              "languages": [
+                "tur-Latn"
+              ],
+              "mteb_version": "2.18.16"
+            }
+          ]
+        },
+        "evaluation_time": 25.021837311098352,
+        "kg_co2_emissions": null,
+        "date": "2026-08-11T08:43:34.083720Z",
+        "evaluation_phases": [
+          {
+            "name": "Data loading",
+            "start": 4.032161086797714e-06,
+            "end": 6.687524263979867,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Dataset transform",
+            "start": 6.687531795119867,
+            "end": 6.6875344440341,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Encoding queries and documents",
+            "start": 6.735912636155263,
+            "end": 21.24591747415252,
+            "split": "train",
+            "subset": "tr"
+          },
+          {
+            "name": "Scoring",
+            "start": 21.245933528989553,
+            "end": 24.96005431120284,
+            "split": "train",
+            "subset": "tr"
+          }
+        ]
+      },
+      {
+        "dataset_revision": "979a211276faa22f671e69d096634193567cfd05",
+        "task_name": "BelebeleRetrieval",
+        "mteb_version": "2.18.16",
+        "scores": {
+          "test": [
+            {
+              "ndcg_at_1": 0.83778,
+              "ndcg_at_3": 0.8729,
+              "ndcg_at_5": 0.8793,
+              "ndcg_at_10": 0.88493,
+              "ndcg_at_20": 0.89217,
+              "ndcg_at_100": 0.89819,
+              "ndcg_at_1000": 0.89954,
+              "map_at_1": 0.83778,
+              "map_at_3": 0.86463,
+              "map_at_5": 0.86819,
+              "map_at_10": 0.87044,
+              "map_at_20": 0.87239,
+              "map_at_100": 0.87331,
+              "map_at_1000": 0.87337,
+              "recall_at_1": 0.83778,
+              "recall_at_3": 0.89667,
+              "recall_at_5": 0.91222,
+              "recall_at_10": 0.93,
+              "recall_at_20": 0.95889,
+              "recall_at_100": 0.99,
+              "recall_at_1000": 1.0,
+              "accuracy": 0.83778,
+              "precision_at_1": 0.83778,
+              "precision_at_3": 0.29889,
+              "precision_at_5": 0.18244,
+              "precision_at_10": 0.093,
+              "precision_at_20": 0.04794,
+              "precision_at_100": 0.0099,
+              "precision_at_1000": 0.001,
+              "mrr_at_1": 0.8377777777777777,
+              "mrr_at_3": 0.8646296296296296,
+              "mrr_at_5": 0.8676296296296296,
+              "mrr_at_10": 0.8702103174603174,
+              "mrr_at_20": 0.8720055372139995,
+              "mrr_at_100": 0.8729395151413755,
+              "mrr_at_1000": 0.8730008378211206,
+              "nauc_ndcg_at_1_max": 0.8410244670966699,
+              "nauc_ndcg_at_1_std": 0.06830117285862436,
+              "nauc_ndcg_at_1_diff1": 0.920135195184682,
+              "nauc_ndcg_at_3_max": 0.8482741017353032,
+              "nauc_ndcg_at_3_std": 0.10547749274165154,
+              "nauc_ndcg_at_3_diff1": 0.8954013102092868,
+              "nauc_ndcg_at_5_max": 0.8494532972001592,
+              "nauc_ndcg_at_5_std": 0.12742254228373348,
+              "nauc_ndcg_at_5_diff1": 0.8968040492636123,
+              "nauc_ndcg_at_10_max": 0.8533332736310034,
+              "nauc_ndcg_at_10_std": 0.149236559008684,
+              "nauc_ndcg_at_10_diff1": 0.8992038888020082,
+              "nauc_ndcg_at_20_max": 0.851087799934009,
+              "nauc_ndcg_at_20_std": 0.1380363533185576,
+              "nauc_ndcg_at_20_diff1": 0.8999492972758715,
+              "nauc_ndcg_at_100_max": 0.8513927339831183,
+              "nauc_ndcg_at_100_std": 0.12804271152305366,
+              "nauc_ndcg_at_100_diff1": 0.9025952980091414,
+              "nauc_ndcg_at_1000_max": 0.8503838452506185,
+              "nauc_ndcg_at_1000_std": 0.12168496863829963,
+              "nauc_ndcg_at_1000_diff1": 0.9025192818485014,
+              "nauc_map_at_1_max": 0.8410244670966699,
+              "nauc_map_at_1_std": 0.06830117285862436,
+              "nauc_map_at_1_diff1": 0.920135195184682,
+              "nauc_map_at_3_max": 0.8472124322730213,
+              "nauc_map_at_3_std": 0.09524294429360917,
+              "nauc_map_at_3_diff1": 0.9024453786958063,
+              "nauc_map_at_5_max": 0.8476935399614086,
+              "nauc_map_at_5_std": 0.10596944315332903,
+              "nauc_map_at_5_diff1": 0.9032210546789997,
+              "nauc_map_at_10_max": 0.8491278258232463,
+              "nauc_map_at_10_std": 0.1139024265659601,
+              "nauc_map_at_10_diff1": 0.904189158426485,
+              "nauc_map_at_20_max": 0.8486364332677006,
+              "nauc_map_at_20_std": 0.11146918219618034,
+              "nauc_map_at_20_diff1": 0.9044222430772109,
+              "nauc_map_at_100_max": 0.8487478052591781,
+              "nauc_map_at_100_std": 0.10976937220975148,
+              "nauc_map_at_100_diff1": 0.9047515896507614,
+              "nauc_map_at_1000_max": 0.8487193359313112,
+              "nauc_map_at_1000_std": 0.10956032718263647,
+              "nauc_map_at_1000_diff1": 0.9047484933605456,
+              "nauc_recall_at_1_max": 0.8410244670966699,
+              "nauc_recall_at_1_std": 0.06830117285862436,
+              "nauc_recall_at_1_diff1": 0.920135195184682,
+              "nauc_recall_at_3_max": 0.851642922678282,
+              "nauc_recall_at_3_std": 0.14283193692467533,
+              "nauc_recall_at_3_diff1": 0.8695643406632181,
+              "nauc_recall_at_5_max": 0.8566109988299108,
+              "nauc_recall_at_5_std": 0.22182037371910784,
+              "nauc_recall_at_5_diff1": 0.8704629531137352,
+              "nauc_recall_at_10_max": 0.8772175536881434,
+              "nauc_recall_at_10_std": 0.3506513716597738,
+              "nauc_recall_at_10_diff1": 0.8759133282942827,
+              "nauc_recall_at_20_max": 0.8692305751129306,
+              "nauc_recall_at_20_std": 0.36314886314886435,
+              "nauc_recall_at_20_diff1": 0.867451485098547,
+              "nauc_recall_at_100_max": 0.9286751737731979,
+              "nauc_recall_at_100_std": 0.6017221703496162,
+              "nauc_recall_at_100_diff1": 0.9074074074074023,
+              "nauc_recall_at_1000_max": null,
+              "nauc_recall_at_1000_std": null,
+              "nauc_recall_at_1000_diff1": null,
+              "nauc_precision_at_1_max": 0.8410244670966699,
+              "nauc_precision_at_1_std": 0.06830117285862436,
+              "nauc_precision_at_1_diff1": 0.920135195184682,
+              "nauc_precision_at_3_max": 0.8516429226782833,
+              "nauc_precision_at_3_std": 0.14283193692467333,
+              "nauc_precision_at_3_diff1": 0.8695643406632189,
+              "nauc_precision_at_5_max": 0.8566109988299109,
+              "nauc_precision_at_5_std": 0.22182037371910543,
+              "nauc_precision_at_5_diff1": 0.8704629531137342,
+              "nauc_precision_at_10_max": 0.8772175536881401,
+              "nauc_precision_at_10_std": 0.35065137165977484,
+              "nauc_precision_at_10_diff1": 0.8759133282942804,
+              "nauc_precision_at_20_max": 0.8692305751129216,
+              "nauc_precision_at_20_std": 0.3631488631488554,
+              "nauc_precision_at_20_diff1": 0.8674514850985429,
+              "nauc_precision_at_100_max": 0.928675173773221,
+              "nauc_precision_at_100_std": 0.6017221703496362,
+              "nauc_precision_at_100_diff1": 0.9074074074074131,
+              "nauc_precision_at_1000_max": null,
+              "nauc_precision_at_1000_std": null,
+              "nauc_precision_at_1000_diff1": null,
+              "nauc_mrr_at_1_max": 0.8410244670966699,
+              "nauc_mrr_at_1_std": 0.06830117285862436,
+              "nauc_mrr_at_1_diff1": 0.920135195184682,
+              "nauc_mrr_at_3_max": 0.8472124322730213,
+              "nauc_mrr_at_3_std": 0.09524294429360917,
+              "nauc_mrr_at_3_diff1": 0.9024453786958063,
+              "nauc_mrr_at_5_max": 0.8483553382502018,
+              "nauc_mrr_at_5_std": 0.10985416276732428,
+              "nauc_mrr_at_5_diff1": 0.9036415761748603,
+              "nauc_mrr_at_10_max": 0.8495478126860134,
+              "nauc_mrr_at_10_std": 0.1158882768620596,
+              "nauc_mrr_at_10_diff1": 0.9052043399121691,
+              "nauc_mrr_at_20_max": 0.8494438987722414,
+              "nauc_mrr_at_20_std": 0.11432323610147117,
+              "nauc_mrr_at_20_diff1": 0.9048993393892497,
+              "nauc_mrr_at_100_max": 0.8494035670115612,
+              "nauc_mrr_at_100_std": 0.11245996264042155,
+              "nauc_mrr_at_100_diff1": 0.9053150039850627,
+              "nauc_mrr_at_1000_max": 0.8493755127679687,
+              "nauc_mrr_at_1000_std": 0.11225290183205333,
+              "nauc_mrr_at_1000_diff1": 0.9053121992092529,
+              "hit_rate_at_1": 0.83778,
+              "hit_rate_at_3": 0.89667,
+              "hit_rate_at_5": 0.91222,
+              "hit_rate_at_10": 0.93,
+              "hit_rate_at_20": 0.95889,
+              "hit_rate_at_100": 0.99,
+              "hit_rate_at_1000": 1.0,
+              "main_score": 0.88493,
+              "hf_subset": "tur_Latn-tur_Latn",
+              "languages": [
+                "tur-Latn",
+                "tur-Latn"
+              ],
+              "mteb_version": "2.18.16"
+            }
+          ]
+        },
+        "evaluation_time": 4.633956309873611,
+        "kg_co2_emissions": null,
+        "date": "2026-08-11T08:43:38.719847Z",
+        "evaluation_phases": [
+          {
+            "name": "Data loading",
+            "start": 0.0,
+            "end": 1.653581827878952,
+            "split": "",
+            "subset": ""
+          },
+          {
+            "name": "Encoding queries and documents",
+            "start": 1.6664734289515764,
+            "end": 4.349166231928393,
+            "split": "test",
+            "subset": "tur_Latn-tur_Latn"
+          },
+          {
+            "name": "Scoring",
+            "start": 4.349181247875094,
+            "end": 4.6296024329494685,
+            "split": "test",
+            "subset": "tur_Latn-tur_Latn"
+          }
+        ]
+      }
+    ],
+    "exceptions": [],
+    "experiment_name": null
+  }
+}

results/suite-comparison.json

@@ -0,0 +1,46 @@
+[
+  {
+    "model": "multilingual E5 base",
+    "parameters": 278043648,
+    "embedding_dimension": 768,
+    "TurHistQuadRetrieval": 0.49726,
+    "XQuADRetrieval": 0.95335,
+    "WebFAQRetrieval": 0.65032,
+    "MKQARetrieval": 0.07213,
+    "BelebeleRetrieval": 0.92503,
+    "macro_average": 0.619618
+  },
+  {
+    "model": "GöktuğTR 270M v2 hard negatives",
+    "parameters": 268098176,
+    "embedding_dimension": 640,
+    "TurHistQuadRetrieval": 0.42198,
+    "XQuADRetrieval": 0.86393,
+    "WebFAQRetrieval": 0.56886,
+    "MKQARetrieval": 0.10331,
+    "BelebeleRetrieval": 0.88493,
+    "macro_average": 0.568602
+  },
+  {
+    "model": "GöktuğTR 270M v1",
+    "parameters": 268098176,
+    "embedding_dimension": 640,
+    "TurHistQuadRetrieval": 0.42196,
+    "XQuADRetrieval": 0.85832,
+    "WebFAQRetrieval": 0.56402,
+    "MKQARetrieval": 0.10296,
+    "BelebeleRetrieval": 0.88222,
+    "macro_average": 0.565896
+  },
+  {
+    "model": "GöktuğTR 118M compact",
+    "parameters": 117653760,
+    "embedding_dimension": 384,
+    "TurHistQuadRetrieval": 0.25299,
+    "XQuADRetrieval": 0.81123,
+    "WebFAQRetrieval": 0.46307,
+    "MKQARetrieval": 0.04855,
+    "BelebeleRetrieval": 0.82451,
+    "macro_average": 0.48007
+  }
+]

results/suite-manifest.json

@@ -0,0 +1,53 @@
+{
+  "schema": "goktugtr-turkish-retrieval-suite-v1",
+  "evaluator": "mteb==2.18.16",
+  "language_filter": "Turkish-only exclusive subsets",
+  "normalized_embeddings": true,
+  "tasks": [
+    {
+      "name": "TurHistQuadRetrieval",
+      "dataset": "asparius/TurHistQuAD",
+      "revision": "2a2b8ddecf1189f530676244d0751e1d0a569e03",
+      "subset": "default",
+      "split": "test",
+      "license": "mit",
+      "domain": "Ottoman history / encyclopaedic QA"
+    },
+    {
+      "name": "XQuADRetrieval",
+      "dataset": "google/xquad",
+      "revision": "51adfef1c1287aab1d2d91b5bead9bcfb9c68583",
+      "subset": "tr",
+      "split": "validation",
+      "license": "cc-by-sa-4.0",
+      "domain": "reading comprehension"
+    },
+    {
+      "name": "WebFAQRetrieval",
+      "dataset": "mteb/WebFAQRetrieval",
+      "revision": "f64f483ad0f31d2e78209d524c14a4a867965959",
+      "subset": "tur",
+      "split": "test",
+      "license": "cc-by-4.0",
+      "domain": "web FAQ"
+    },
+    {
+      "name": "MKQARetrieval",
+      "dataset": "mteb/MKQARetrieval",
+      "revision": "3e069e7a30079d214a859741f5a0de75cf878867",
+      "subset": "tr",
+      "split": "train",
+      "license": "cc-by-3.0",
+      "domain": "open-domain knowledge QA"
+    },
+    {
+      "name": "BelebeleRetrieval",
+      "dataset": "mteb/belebele",
+      "revision": "979a211276faa22f671e69d096634193567cfd05",
+      "subset": "tur_Latn-tur_Latn",
+      "split": "test",
+      "license": "cc-by-sa-4.0",
+      "domain": "multilingual reading comprehension"
+    }
+  ]
+}

results/trainer-state.json

@@ -0,0 +1,652 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 125,
+  "global_step": 782,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0128,
+      "grad_norm": 6.78125,
+      "learning_rate": 1.8e-06,
+      "loss": 1.1243257522583008,
+      "step": 10
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 5.125,
+      "learning_rate": 3.7999999999999996e-06,
+      "loss": 1.127305030822754,
+      "step": 20
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 6.625,
+      "learning_rate": 5.7999999999999995e-06,
+      "loss": 0.9744441986083985,
+      "step": 30
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 4.25,
+      "learning_rate": 7.8e-06,
+      "loss": 1.1323395729064942,
+      "step": 40
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 10.3125,
+      "learning_rate": 7.902964959568732e-06,
+      "loss": 1.4813992500305175,
+      "step": 50
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 6.34375,
+      "learning_rate": 7.795148247978437e-06,
+      "loss": 0.995882797241211,
+      "step": 60
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 5.125,
+      "learning_rate": 7.68733153638814e-06,
+      "loss": 0.701082181930542,
+      "step": 70
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 4.59375,
+      "learning_rate": 7.579514824797844e-06,
+      "loss": 1.2655192375183106,
+      "step": 80
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 6.15625,
+      "learning_rate": 7.471698113207547e-06,
+      "loss": 1.1107177734375,
+      "step": 90
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 5.40625,
+      "learning_rate": 7.36388140161725e-06,
+      "loss": 0.9124648094177246,
+      "step": 100
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 6.03125,
+      "learning_rate": 7.256064690026954e-06,
+      "loss": 0.934360408782959,
+      "step": 110
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 5.84375,
+      "learning_rate": 7.148247978436657e-06,
+      "loss": 0.8691039085388184,
+      "step": 120
+    },
+    {
+      "epoch": 0.16,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8939999938011169,
+      "eval_loss": 0.7023630142211914,
+      "eval_runtime": 26.3381,
+      "eval_samples_per_second": 75.936,
+      "eval_steps_per_second": 9.492,
+      "step": 125
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 4.25,
+      "learning_rate": 7.0404312668463605e-06,
+      "loss": 0.7453449726104736,
+      "step": 130
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 5.1875,
+      "learning_rate": 6.932614555256064e-06,
+      "loss": 1.0471577644348145,
+      "step": 140
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 7.09375,
+      "learning_rate": 6.824797843665768e-06,
+      "loss": 0.8312005043029785,
+      "step": 150
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 6.59375,
+      "learning_rate": 6.7169811320754715e-06,
+      "loss": 0.978731632232666,
+      "step": 160
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 4.4375,
+      "learning_rate": 6.609164420485175e-06,
+      "loss": 1.5524386405944823,
+      "step": 170
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 5.5,
+      "learning_rate": 6.501347708894878e-06,
+      "loss": 0.9334232330322265,
+      "step": 180
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 6.03125,
+      "learning_rate": 6.3935309973045816e-06,
+      "loss": 0.9607770919799805,
+      "step": 190
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 6.0625,
+      "learning_rate": 6.285714285714285e-06,
+      "loss": 0.7139860630035401,
+      "step": 200
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 5.1875,
+      "learning_rate": 6.177897574123988e-06,
+      "loss": 0.9790145874023437,
+      "step": 210
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 10.6875,
+      "learning_rate": 6.0700808625336925e-06,
+      "loss": 0.8579249382019043,
+      "step": 220
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 6.125,
+      "learning_rate": 5.962264150943396e-06,
+      "loss": 0.7599565505981445,
+      "step": 230
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 4.21875,
+      "learning_rate": 5.8544474393531e-06,
+      "loss": 0.6702170848846436,
+      "step": 240
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 6.90625,
+      "learning_rate": 5.7466307277628035e-06,
+      "loss": 0.7464066505432129,
+      "step": 250
+    },
+    {
+      "epoch": 0.32,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8930000066757202,
+      "eval_loss": 0.6991313695907593,
+      "eval_runtime": 26.5563,
+      "eval_samples_per_second": 75.312,
+      "eval_steps_per_second": 9.414,
+      "step": 250
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 5.1875,
+      "learning_rate": 5.638814016172507e-06,
+      "loss": 1.2767579078674316,
+      "step": 260
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 6.28125,
+      "learning_rate": 5.53099730458221e-06,
+      "loss": 0.7426451683044434,
+      "step": 270
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 11.875,
+      "learning_rate": 5.423180592991914e-06,
+      "loss": 1.1208632469177247,
+      "step": 280
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 5.96875,
+      "learning_rate": 5.315363881401617e-06,
+      "loss": 0.645590353012085,
+      "step": 290
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 5.53125,
+      "learning_rate": 5.20754716981132e-06,
+      "loss": 0.7049435138702392,
+      "step": 300
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 6.75,
+      "learning_rate": 5.099730458221024e-06,
+      "loss": 0.6756068706512451,
+      "step": 310
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 5.34375,
+      "learning_rate": 4.991913746630728e-06,
+      "loss": 0.7691781044006347,
+      "step": 320
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 6.15625,
+      "learning_rate": 4.884097035040431e-06,
+      "loss": 0.7604856967926026,
+      "step": 330
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 10.875,
+      "learning_rate": 4.776280323450135e-06,
+      "loss": 1.0698866844177246,
+      "step": 340
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 10.3125,
+      "learning_rate": 4.668463611859838e-06,
+      "loss": 0.8213407516479492,
+      "step": 350
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 11.6875,
+      "learning_rate": 4.5606469002695414e-06,
+      "loss": 0.7953258991241455,
+      "step": 360
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 12.0,
+      "learning_rate": 4.452830188679245e-06,
+      "loss": 0.8094965934753418,
+      "step": 370
+    },
+    {
+      "epoch": 0.48,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8934999704360962,
+      "eval_loss": 0.69776850938797,
+      "eval_runtime": 26.4093,
+      "eval_samples_per_second": 75.731,
+      "eval_steps_per_second": 9.466,
+      "step": 375
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 5.1875,
+      "learning_rate": 4.345013477088948e-06,
+      "loss": 0.7656964778900146,
+      "step": 380
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 4.6875,
+      "learning_rate": 4.2371967654986516e-06,
+      "loss": 0.6850308418273926,
+      "step": 390
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 5.09375,
+      "learning_rate": 4.129380053908355e-06,
+      "loss": 0.967844295501709,
+      "step": 400
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 6.6875,
+      "learning_rate": 4.021563342318059e-06,
+      "loss": 0.9944411277770996,
+      "step": 410
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 7.40625,
+      "learning_rate": 3.9137466307277625e-06,
+      "loss": 1.2628602981567383,
+      "step": 420
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 12.125,
+      "learning_rate": 3.8059299191374663e-06,
+      "loss": 0.7692550182342529,
+      "step": 430
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 4.34375,
+      "learning_rate": 3.6981132075471697e-06,
+      "loss": 0.6217018604278565,
+      "step": 440
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 4.1875,
+      "learning_rate": 3.590296495956873e-06,
+      "loss": 0.7464039325714111,
+      "step": 450
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 4.875,
+      "learning_rate": 3.4824797843665764e-06,
+      "loss": 0.8691292762756347,
+      "step": 460
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 5.1875,
+      "learning_rate": 3.3746630727762802e-06,
+      "loss": 0.6637392044067383,
+      "step": 470
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 4.75,
+      "learning_rate": 3.2668463611859836e-06,
+      "loss": 1.0880215644836426,
+      "step": 480
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 4.5625,
+      "learning_rate": 3.1590296495956874e-06,
+      "loss": 0.7366327285766602,
+      "step": 490
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 5.46875,
+      "learning_rate": 3.0512129380053908e-06,
+      "loss": 0.8034601211547852,
+      "step": 500
+    },
+    {
+      "epoch": 0.64,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8930000066757202,
+      "eval_loss": 0.6969029307365417,
+      "eval_runtime": 26.2197,
+      "eval_samples_per_second": 76.279,
+      "eval_steps_per_second": 9.535,
+      "step": 500
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 5.28125,
+      "learning_rate": 2.943396226415094e-06,
+      "loss": 0.7920682907104493,
+      "step": 510
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 4.96875,
+      "learning_rate": 2.835579514824798e-06,
+      "loss": 0.8281051635742187,
+      "step": 520
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 5.25,
+      "learning_rate": 2.7277628032345013e-06,
+      "loss": 0.9947348594665527,
+      "step": 530
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 4.6875,
+      "learning_rate": 2.6199460916442047e-06,
+      "loss": 0.9334335327148438,
+      "step": 540
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 5.5,
+      "learning_rate": 2.512129380053908e-06,
+      "loss": 1.0346774101257323,
+      "step": 550
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 5.15625,
+      "learning_rate": 2.404312668463612e-06,
+      "loss": 0.9480579376220704,
+      "step": 560
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 5.625,
+      "learning_rate": 2.2964959568733152e-06,
+      "loss": 0.6826355457305908,
+      "step": 570
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 5.25,
+      "learning_rate": 2.1886792452830186e-06,
+      "loss": 1.3600321769714356,
+      "step": 580
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 5.0625,
+      "learning_rate": 2.080862533692722e-06,
+      "loss": 0.8289908409118653,
+      "step": 590
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 6.5625,
+      "learning_rate": 1.9730458221024258e-06,
+      "loss": 0.9872503280639648,
+      "step": 600
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 5.40625,
+      "learning_rate": 1.8652291105121293e-06,
+      "loss": 0.7281249046325684,
+      "step": 610
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 5.1875,
+      "learning_rate": 1.7574123989218327e-06,
+      "loss": 0.9853657722473145,
+      "step": 620
+    },
+    {
+      "epoch": 0.8,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8939999938011169,
+      "eval_loss": 0.6965240836143494,
+      "eval_runtime": 26.199,
+      "eval_samples_per_second": 76.339,
+      "eval_steps_per_second": 9.542,
+      "step": 625
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 4.53125,
+      "learning_rate": 1.6495956873315365e-06,
+      "loss": 0.8656760215759277,
+      "step": 630
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 5.875,
+      "learning_rate": 1.5417789757412399e-06,
+      "loss": 1.5203048706054687,
+      "step": 640
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 5.65625,
+      "learning_rate": 1.4339622641509432e-06,
+      "loss": 0.9369791984558106,
+      "step": 650
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 5.03125,
+      "learning_rate": 1.3261455525606468e-06,
+      "loss": 0.7103929042816162,
+      "step": 660
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 6.875,
+      "learning_rate": 1.2183288409703502e-06,
+      "loss": 0.9066259384155273,
+      "step": 670
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 5.1875,
+      "learning_rate": 1.110512129380054e-06,
+      "loss": 0.7295275688171386,
+      "step": 680
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 5.25,
+      "learning_rate": 1.0026954177897574e-06,
+      "loss": 0.8187677383422851,
+      "step": 690
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 5.84375,
+      "learning_rate": 8.94878706199461e-07,
+      "loss": 0.764503288269043,
+      "step": 700
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 4.875,
+      "learning_rate": 7.870619946091643e-07,
+      "loss": 0.8578934669494629,
+      "step": 710
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 0.07373046875,
+      "learning_rate": 6.792452830188678e-07,
+      "loss": 0.716743278503418,
+      "step": 720
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 0.010498046875,
+      "learning_rate": 5.714285714285714e-07,
+      "loss": 0.0003363490104675293,
+      "step": 730
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 0.0118408203125,
+      "learning_rate": 4.6361185983827487e-07,
+      "loss": 0.00033761262893676757,
+      "step": 740
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.361328125,
+      "learning_rate": 3.557951482479784e-07,
+      "loss": 0.0005402326583862305,
+      "step": 750
+    },
+    {
+      "epoch": 0.96,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8924999833106995,
+      "eval_loss": 0.6961512565612793,
+      "eval_runtime": 26.8581,
+      "eval_samples_per_second": 74.466,
+      "eval_steps_per_second": 9.308,
+      "step": 750
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.00182342529296875,
+      "learning_rate": 2.4797843665768193e-07,
+      "loss": 0.0005559921264648438,
+      "step": 760
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.037841796875,
+      "learning_rate": 1.4016172506738543e-07,
+      "loss": 0.0004771590232849121,
+      "step": 770
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 0.0019989013671875,
+      "learning_rate": 3.234501347708895e-08,
+      "loss": 0.002035796642303467,
+      "step": 780
+    },
+    {
+      "epoch": 1.0,
+      "eval_goktugtr-validation_cosine_accuracy": 0.8934999704360962,
+      "eval_loss": 0.69606614112854,
+      "eval_runtime": 27.0611,
+      "eval_samples_per_second": 73.907,
+      "eval_steps_per_second": 9.238,
+      "step": 782
+    },
+    {
+      "epoch": 1.0,
+      "step": 782,
+      "total_flos": 0.0,
+      "train_loss": 0.8376862271057676,
+      "train_runtime": 2528.4185,
+      "train_samples_per_second": 19.775,
+      "train_steps_per_second": 0.309
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 782,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 125,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

sentence_bert_config.json

@@ -0,0 +1,10 @@
+{
+    "transformer_task": "feature-extraction",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        }
+    },
+    "module_output_name": "token_embeddings"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd5f1ae5d7b01aadec7b28edcb1b218ff7ac3b73345cd73b40f463c9c3bab1a4
+size 33385259

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "max_length": 256,
+  "model_max_length": 256,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "left",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "stride": 0,
+  "tokenizer_class": "GemmaTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

training/evaluate_suite.py

@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+"""Run a pinned five-task Turkish retrieval suite with official MTEB evaluators."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import mteb
+import torch
+from mteb.models import SentenceTransformerEncoderWrapper
+from sentence_transformers import SentenceTransformer
+
+from goktugtr.text import HARIER_TASK
+
+TASK_NAMES = [
+    "TurHistQuadRetrieval",
+    "XQuADRetrieval",
+    "WebFAQRetrieval",
+    "MKQARetrieval",
+    "BelebeleRetrieval",
+]
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", required=True)
+    parser.add_argument("--revision")
+    parser.add_argument("--label")
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--batch-size", type=int, default=16)
+    parser.add_argument("--max-seq-length", type=int, default=512)
+    parser.add_argument(
+        "--prompt-style",
+        choices=["auto", "plain", "harrier", "e5"],
+        default="auto",
+        help="Select the query/document format explicitly for fair local-model evaluation.",
+    )
+    parser.add_argument("--tasks", nargs="+", choices=TASK_NAMES, default=TASK_NAMES)
+    parser.add_argument("--overwrite", action="store_true")
+    return parser.parse_args()
+
+
+def prompts_for(model_id: str, style: str = "auto") -> dict[str, str] | None:
+    if style == "plain":
+        return None
+    if style == "harrier":
+        return {
+            "Retrieval-query": f"Instruct: {HARIER_TASK}\nQuery: ",
+            "Retrieval-document": "",
+        }
+    if style == "e5":
+        return {"Retrieval-query": "query: ", "Retrieval-document": "passage: "}
+    normalized = model_id.casefold()
+    if "harrier" in normalized or "goktugtr" in normalized:
+        return {
+            "Retrieval-query": f"Instruct: {HARIER_TASK}\nQuery: ",
+            "Retrieval-document": "",
+        }
+    if "e5" in normalized:
+        return {"Retrieval-query": "query: ", "Retrieval-document": "passage: "}
+    return None
+
+
+def build_tasks(task_names: list[str] | None = None) -> list:
+    selected = task_names or TASK_NAMES
+    return [
+        mteb.get_task(
+            task_name=name,
+            languages=["tur"],
+            exclusive_language_filter=True,
+        )
+        for name in selected
+    ]
+
+
+def main() -> None:
+    args = parse_args()
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    tasks = build_tasks(args.tasks)
+    model = SentenceTransformer(
+        args.model,
+        revision=args.revision,
+        model_kwargs={"dtype": torch.bfloat16},
+    )
+    model.max_seq_length = args.max_seq_length
+    parameter_count = sum(parameter.numel() for parameter in model.parameters())
+    wrapper = SentenceTransformerEncoderWrapper(
+        model=model,
+        model_prompts=prompts_for(args.model, args.prompt_style),
+    )
+    result = mteb.evaluate(
+        wrapper,
+        tasks,
+        cache=None,
+        overwrite_strategy="always" if args.overwrite else "only-missing",
+        encode_kwargs={
+            "batch_size": args.batch_size,
+            "normalize_embeddings": True,
+        },
+        show_progress_bar=True,
+        co2_tracker=False,
+        public_only=True,
+    )
+    task_scores = {}
+    for task_result in result.task_results:
+        task_scores[task_result.task_name] = float(task_result.get_score())
+    task_metadata = []
+    for task in tasks:
+        metadata = task.metadata.model_dump(mode="json")
+        task_metadata.append(
+            {
+                "name": metadata["name"],
+                "dataset": metadata["dataset"],
+                "license": metadata["license"],
+                "domains": metadata["domains"],
+                "eval_splits": metadata["eval_splits"],
+                "subsets": list(task.hf_subsets),
+            }
+        )
+    payload = {
+        "suite": "goktugtr-turkish-retrieval-suite-v1",
+        "language_filter": "turkish_only_exclusive",
+        "model": args.label or args.model,
+        "model_source": args.model,
+        "model_revision": args.revision,
+        "parameters": parameter_count,
+        "embedding_dimension": model.get_sentence_embedding_dimension(),
+        "inference_dtype": "bfloat16",
+        "normalized_embeddings": True,
+        "prompt_style": args.prompt_style,
+        "device": torch.cuda.get_device_name(0) if torch.cuda.is_available() else "cpu",
+        "mteb_version": mteb.__version__,
+        "task_main_scores": task_scores,
+        "macro_average": sum(task_scores.values()) / len(task_scores),
+        "tasks": task_metadata,
+        "raw_mteb_result": result.model_dump(mode="json"),
+    }
+    args.output.write_text(
+        json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+    )
+    print(
+        json.dumps(
+            {k: v for k, v in payload.items() if k != "raw_mteb_result"},
+            ensure_ascii=False,
+            indent=2,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

training/mine_hard_negatives.py

@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+"""Mine difficult, model-selected negatives without touching benchmark data."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import time
+from pathlib import Path
+
+import faiss
+import numpy as np
+import torch
+from datasets import Dataset, load_dataset
+from sentence_transformers import SentenceTransformer
+
+from goktugtr.text import training_query
+
+TOKEN_RE = re.compile(r"\w+", re.UNICODE)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", default="outputs/goktugtr-270m/final")
+    parser.add_argument("--train", type=Path, default=Path("data/processed/train.parquet"))
+    parser.add_argument(
+        "--validation", type=Path, default=Path("data/processed/validation.parquet")
+    )
+    parser.add_argument("--output-dir", type=Path, default=Path("data/hard-negatives-v1"))
+    parser.add_argument("--max-rows", type=int, default=50_000)
+    parser.add_argument("--candidate-rows", type=int, default=100_000)
+    parser.add_argument("--batch-size", type=int, default=32)
+    parser.add_argument("--search-k", type=int, default=32)
+    parser.add_argument("--query-style", choices=["harrier", "e5", "plain"], default="harrier")
+    parser.add_argument("--max-positive-overlap", type=float, default=0.80)
+    return parser.parse_args()
+
+
+def normalized(text: str) -> str:
+    return " ".join(text.casefold().split())
+
+
+def token_jaccard(first: str, second: str) -> float:
+    left = set(TOKEN_RE.findall(first.casefold()))
+    right = set(TOKEN_RE.findall(second.casefold()))
+    if not left and not right:
+        return 1.0
+    return len(left & right) / max(1, len(left | right))
+
+
+def select_hard_negative(
+    positive: str,
+    original_negative: str,
+    candidates: list[str],
+    candidate_indices: list[int],
+    scores: list[float],
+    *,
+    max_positive_overlap: float,
+) -> tuple[str, float | None, int | None]:
+    """Pick the highest-ranked plausible negative, falling back to the labeled one."""
+    positive_norm = normalized(positive)
+    for rank, (candidate_index, score) in enumerate(
+        zip(candidate_indices, scores, strict=True), start=1
+    ):
+        candidate = candidates[candidate_index]
+        candidate_norm = normalized(candidate)
+        if not candidate_norm or candidate_norm == positive_norm:
+            continue
+        if token_jaccard(positive, candidate) >= max_positive_overlap:
+            continue
+        return candidate, float(score), rank
+    return original_negative, None, None
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def main() -> None:
+    args = parse_args()
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    full_train = load_dataset("parquet", data_files=str(args.train), split="train")
+    mining_rows = full_train.select(range(min(args.max_rows, len(full_train))))
+    candidate_rows = full_train.select(range(min(args.candidate_rows, len(full_train))))
+    candidates = list(dict.fromkeys(candidate_rows["negative"]))
+
+    model = SentenceTransformer(args.model, model_kwargs={"dtype": torch.bfloat16})
+    model.max_seq_length = 256
+    queries = [training_query(text, args.query_style) for text in mining_rows["query"]]
+
+    started = time.perf_counter()
+    candidate_embeddings = model.encode(
+        candidates,
+        batch_size=args.batch_size,
+        normalize_embeddings=True,
+        convert_to_numpy=True,
+        show_progress_bar=True,
+    ).astype("float32")
+    query_embeddings = model.encode(
+        queries,
+        batch_size=args.batch_size,
+        normalize_embeddings=True,
+        convert_to_numpy=True,
+        show_progress_bar=True,
+    ).astype("float32")
+
+    index = faiss.IndexHNSWFlat(candidate_embeddings.shape[1], 32, faiss.METRIC_INNER_PRODUCT)
+    index.hnsw.efConstruction = 80
+    index.hnsw.efSearch = 128
+    index.add(candidate_embeddings)
+    scores, indices = index.search(query_embeddings, min(args.search_k, len(candidates)))
+
+    output_rows: list[dict[str, object]] = []
+    mined_count = 0
+    chosen_scores: list[float] = []
+    chosen_ranks: list[int] = []
+    for row, row_indices, row_scores in zip(mining_rows, indices, scores, strict=True):
+        negative, score, rank = select_hard_negative(
+            row["positive"],
+            row["negative"],
+            candidates,
+            row_indices.tolist(),
+            row_scores.tolist(),
+            max_positive_overlap=args.max_positive_overlap,
+        )
+        if rank is not None:
+            mined_count += 1
+            chosen_scores.append(float(score))
+            chosen_ranks.append(rank)
+        output_rows.append(
+            {
+                "id": row["id"],
+                "query": row["query"],
+                "positive": row["positive"],
+                "negative": negative,
+                "original_negative": row["negative"],
+                "source": row["source"],
+                "mining_score": score,
+                "mining_rank": rank,
+            }
+        )
+
+    train_output = args.output_dir / "train.parquet"
+    validation_output = args.output_dir / "validation.parquet"
+    Dataset.from_list(output_rows).to_parquet(str(train_output))
+    validation = load_dataset("parquet", data_files=str(args.validation), split="train")
+    validation.to_parquet(str(validation_output))
+    elapsed = time.perf_counter() - started
+    report = {
+        "method": "dense_hnsw_search_over_original_labeled_negatives",
+        "miner_model": args.model,
+        "query_style": args.query_style,
+        "benchmark_data_used": False,
+        "rows": len(output_rows),
+        "candidate_documents": len(candidates),
+        "search_k": args.search_k,
+        "max_positive_token_jaccard": args.max_positive_overlap,
+        "mined_rows": mined_count,
+        "fallback_rows": len(output_rows) - mined_count,
+        "mean_selected_cosine": float(np.mean(chosen_scores)) if chosen_scores else None,
+        "mean_selected_rank": float(np.mean(chosen_ranks)) if chosen_ranks else None,
+        "seconds": elapsed,
+        "files": {
+            "train.parquet": {"bytes": train_output.stat().st_size, "sha256": sha256(train_output)},
+            "validation.parquet": {
+                "bytes": validation_output.stat().st_size,
+                "sha256": sha256(validation_output),
+            },
+        },
+    }
+    (args.output_dir / "mining-report.json").write_text(
+        json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+    )
+    print(json.dumps(report, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()

training/train.py

@@ -0,0 +1,213 @@
+#!/usr/bin/env python3
+"""Fine-tune a Turkish dense retriever with reproducible settings."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import platform
+import random
+import re
+import time
+from pathlib import Path
+
+import datasets
+import numpy as np
+import sentence_transformers
+import torch
+import transformers
+import yaml
+from datasets import load_dataset
+from sentence_transformers import (
+    SentenceTransformer,
+    SentenceTransformerTrainer,
+    SentenceTransformerTrainingArguments,
+    losses,
+)
+from sentence_transformers.evaluation import TripletEvaluator
+from sentence_transformers.training_args import BatchSamplers
+
+from goktugtr.text import training_query
+
+
+def seed_everything(seed: int) -> None:
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--config", type=Path, default=Path("configs/train_270m.yaml"))
+    parser.add_argument("--data-dir", type=Path, default=Path("data/processed"))
+    parser.add_argument("--output-dir", type=Path, default=Path("outputs/goktugtr-270m"))
+    parser.add_argument("--max-train-rows", type=int)
+    parser.add_argument("--max-steps", type=int, default=-1)
+    return parser.parse_args()
+
+
+def latest_complete_checkpoint(output_dir: Path) -> Path | None:
+    """Return the newest checkpoint that contains all trainer resume state."""
+    candidates: list[tuple[int, Path]] = []
+    for path in output_dir.glob("checkpoint-*"):
+        match = re.fullmatch(r"checkpoint-(\d+)", path.name)
+        if not match:
+            continue
+        required = (
+            "model.safetensors",
+            "optimizer.pt",
+            "scheduler.pt",
+            "trainer_state.json",
+            "rng_state.pth",
+        )
+        if all((path / filename).is_file() for filename in required):
+            candidates.append((int(match.group(1)), path))
+    return max(candidates, default=(0, None), key=lambda item: item[0])[1]
+
+
+def main() -> None:
+    args = parse_args()
+    config = yaml.safe_load(args.config.read_text(encoding="utf-8"))
+    seed_everything(int(config["seed"]))
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+
+    dataset = load_dataset(
+        "parquet",
+        data_files={
+            "train": str(args.data_dir / "train.parquet"),
+            "validation": str(args.data_dir / "validation.parquet"),
+        },
+    )
+    train = dataset["train"]
+    if args.max_train_rows:
+        train = train.select(range(min(args.max_train_rows, len(train))))
+
+    def add_prompt(row: dict[str, str]) -> dict[str, str]:
+        return {
+            "anchor": training_query(row["query"], config.get("query_style", "harrier")),
+            "positive": row["positive"],
+            "negative": row["negative"],
+        }
+
+    remove_columns = dataset["train"].column_names
+    train = train.map(add_prompt, remove_columns=remove_columns, desc="Formatting train queries")
+    validation = dataset["validation"].map(
+        add_prompt, remove_columns=remove_columns, desc="Formatting validation queries"
+    )
+    train = train.select_columns(["anchor", "positive", "negative"])
+    validation = validation.select_columns(["anchor", "positive", "negative"])
+
+    model_kwargs = {"dtype": torch.bfloat16} if bool(config["bf16"]) else {}
+    processor_kwargs = {"padding_side": config.get("padding_side", "right")}
+    model = SentenceTransformer(
+        config["base_model"],
+        revision=config.get("base_model_revision"),
+        model_kwargs=model_kwargs,
+        processor_kwargs=processor_kwargs,
+    )
+    model.max_seq_length = int(config["max_seq_length"])
+
+    evaluator = TripletEvaluator(
+        anchors=validation["anchor"],
+        positives=validation["positive"],
+        negatives=validation["negative"],
+        name="goktugtr-validation",
+        batch_size=8,
+        show_progress_bar=True,
+    )
+    baseline = evaluator(model, output_path=str(args.output_dir), epoch=0, steps=0)
+    (args.output_dir / "baseline_triplet.json").write_text(
+        json.dumps(baseline, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+    )
+
+    training_args = SentenceTransformerTrainingArguments(
+        output_dir=str(args.output_dir),
+        num_train_epochs=float(config["epochs"]),
+        max_steps=args.max_steps,
+        per_device_train_batch_size=int(config["per_device_batch_size"]),
+        per_device_eval_batch_size=8,
+        gradient_accumulation_steps=int(config["gradient_accumulation_steps"]),
+        learning_rate=float(config["learning_rate"]),
+        warmup_steps=float(config["warmup_ratio"]),
+        bf16=bool(config["bf16"]),
+        tf32=True,
+        gradient_checkpointing=bool(config["gradient_checkpointing"]),
+        gradient_checkpointing_kwargs={"use_reentrant": False},
+        optim="adamw_torch_fused",
+        batch_sampler=BatchSamplers.NO_DUPLICATES,
+        eval_strategy="steps",
+        eval_steps=int(config["eval_steps"]),
+        save_strategy="steps",
+        save_steps=int(config["save_steps"]),
+        save_total_limit=2,
+        logging_steps=int(config["logging_steps"]),
+        dataloader_num_workers=2,
+        dataloader_pin_memory=True,
+        report_to="none",
+        run_name=config["project_name"],
+        seed=int(config["seed"]),
+    )
+    if config.get("loss") == "cached_multiple_negatives_ranking":
+        loss = losses.CachedMultipleNegativesRankingLoss(
+            model,
+            mini_batch_size=int(config.get("loss_mini_batch_size", 2)),
+            scale=20.0,
+        )
+    else:
+        loss = losses.MultipleNegativesRankingLoss(model, scale=20.0)
+    trainer = SentenceTransformerTrainer(
+        model=model,
+        args=training_args,
+        train_dataset=train,
+        eval_dataset=validation,
+        loss=loss,
+        evaluator=evaluator,
+    )
+    resume_checkpoint = latest_complete_checkpoint(args.output_dir)
+    if resume_checkpoint:
+        print(f"Resuming from complete checkpoint: {resume_checkpoint}", flush=True)
+    torch.cuda.reset_peak_memory_stats()
+    training_started = time.perf_counter()
+    train_output = trainer.train(
+        resume_from_checkpoint=str(resume_checkpoint) if resume_checkpoint else None
+    )
+    training_seconds = time.perf_counter() - training_started
+    trainer.state.save_to_json(str(args.output_dir / "trainer_state.json"))
+    final_dir = args.output_dir / "final"
+    model.save_pretrained(str(final_dir), safe_serialization=True)
+    final_metrics = evaluator(model, output_path=str(args.output_dir), epoch=1, steps=-1)
+    (args.output_dir / "final_triplet.json").write_text(
+        json.dumps(final_metrics, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+    )
+    environment = {
+        "python": platform.python_version(),
+        "torch": torch.__version__,
+        "transformers": transformers.__version__,
+        "sentence_transformers": sentence_transformers.__version__,
+        "datasets": datasets.__version__,
+        "cuda": torch.version.cuda,
+        "gpu": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
+        "gpu_total_memory_gb": (
+            round(torch.cuda.get_device_properties(0).total_memory / 2**30, 3)
+            if torch.cuda.is_available()
+            else None
+        ),
+        "config": config,
+        "train_rows": len(train),
+        "validation_rows": len(validation),
+        "training_seconds": training_seconds,
+        "resumed_from_checkpoint": (
+            str(resume_checkpoint) if resume_checkpoint is not None else None
+        ),
+        "training_metrics": train_output.metrics,
+        "max_gpu_memory_gb": round(torch.cuda.max_memory_allocated() / 2**30, 3),
+    }
+    (args.output_dir / "environment.json").write_text(
+        json.dumps(environment, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+    )
+    print(json.dumps({"baseline": baseline, "final": final_metrics}, indent=2))
+
+
+if __name__ == "__main__":
+    main()

training/train.yaml

@@ -0,0 +1,22 @@
+project_name: goktugtr-retrieval-270m-v2-hard-negatives
+base_model: outputs/goktugtr-270m/final
+base_model_revision: null
+query_style: harrier
+padding_side: left
+dataset_id: local/goktugtr-hard-negatives-v1
+seed: 3407
+max_seq_length: 256
+train_rows: 50000
+validation_rows: 2000
+epochs: 1
+learning_rate: 8.0e-6
+warmup_ratio: 0.05
+per_device_batch_size: 16
+gradient_accumulation_steps: 4
+loss: cached_multiple_negatives_ranking
+loss_mini_batch_size: 8
+gradient_checkpointing: true
+bf16: true
+eval_steps: 125
+save_steps: 125
+logging_steps: 10