GoktugD/DUSUNEN-Rota-270M-v1

DUSUNEN Rota 270M v1

A compact Turkish dense retriever trained and evaluated in the open.

DUSUNEN Rota maps Turkish search queries and passages into a shared 640-dimensional vector space. It is intended for first-stage retrieval, semantic search and RAG candidate generation.

This release includes the weights, exact training configuration, data-preparation code, benchmark implementation, raw result files and environment manifest.

Measured results

All systems are evaluated by the same implementation on the pinned mteb/TurHistQuadRetrieval revision b6e74379b7486da28ce81c3d459cd7bbd87d4987. The benchmark contains 1,024 Turkish queries and 1,213 documents and is separate from the MS MARCO-derived training data.

Model	Params	Dim	MRR@10	nDCG@10	Recall@10	Recall@100
DUSUNEN Rota 270M v1	268.1M	640	0.5942	0.4355	0.4888	0.6987
Microsoft Harrier OSS 270M (base)	268.1M	640	0.6241	0.4550	0.5068	0.7090
Mursit Base TR Retrieval	155.6M	768	0.5870	0.4307	0.4868	0.7173
multilingual-e5-base	278.0M	768	0.6931	0.4984	0.5332	0.7368

Scores come from exact inner-product search over L2-normalized embeddings. Throughput is reported in results/turhistquad-results.json, but should only be compared on equivalent hardware and software. Similarity scores are ranking signals, not calibrated probabilities.

The source-domain triplet accuracy improved from 0.8785 to 0.9000, while TurHistQuad MRR@10 decreased relative to the untouched Harrier base. DUSUNEN Rota v1 slightly exceeds Mursit on MRR@10, nDCG@10 and Recall@10, but does not beat the base Harrier or multilingual-e5-base on this benchmark. This is an experimental Turkish domain-adaptation result, not a universal upgrade or a state-of-the-art claim. The divergence is useful evidence of domain shift from machine-translated MS MARCO triplets to historical Turkish retrieval.

Use

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("GoktugD/DUSUNEN-Rota-270M-v1")

task = "Given a Turkish web search query, retrieve relevant passages that answer the query"
query = f"Instruct: {task}\nQuery: İstanbul'un nüfusu kaçtır?"
documents = [
    "İstanbul, Türkiye'nin en kalabalık şehridir.",
    "Ankara, Türkiye'nin başkentidir.",
]

query_embedding = model.encode(query, normalize_embeddings=True)
document_embeddings = model.encode(documents, normalize_embeddings=True)
scores = document_embeddings @ query_embedding
print(scores)

Queries require the instruction format shown above. Documents are encoded as plain text. For vector search, store normalized float32 embeddings and use cosine similarity or inner product.

Training recipe

• Base: microsoft/harrier-oss-v1-270m
• Base revision: 31de22b673913c7d658c0f03f792d77c2dcf8ebd
• Data: 100,000 train / 2,000 validation triplets
• Objective: Cached Multiple Negatives Ranking Loss
• Sequence length: 256 tokens
• Effective batch size: 64
• Optimizer learning rate: 2e-5
• Schedule: 5% warmup, one epoch
• Precision: BF16
• Memory controls: gradient checkpointing and loss mini-batches of eight
• Seed: 3407
• Hardware: one NVIDIA GeForce RTX 5060 Laptop GPU (8 GB)

The public dataset is a deterministic, quality-filtered subset of newmindai/ms-marco-turkish-triplets. That upstream dataset is a machine translation of MS MARCO; it is not human-authored Turkish. Filtering removes duplicates, malformed rows, identical pairs, extreme lengths, repeated-character artifacts and high positive–negative lexical overlap. See the dataset card for provenance, counts and SHA-256 checksums.

Exact-overlap audit

An additional SHA-256 audit was run after applying the project's case-folded text normalization. It found zero exact query-to-query overlaps, zero exact passage-to-document overlaps and zero exact overlaps across any training and benchmark text. The machine-readable report is included as results/turhistquad-overlap-audit.json.

This is deliberately described as an exact normalized-text audit. It does not prove the absence of paraphrases, shared facts or semantic near-duplicates.

Browser demo

The linked Search Lab runs a validated mixed-precision ONNX export locally in the visitor's browser with Transformers.js and WebAssembly. The 168M-parameter embedding table uses per-dimension int8 quantization while the transformer blocks remain float to prevent the severe long-sequence drift observed with full dynamic int8. Queries are not sent to a hosted inference API, and the demo does not require paid compute. Its 24-document corpus is original and separate from training and benchmark data.

The repository includes onnx/model_quantized.onnx and a PyTorch-to-ONNX agreement report in results/onnx-browser-validation.json. Across 24 documents and four queries, mean embedding cosine agreement is 0.999956, the minimum is 0.999897 and top-1 agreement is 100%.

Reproduce

Relevant files in this repository:

• training/train.py — fine-tuning and validation
• training/evaluate_retrieval.py — exact retrieval benchmark
• training/audit_contamination.py — exact normalized-text overlap audit
• training/export_browser_model.py — mixed-precision ONNX export and agreement check
• training/build_release.py — fail-closed release assembly and SHA-256 manifest
• training/validate_release.py — release, metric, ONNX and dataset verification
• training/train_270m.yaml — all training hyperparameters
• results/turhistquad-results.json — machine-readable metrics
• results/environment.json — package, GPU and peak-memory record

python training/train.py \
  --data-dir data/processed \
  --output-dir outputs/DUSUNEN-Rota-270M-v1

python training/evaluate_retrieval.py \
  --models outputs/DUSUNEN-Rota-270M-v1/final \
  --output results/turhistquad-results.json

Limitations

• The training corpus inherits translation errors, dated facts and web-text biases from its upstream source.
• Evaluation covers a small Turkish historical question-answer retrieval task; it does not establish quality for every domain or dialect.
• The model is a retriever, not a fact checker. A high similarity score does not establish that a passage is correct.
• Long documents should be chunked before encoding. Performance depends on the chunking policy and target corpus.
• Users should evaluate domain shift, privacy and representational harms before production deployment.

License and attribution

Model weights are released under MIT, matching the Harrier base model. The DUSUNEN training dataset is released separately under Apache-2.0, matching its upstream source. See the linked repositories for full attribution.