Add model weights, tokenizer and bilingual model card

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/tabibench.png filter=lfs diff=lfs merge=lfs -text
+assets/tabibench_en.png filter=lfs diff=lfs merge=lfs -text
+assets/trglue.png filter=lfs diff=lfs merge=lfs -text
+assets/trglue_en.png filter=lfs diff=lfs merge=lfs -text
+banner.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,138 @@
+---
+license: apache-2.0
+language:
+- tr
+library_name: transformers
+pipeline_tag: fill-mask
+tags:
+- arxiv:2608.25768
+- modernbert
+- turkish
+- encoder
+- foundation-model
+---
+
+<p align="center">
+  <img src="banner.png" alt="MoganBERT-TR" width="600"/>
+</p>
+
+<p align="center">
+  <a href="https://huggingface.co/papers/2608.25768"><img src="https://img.shields.io/badge/📄_Paper-arXiv-B31B1B?style=flat-square" alt="Paper"/></a>
+  <a href="https://moganai.github.io/"><img src="https://img.shields.io/badge/🌐_Blog-MoganAI-2E7D5B?style=flat-square" alt="Blog"/></a>
+  <a href="https://huggingface.co/collections/moganai/moganbert"><img src="https://img.shields.io/badge/🤗_Model_Collection-2C3E50?style=flat-square" alt="Model Collection"/></a>
+</p>
+
+# MoganBERT-TR
+
+MoganBERT-TR is a 149.4M-parameter Turkish encoder with an 8,192-token context window, trained from scratch on 237.3B tokens. It follows the ModernBERT architecture — 22 layers, hidden size 768, 12 heads, alternating local/global attention — with a 50,048-token SentencePiece Unigram tokenizer trained for Turkish.
+
+Unlike other Turkish encoders, it is pretrained with a two-stage **CLM→MLM curriculum** rather than pure MLM.
+
+## Usage
+
+```python
+from transformers import pipeline
+
+unmask = pipeline("fill-mask", model="moganai/MoganBERT-TR")
+unmask("Türkiye'nin başkenti [MASK].")
+```
+
+Use `classifier_pooling="mean"` (the config default) when fine-tuning — ModernBERT has no NSP, so `[CLS]` is not pretrained.
+
+For embeddings and retrieval use [MoganBERT-embed](https://huggingface.co/moganai/MoganBERT-embed) or [MoganColBERT-TR](https://huggingface.co/moganai/MoganColBERT-TR); the raw encoder is anisotropic and unsuitable for direct cosine similarity.
+
+## Training
+
+Pretrained on 4×H100 over 237.3B tokens: causal language modelling for the first 16.6% of the run, masked language modelling for the remainder, with the transition made inside the stable phase of a WSD schedule. Annealing extends the context 1024→8192, scales the global RoPE θ to 160k and lowers the mask rate to 10%. This checkpoint decays the final portion at 1024 context.
+
+The corpus was built from FineWeb2, raw Common Crawl WARC records and printed/institutional sources, filtered with a fastText quality classifier distilled from a fine-tuned Turkish BERT. The mixture is ~73% Turkish, ~17% English and ~10% code.
+
+## Results
+
+Evaluated on TrGLUE (8 tasks, 5 official seeds, official `run_trglue.py`) and TabiBench (28 datasets, 8 categories).
+
+<img src="assets/trglue_en.png" alt="TrGLUE" width="850"/>
+
+<img src="assets/tabibench_en.png" alt="TabiBench" width="850"/>
+
+TabiBench is single-seed; sub-one-point margins are not an ordering. Full per-task numbers are in the [paper](https://huggingface.co/papers/2608.25768).
+
+## Model Family
+
+| Model | Params | Purpose |
+|---|---:|---|
+| [**MoganBERT-TR**](https://huggingface.co/moganai/MoganBERT-TR) | 149.4M | Base encoder (this model) |
+| [MoganBERT-embed](https://huggingface.co/moganai/MoganBERT-embed) | 149M | Single-vector embeddings |
+| [MoganColBERT-TR](https://huggingface.co/moganai/MoganColBERT-TR) | 148.9M | Multi-vector retrieval |
+
+## Citation
+
+```bibtex
+@article{yilmaz2026moganbert,
+  title   = {MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-MLM Curriculum},
+  author  = {Furkan Yilmaz and Habibe Aleyna Tasdemir and Muhammed Faruk Gozay},
+  journal = {arXiv preprint arXiv:2608.25768},
+  year    = {2026}
+}
+```
+
+---
+---
+
+<p align="center">
+  <img src="banner.png" alt="MoganBERT-TR" width="600"/>
+</p>
+
+# MoganBERT-TR (Türkçe)
+
+MoganBERT-TR, 8.192 token bağlam penceresine sahip, 237.3 milyar token üzerinde sıfırdan eğitilmiş 149.4M parametreli bir Türkçe encoder'dır. ModernBERT mimarisini izler — 22 katman, gizli boyut 768, 12 başlık, dönüşümlü yerel/küresel dikkat — ve Türkçe için eğitilmiş 50.048 tokenlık bir SentencePiece Unigram tokenizer kullanır.
+
+Diğer Türkçe encoder'lardan farklı olarak saf MLM yerine iki aşamalı bir **CLM→MLM müfredatı** ile ön-eğitilmiştir.
+
+## Kullanım
+
+```python
+from transformers import pipeline
+
+unmask = pipeline("fill-mask", model="moganai/MoganBERT-TR")
+unmask("Türkiye'nin başkenti [MASK].")
+```
+
+İnce ayarda `classifier_pooling="mean"` (config varsayılanı) kullanın — ModernBERT'te NSP olmadığı için `[CLS]` ön-eğitilmiş değildir.
+
+Gömme ve arama için [MoganBERT-embed](https://huggingface.co/moganai/MoganBERT-embed) veya [MoganColBERT-TR](https://huggingface.co/moganai/MoganColBERT-TR) kullanın; ham encoder anizotropiktir ve doğrudan kosinüs benzerliğine uygun değildir.
+
+## Eğitim
+
+4×H100 üzerinde 237.3 milyar token ile ön-eğitildi: eğitimin ilk %16.6'sı nedensel dil modellemesi, kalanı maskeli dil modellemesi; geçiş WSD çizelgesinin durağan fazı içinde yapıldı. Tavlama bağlamı 1024→8192'ye çıkarır, küresel RoPE θ'yı 160k'ya ölçekler ve maske oranını %10'a indirir. Bu checkpoint son bölümü 1024 bağlamda sönümler.
+
+Külliyat FineWeb2, ham Common Crawl WARC kayıtları ve basılı/kurumsal kaynaklardan derlendi; ince ayarlanmış bir Türkçe BERT'ten damıtılan fastText kalite sınıflandırıcısıyla filtrelendi. Karışım ~%73 Türkçe, ~%17 İngilizce ve ~%10 kod.
+
+## Sonuçlar
+
+TrGLUE (8 görev, 5 resmî tohum, resmî `run_trglue.py`) ve TabiBench (28 veri kümesi, 8 kategori) üzerinde ölçüldü.
+
+<img src="assets/trglue.png" alt="TrGLUE" width="850"/>
+
+<img src="assets/tabibench.png" alt="TabiBench" width="850"/>
+
+TabiBench tek tohumludur; bir puanın altındaki farklar bir sıralama değildir. Görev bazlı tüm sayılar [makalede](https://huggingface.co/papers/2608.25768).
+
+## Model Ailesi
+
+| Model | Parametre | Amaç |
+|---|---:|---|
+| [**MoganBERT-TR**](https://huggingface.co/moganai/MoganBERT-TR) | 149.4M | Temel encoder (bu model) |
+| [MoganBERT-embed](https://huggingface.co/moganai/MoganBERT-embed) | 149M | Tek vektörlü gömme |
+| [MoganColBERT-TR](https://huggingface.co/moganai/MoganColBERT-TR) | 148.9M | Çok vektörlü retrieval |
+
+## Atıf
+
+```bibtex
+@article{yilmaz2026moganbert,
+  title   = {MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-MLM Curriculum},
+  author  = {Furkan Yilmaz and Habibe Aleyna Tasdemir and Muhammed Faruk Gozay},
+  journal = {arXiv preprint arXiv:2608.25768},
+  year    = {2026}
+}
+```

config.json

@@ -0,0 +1,42 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 2,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 3,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "local_attention": 128,
+  "local_rope_theta": 10000.0,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 1,
+  "repad_logits_with_grad": false,
+  "sep_token_id": 3,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": true,
+  "transformers_version": "4.57.6",
+  "vocab_size": 50048
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:083817ab314e14625eda60d187f5a4b77fd547ccf7520e464c8f62db4717cf26
+size 597650712

special_tokens_map.json

@@ -0,0 +1,32 @@
+{
+  "additional_special_tokens": [
+    "[unused0]",
+    "[unused1]",
+    "[unused2]",
+    "[unused3]",
+    "[unused4]",
+    "[unused5]",
+    "[unused6]",
+    "[unused7]",
+    "[unused8]",
+    "[unused9]",
+    "[unused10]",
+    "[unused11]",
+    "[unused12]",
+    "[unused13]",
+    "[unused14]",
+    "[unused15]"
+  ],
+  "cls_token": "[CLS]",
+  "eos_token": "[EOS]",
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,208 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "[EOS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "[unused0]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "[unused1]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "[unused2]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "[unused3]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "[unused4]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "[unused5]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "[unused6]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "[unused7]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "[unused8]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "[unused9]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "[unused10]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "[unused11]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "[unused12]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "[unused13]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "[unused14]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "[unused15]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "[unused0]",
+    "[unused1]",
+    "[unused2]",
+    "[unused3]",
+    "[unused4]",
+    "[unused5]",
+    "[unused6]",
+    "[unused7]",
+    "[unused8]",
+    "[unused9]",
+    "[unused10]",
+    "[unused11]",
+    "[unused12]",
+    "[unused13]",
+    "[unused14]",
+    "[unused15]"
+  ],
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "eos_token": "[EOS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}