Update README.md

README.md

@@ -18,8 +18,6 @@ tags:
 - mwirelabs
 - token-efficiency
 license: cc-by-4.0
-datasets:
-- MWirelabs/NE-BERT-Raw-Corpus
 pipeline_tag: fill-mask
 model-index:
 - name: NE-BERT
@@ -54,7 +52,29 @@ Built on the **ModernBERT** architecture, it supports a context length of **$102
 
 ---
 
-## Evaluation and Benchmarks: Regional SOTA
+## 💾 Training Data & Strategy
+
+NE-BERT was trained on a meticulously curated corpus using a **Smart-Weighted Sampling** strategy to ensure the low-resource languages were not drowned out by anchor languages.
+
+<div align="center">
+  <img src="https://huggingface.co/MWirelabs/ne-bert/resolve/main/ne_bert_data_dist.png" alt="Data Distribution Pie Chart" width="600"/>
+</div>
+
+| Language | HF Tag | Script | Corpus Size | Training Strategy |
+| :--- | :--- | :--- | :--- | :--- |
+| **Assamese** | `asm-Beng` | Bengali-Assamese | $\approx 1\text{M}$ Sentences | Native |
+| **Meitei (Manipuri)** | `mni-Beng` | Bengali-Assamese | $\approx 1.3\text{M}$ Sentences | Native |
+| **Khasi** | `kha-Latn` | Roman | $\approx 1\text{M}$ Sentences | Native |
+| **Mizo** | `lus-Latn` | Roman | $\approx 1\text{M}$ Sentences | Native |
+| **Nyishi** | `njz-Latn` | Roman | $\approx 55\text{k}$ Sentences | **Oversampled** ($20\text{x}$) |
+| **Garo** | `grt-Latn` | Roman | $\approx 10\text{k}$ Sentences | **Oversampled** ($20\text{x}$) |
+| **Pnar** | `pbv-Latn` | Roman | $\approx 1\text{k}$ Sentences | **Oversampled** ($100\text{x}$) |
+| **Kokborok** | `trp-Latn` | Roman | $\approx 2.5\text{k}$ Sentences | **Oversampled** ($100\text{x}$) |
+| **Anchor Languages** | `eng-Latn`/`hin-Deva` | Roman/Devanagari | $\approx 660\text{k}$ Sentences | Downsampled |
+
+---
+
+## 📈 Evaluation and Benchmarks: Regional SOTA
 
 We evaluated NE-BERT against industry-standard multilingual models (mBERT, XLM-R, IndicBERT) on a final, complex, held-out test set to ensure reproducibility and rigor.
 
@@ -62,7 +82,9 @@ We evaluated NE-BERT against industry-standard multilingual models (mBERT, XLM-R
 
 Perplexity measures the model's fluency and understanding of text (lower is better). This comparison proves NE-BERT's superior language modeling across the board, particularly in low-resource settings.
 
-
+<div align="center">
+  <img src="https://huggingface.co/MWirelabs/ne-bert/resolve/main/ppl_benchmark_chart.png" alt="Perplexity Benchmark Chart" width="800"/>
+</div>
 
 | Language | **NE-BERT** | mBERT | IndicBERT | Verdict |
 | :--- | :--- | :--- | :--- | :--- |
@@ -73,67 +95,27 @@ Perplexity measures the model's fluency and understanding of text (lower is bett
 | Mizo ($\text{lus}$) | **3.09** | 3.13 | 6.45 | **Best Specialized Model** |
 | **Garo** ($\text{grt}$) | **3.80** | 3.32 | 8.64 | **Crushes IndicBERT** |
 
-*Note: While XLM-R shows low PPL scores (which is often due to its highly fragmenting tokenizer), **NE-BERT** is the clear **Regional SOTA** winner against the most relevant competitors (mBERT and IndicBERT), proving its linguistic advantage.*
-
 ### 2. Efficiency: Token Fertility (Inference Speed)
 
 Token Fertility (Tokens per Word) is the key metric for inference speed and memory footprint (lower is better). NE-BERT's custom Unigram tokenizer delivers massive efficiency gains.
 
-
-
-| Language | **NE-BERT** | mBERT | XLM-R | IndicBERT |
-| :--- | :--- | :--- | :--- | :--- |
-| **Assamese** ($\text{asm}$) | **1.46** | 4.20 | 2.75 | 2.69 |
-| **Meitei** ($\text{mni}$) | **2.12** | 4.22 | 3.77 | 2.50 |
-| **Garo** ($\text{grt}$) | **2.12** | 3.62 | 3.34 | 3.95 |
-| **Pnar** ($\text{pbv}$) | **1.43** | 1.74 | 1.64 | 1.93 |
+<div align="center">
+  <img src="https://huggingface.co/MWirelabs/ne-bert/resolve/main/fertility_benchmark_chart.png" alt="Token Fertility Benchmark Chart" width="600"/>
+</div>
 
 *Result: NE-BERT is **$2\text{x}$ to $3\text{x}$ more token-efficient** on major languages than mBERT and XLM-R, translating directly to **faster inference** and **lower VRAM consumption** in production.*
 
 ---
 
-## Supported Languages and Data
-
-The model was trained on a custom corpus curated by **MWirelabs**, containing $\approx 8.3$ Million sentences.
-
-| Language | HF Tag | Script | Corpus Size | Training Strategy |
-| :--- | :--- | :--- | :--- | :--- |
-| **Assamese** | `asm-Beng` | Bengali-Assamese | $\approx 1\text{M}$ Sentences | Native |
-| **Meitei (Manipuri)** | `mni-Beng` | Bengali-Assamese | $\approx 1.3\text{M}$ Sentences | Native |
-| **Khasi** | `kha-Latn` | Roman | $\approx 1\text{M}$ Sentences | Native |
-| **Mizo** | `lus-Latn` | Roman | $\approx 1\text{M}$ Sentences | Native |
-| **Nyishi** | `njz-Latn` | Roman | $\approx 55\text{k}$ Sentences | **Oversampled** ($20\text{x}$) |
-| **Garo** | `grt-Latn` | Roman | $\approx 10\text{k}$ Sentences | **Oversampled** ($20\text{x}$) |
-| **Pnar** | `pbv-Latn` | Roman | $\approx 1\text{k}$ Sentences | **Oversampled** ($100\text{x}$) |
-| **Kokborok** | `trp-Latn` | Roman | $\approx 2.5\text{k}$ Sentences | **Oversampled** ($100\text{x}$) |
-| **Anchor Languages** | `eng-Latn`/`hin-Deva` | Roman/Devanagari | $\approx 660\text{k}$ Sentences | Downsampled |
-
-### Note on Data Strategy
-To prevent overfitting on the heavily upsampled micro-languages, we utilized **Dynamic Masking** during training. This forced the model to learn semantic relationships rather than memorizing token sequences across epochs.
-
-## Quick Use
-
-You can use NE-BERT directly with the Hugging Face `pipeline`. 
-**Note:** NE-BERT uses `<mask>` (XML style) instead of `[MASK]`.
-
-```python
-from transformers import pipeline
-
-# 1. Load Model
-unmasker = pipeline("fill-mask", model="MWirelabs/ne-bert", tokenizer="MWirelabs/ne-bert")
-
-# 2. Test Example (Khasi)
-# Input: "I go to [mask]" (Market/School/Home)
-sentence = "Nga leit sha <mask>."
+## Training Performance
 
-predictions = unmasker(sentence)
-for p in predictions[:3]:
-    print(f"{p['token_str']}: {p['score']:.1%}")
+<div align="center">
+  <img src="https://huggingface.co/MWirelabs/ne-bert/resolve/main/ne_bert_loss_chart.png" alt="Training Convergence Chart" width="800"/>
+</div>
 
-# Expected Output (Based on V3 visual test and plausible predictions):
-# iing: 8.2% (Home)
-# skul: 7.5% (School)
-# iew: 6.9% (Market)
+* **Final Training Loss:** 1.62
+* **Final Validation Loss:** 1.64
+* **Convergence:** The model achieved optimal convergence where validation loss tracked closely with training loss, indicating robust generalization despite the small dataset size of rare languages.
 
 ## Technical Specifications