Update AGIFORMER with Turkish benchmark
Browse files
benchmark/benchmark_report.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Kaşgarlı Testi - Benchmark Results
|
| 2 |
+
|
| 3 |
+
## Hypothesis
|
| 4 |
+
**H1:** Byte-level models learn agglutinative languages (Turkish) more efficiently than analytic languages (English).
|
| 5 |
+
|
| 6 |
+
## Experimental Setup
|
| 7 |
+
- **Model:** AGIFORMER (identical architecture, 50M parameters)
|
| 8 |
+
- **Hyperparameters:** Same for both (d_model=512, n_layers=6, thinking_steps=3)
|
| 9 |
+
- **Training:** 5000 steps, batch_size=4, lr=3e-4
|
| 10 |
+
- **English Dataset:** enwik8 (100MB Wikipedia)
|
| 11 |
+
- **Turkish Dataset:** trwiki (Turkish Wikipedia)
|
| 12 |
+
|
| 13 |
+
## Results
|
| 14 |
+
|
| 15 |
+
### Final BPC (Lower is Better)
|
| 16 |
+
| Language | Validation BPC |
|
| 17 |
+
|----------|----------------|
|
| 18 |
+
| English | 2.2578 |
|
| 19 |
+
| Turkish | 2.1226 |
|
| 20 |
+
|
| 21 |
+
**Difference:** 0.1352 BPC
|
| 22 |
+
|
| 23 |
+
### Convergence Speed
|
| 24 |
+
Steps to reach BPC < 2.5:
|
| 25 |
+
- English: Not reached
|
| 26 |
+
- Turkish: 1550
|
| 27 |
+
|
| 28 |
+
## Conclusion
|
| 29 |
+
|
| 30 |
+
Turkish model outperformed English, confirming the hypothesis.
|
| 31 |
+
|
| 32 |
+
## Visualization
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
---
|
| 36 |
+
**Generated:** 2025-11-22
|
| 37 |
+
**Experimenter:** inkbytefo
|