Switch org-card benchmarks to accuracy metrics (higher is better), run on local GPU
Browse files
README.md
CHANGED
|
@@ -16,16 +16,13 @@ pinned: false
|
|
| 16 |
- [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M) β 4.5B-token pretrain (2.7Γ scale-up)
|
| 17 |
- LUNA-1B β in progress
|
| 18 |
|
| 19 |
-
**Scaling benchmarks β 100M β 300M** *(latest base checkpoints, same 4.5B-token corpus, held-out ~100K-token eval
|
| 20 |
|
| 21 |
| metric | LUNA-100M | LUNA-300M | gain |
|
| 22 |
|---|---|---|---|
|
| 23 |
-
|
|
| 24 |
-
|
|
| 25 |
-
|
|
| 26 |
-
| Perplexity β code | 51.9 | 43.9 | β15.4% |
|
| 27 |
-
| Perplexity β Kannada | 15.7 | 13.5 | β13.8% |
|
| 28 |
-
| Inference (CPU fp32) | 3,303 tok/s | 1,222 tok/s | 2.7Γ params |
|
| 29 |
|
| 30 |
**Tokenizer (multilingual BPE)**
|
| 31 |
- [LUNA-1B-Tokenizer](https://huggingface.co/ASTERIZER/LUNA-1B-Tokenizer) β 128K / 64K / 8K vocab
|
|
|
|
| 16 |
- [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M) β 4.5B-token pretrain (2.7Γ scale-up)
|
| 17 |
- LUNA-1B β in progress
|
| 18 |
|
| 19 |
+
**Scaling benchmarks β LUNA-100M β LUNA-300M** *(latest base checkpoints, same 4.5B-token corpus, held-out ~100K-token eval on RTX 4060 Ti β higher is better)*
|
| 20 |
|
| 21 |
| metric | LUNA-100M | LUNA-300M | gain |
|
| 22 |
|---|---|---|---|
|
| 23 |
+
| Next-token prediction accuracy | 32.5% | 32.7% | β0.8% |
|
| 24 |
+
| Word-prediction accuracy (LAMBADA-style) | 16.3% | 17.1% | β4.4% |
|
| 25 |
+
| Throughput (RTX 4060 Ti) | 39.2K tok/s | 14.5K tok/s | 2.7Γ params |
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
**Tokenizer (multilingual BPE)**
|
| 28 |
- [LUNA-1B-Tokenizer](https://huggingface.co/ASTERIZER/LUNA-1B-Tokenizer) β 128K / 64K / 8K vocab
|