REZIRETSA commited on
Commit
4d21e2a
Β·
verified Β·
1 Parent(s): 3ddbb6e

Switch org-card benchmarks to accuracy metrics (higher is better), run on local GPU

Browse files
Files changed (1) hide show
  1. README.md +4 -7
README.md CHANGED
@@ -16,16 +16,13 @@ pinned: false
16
  - [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M) β€” 4.5B-token pretrain (2.7Γ— scale-up)
17
  - LUNA-1B β€” in progress
18
 
19
- **Scaling benchmarks β€” 100M β†’ 300M** *(latest base checkpoints, same 4.5B-token corpus, held-out ~100K-token eval: EN / KN / TE / TA / HI + code)*
20
 
21
  | metric | LUNA-100M | LUNA-300M | gain |
22
  |---|---|---|---|
23
- | Eval loss | 3.88 | 3.81 | ↓1.6% |
24
- | Perplexity β€” overall | 48.2 | 45.2 | ↓6.1% |
25
- | Perplexity β€” English | 41.9 | 35.2 | ↓16.0% |
26
- | Perplexity β€” code | 51.9 | 43.9 | ↓15.4% |
27
- | Perplexity β€” Kannada | 15.7 | 13.5 | ↓13.8% |
28
- | Inference (CPU fp32) | 3,303 tok/s | 1,222 tok/s | 2.7Γ— params |
29
 
30
  **Tokenizer (multilingual BPE)**
31
  - [LUNA-1B-Tokenizer](https://huggingface.co/ASTERIZER/LUNA-1B-Tokenizer) β€” 128K / 64K / 8K vocab
 
16
  - [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M) β€” 4.5B-token pretrain (2.7Γ— scale-up)
17
  - LUNA-1B β€” in progress
18
 
19
+ **Scaling benchmarks β€” LUNA-100M β†’ LUNA-300M** *(latest base checkpoints, same 4.5B-token corpus, held-out ~100K-token eval on RTX 4060 Ti β€” higher is better)*
20
 
21
  | metric | LUNA-100M | LUNA-300M | gain |
22
  |---|---|---|---|
23
+ | Next-token prediction accuracy | 32.5% | 32.7% | ↑0.8% |
24
+ | Word-prediction accuracy (LAMBADA-style) | 16.3% | 17.1% | ↑4.4% |
25
+ | Throughput (RTX 4060 Ti) | 39.2K tok/s | 14.5K tok/s | 2.7Γ— params |
 
 
 
26
 
27
  **Tokenizer (multilingual BPE)**
28
  - [LUNA-1B-Tokenizer](https://huggingface.co/ASTERIZER/LUNA-1B-Tokenizer) β€” 128K / 64K / 8K vocab