Cukinator commited on
Commit
944dc33
Β·
verified Β·
1 Parent(s): 8f4b2d7

Update README: recalculated metrics 2026-06-12

Browse files
Files changed (1) hide show
  1. README.md +22 -22
README.md CHANGED
@@ -31,38 +31,38 @@ Code: [github.com/Cukinator/1.58bits](https://github.com/Cukinator/1.58bits)
31
 
32
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Ξ” uniform | P/s | Bytes/s | Nivel |
33
  |-----|-------------|--------:|------------:|---------:|----:|----:|----------:|----:|--------:|-------|
34
- | run_01 | Transformer + BPE 16K + FP16 | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
35
- | run_02a_byte_only_heads | Transformer + Byte + 4 heads (no LBD) | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
36
- | run_02 | Transformer + Byte + LocalByteDecoder | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
37
- | run_03 | MLGRU + Byte + FP16 | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
38
- | run_04 | MLGRU + Byte + Ternary | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
39
- | run_05 | + FPResidual | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
40
- | run_05b_kernel_strict | MLGRU kernel-strict (no W_o) | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
41
- | run_06 | + Bolmo patch embedding | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
42
- | run_07 | + DeleteGate (CPU-1 completo) ⭐ | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
43
- | run_08 | Folded Transformer + Byte + Ternary | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
44
- | run_09 | + PFNet | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
45
- | run_10 | + decay per-canal aprendido | β€” | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
46
 
47
  ## Small runs β€” 10M (round 1, 2 tok/param)
48
 
49
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Nivel |
50
  |-----|-------------|--------:|------------:|---------:|----:|----:|-------|
51
- | run_13 | Small CPU-1 + BPE 4K | β€” | β€” | β€” | β€” | β€” | byte |
52
- | run_14 | Small CPU-1 byte (Qwen distill.) | β€” | β€” | β€” | β€” | β€” | byte |
53
- | run_15 | Small CPU-1 byte + hidden distill. | β€” | β€” | β€” | β€” | β€” | byte |
54
- | run_16 | Small CPU-1 raw bytes, no teacher | β€” | β€” | β€” | β€” | β€” | byte |
55
 
56
  ## Round 2 β€” re-runs a 15 tok/param
57
 
58
  | Run | Arquitectura | Params M | Loss r1 | PPL r1 | Loss r2 | PPL r2 | Ξ” loss | BPB r2 | Nivel |
59
  |-----|-------------|--------:|--------:|-------:|--------:|-------:|-------:|-------:|-------|
60
- | run_04_r2 | run_04 @ 15 tok/param | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
61
- | run_07_r2 | run_07 @ 15 tok/param | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
62
- | run_13_r2 | run_13 @ 15 tok/param | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
63
- | run_14_r2 | run_14 @ 15 tok/param | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
64
- | run_15_r2 | run_15 @ 15 tok/param | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
65
- | run_16_r2 | run_16 @ 15 tok/param | β€” | β€” | β€” | β€” | β€” | β€” | β€” | byte |
66
 
67
  ## Licencia
68
 
 
31
 
32
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Ξ” uniform | P/s | Bytes/s | Nivel |
33
  |-----|-------------|--------:|------------:|---------:|----:|----:|----------:|----:|--------:|-------|
34
+ | run_01 | Transformer + BPE 16K + FP16 | 54.7 | 218.9 | 4.6648 | 106.14 | β€” | β€” | 72.9 | 73 | token |
35
+ | run_02a_byte_only_heads | Transformer + Byte + 4 heads (no LBD) | 38.5 | 154.0 | 2.3079 | 10.05 | 3.3295 | -3.2373 | 81.3 | 325 | byte |
36
+ | run_02 | Transformer + Byte + LocalByteDecoder | 38.8 | 155.3 | 1.7154 | 5.56 | 2.4748 | -3.8298 | 69.9 | 280 | byte |
37
+ | run_03 | MLGRU + Byte + FP16 | 38.8 | 155.3 | 1.8697 | 6.49 | 2.6974 | -3.6755 | 57.1 | 228 | byte |
38
+ | run_04 | MLGRU + Byte + Ternary | 38.9 | 155.4 | 5.5671 | 261.68 | 8.0317 | 0.0220 | 8.1 | 32 | byte |
39
+ | run_05 | + FPResidual | 39.0 | 156.0 | 5.5516 | 257.65 | 8.0093 | 0.0064 | 7.8 | 31 | byte |
40
+ | run_05b_kernel_strict | MLGRU kernel-strict (no W_o) | 35.8 | 143.4 | 5.5940 | 268.82 | 8.0705 | 0.0489 | 8.4 | 34 | byte |
41
+ | run_06 | + Bolmo patch embedding | 39.0 | 156.0 | 5.5561 | 258.81 | 8.0157 | 0.0109 | 7.5 | 30 | byte |
42
+ | run_07 | + DeleteGate (CPU-1 completo) ⭐ | 39.0 | 156.1 | 5.5561 | 258.81 | 8.0157 | 0.0109 | 7.6 | 30 | byte |
43
+ | run_08 | Folded Transformer + Byte + Ternary | 38.9 | 155.4 | 5.5621 | 260.37 | 8.0244 | 0.0169 | 8.0 | 32 | byte |
44
+ | run_09 | + PFNet | 39.4 | 157.7 | 5.5330 | 252.91 | 7.9825 | -0.0122 | 7.3 | 29 | byte |
45
+ | run_10 | + decay per-canal aprendido | 39.4 | 157.7 | 5.5337 | 253.08 | 7.9835 | -0.0115 | 7.1 | 28 | byte |
46
 
47
  ## Small runs β€” 10M (round 1, 2 tok/param)
48
 
49
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Nivel |
50
  |-----|-------------|--------:|------------:|---------:|----:|----:|-------|
51
+ | run_13 | Small CPU-1 + BPE 4K | 12.5 | 50.2 | 30.5375 | 18293109063475.55 | β€” | token |
52
+ | run_14 | Small CPU-1 byte (Qwen distill.) | 10.7 | 42.8 | 5.5755 | 263.88 | 8.0437 | byte |
53
+ | run_15 | Small CPU-1 byte + hidden distill. | 10.7 | 42.9 | 5.5755 | 263.88 | 8.0437 | byte |
54
+ | run_16 | Small CPU-1 raw bytes, no teacher | 10.7 | 42.8 | 5.5754 | 263.85 | 8.0436 | byte |
55
 
56
  ## Round 2 β€” re-runs a 15 tok/param
57
 
58
  | Run | Arquitectura | Params M | Loss r1 | PPL r1 | Loss r2 | PPL r2 | Ξ” loss | BPB r2 | Nivel |
59
  |-----|-------------|--------:|--------:|-------:|--------:|-------:|-------:|-------:|-------|
60
+ | run_04_r2 | run_04 @ 15 tok/param | 38.9 | 5.5671 | 261.68 | β€” | β€” | β€” | β€” | byte |
61
+ | run_07_r2 | run_07 @ 15 tok/param | 39.0 | 5.5561 | 258.81 | β€” | β€” | β€” | β€” | byte |
62
+ | run_13_r2 | run_13 @ 15 tok/param | 12.5 | 30.5375 | 18293109063475.55 | 25.4264 | 110288596599.28 | -5.11118 | β€” | token |
63
+ | run_14_r2 | run_14 @ 15 tok/param | 10.7 | 5.5755 | 263.88 | 5.5755 | 263.88 | -0.00000 | 8.0437 | byte |
64
+ | run_15_r2 | run_15 @ 15 tok/param | 10.7 | 5.5755 | 263.88 | 5.5755 | 263.88 | -0.00000 | 8.0437 | byte |
65
+ | run_16_r2 | run_16 @ 15 tok/param | 10.7 | 5.5754 | 263.85 | 5.5754 | 263.85 | +0.00000 | 8.0436 | byte |
66
 
67
  ## Licencia
68