Cukinator commited on
Commit
8600b96
Β·
verified Β·
1 Parent(s): ebb533e

Update README: recalculated metrics 2026-06-12

Browse files
Files changed (1) hide show
  1. README.md +44 -0
README.md CHANGED
@@ -31,11 +31,55 @@ Code: [github.com/Cukinator/1.58bits](https://github.com/Cukinator/1.58bits)
31
 
32
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Ξ” uniform | P/s | Bytes/s | Nivel |
33
  |-----|-------------|--------:|------------:|---------:|----:|----:|----------:|----:|--------:|-------|
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
  ## Small runs β€” 10M (round 1, 2 tok/param)
36
 
37
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Nivel |
38
  |-----|-------------|--------:|------------:|---------:|----:|----:|-------|
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
 
40
  ## Licencia
41
 
 
31
 
32
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Ξ” uniform | P/s | Bytes/s | Nivel |
33
  |-----|-------------|--------:|------------:|---------:|----:|----:|----------:|----:|--------:|-------|
34
+ | run_01 | Transformer + BPE 16K + FP16 | 54.7 | 218.9 | 4.6648 | 106.14 | β€” | β€” | 78.0 | 78 | token |
35
+ | run_02a_byte_only_heads | Transformer + Byte + 4 heads (no LBD) | 38.5 | 154.0 | 2.3079 | 10.05 | 3.3295 | -3.2373 | 85.9 | 344 | byte |
36
+ | run_02 | Transformer + Byte + LocalByteDecoder | 38.8 | 155.3 | 1.7154 | 5.56 | 2.4748 | -3.8298 | 86.1 | 344 | byte |
37
+ | run_03 | MLGRU + Byte + FP16 | 38.8 | 155.3 | 1.8697 | 6.49 | 2.6974 | -3.6755 | 59.9 | 240 | byte |
38
+ | run_04 | MLGRU + Byte + Ternary | 38.9 | 155.4 | 5.5671 | 261.68 | 8.0317 | 0.0220 | 8.0 | 32 | byte |
39
+ | run_05 | + FPResidual | 39.0 | 156.0 | 5.5516 | 257.65 | 8.0093 | 0.0064 | 7.6 | 30 | byte |
40
+ | run_05b_kernel_strict | MLGRU kernel-strict (no W_o) | 35.8 | 143.4 | 5.5940 | 268.82 | 8.0705 | 0.0489 | 8.7 | 35 | byte |
41
+ | run_06 | + Bolmo patch embedding | 39.0 | 156.0 | 5.5561 | 258.81 | 8.0157 | 0.0109 | 7.8 | 31 | byte |
42
+ | run_07 | + DeleteGate (CPU-1 completo) ⭐ | 39.0 | 156.1 | 5.5561 | 258.81 | 8.0157 | 0.0109 | 7.8 | 31 | byte |
43
+ | run_08 | Folded Transformer + Byte + Ternary | 38.9 | 155.4 | 5.5621 | 260.37 | 8.0244 | 0.0169 | 8.0 | 32 | byte |
44
+ | run_09 | + PFNet | 39.4 | 157.7 | 5.5330 | 252.91 | 7.9825 | -0.0122 | 7.3 | 29 | byte |
45
+ | run_10 | + decay per-canal aprendido | 39.4 | 157.7 | 5.5337 | 253.08 | 7.9835 | -0.0115 | 7.1 | 28 | byte |
46
 
47
  ## Small runs β€” 10M (round 1, 2 tok/param)
48
 
49
  | Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Nivel |
50
  |-----|-------------|--------:|------------:|---------:|----:|----:|-------|
51
+ | run_13 | Small CPU-1 + BPE 4K | 12.5 | 50.2 | 30.5375 | 18293109063475.55 | β€” | token |
52
+ | run_14 | Small CPU-1 byte (Qwen distill.) | 10.7 | 42.8 | 5.5755 | 263.88 | 8.0437 | byte |
53
+ | run_15 | Small CPU-1 byte + hidden distill. | 10.7 | 42.9 | 5.5755 | 263.88 | 8.0437 | byte |
54
+ | run_16 | Small CPU-1 raw bytes, no teacher | 10.7 | 42.8 | 5.5754 | 263.85 | 8.0436 | byte |
55
+
56
+ ## Round 2 β€” re-runs a 15 tok/param
57
+
58
+ | Run | Arquitectura | Params M | Loss r1 | PPL r1 | Loss r2 | PPL r2 | Ξ” loss | BPB r2 | Nivel |
59
+ |-----|-------------|--------:|--------:|-------:|--------:|-------:|-------:|-------:|-------|
60
+ | run_04_r2 | run_04 @ 15 tok/param | 38.9 | 5.5671 | 261.68 | β€” | β€” | β€” | β€” | byte |
61
+ | run_07_r2 | run_07 @ 15 tok/param | 39.0 | 5.5561 | 258.81 | β€” | β€” | β€” | β€” | byte |
62
+ | run_13_r2 | run_13 @ 15 tok/param | 12.5 | 30.5375 | 18293109063475.55 | 25.4264 | 110288596599.28 | -5.11118 | β€” | token |
63
+ | run_14_r2 | run_14 @ 15 tok/param | 10.7 | 5.5755 | 263.88 | 5.5755 | 263.88 | -0.00000 | 8.0437 | byte |
64
+ | run_15_r2 | run_15 @ 15 tok/param | 10.7 | 5.5755 | 263.88 | 5.5755 | 263.88 | -0.00000 | 8.0437 | byte |
65
+ | run_16_r2 | run_16 @ 15 tok/param | 10.7 | 5.5754 | 263.85 | 5.5754 | 263.85 | +0.00000 | 8.0436 | byte |
66
+
67
+ ## Round 3 β€” v3 runs (bf16 AMP + lrΓ—2 + 50 tok/p)
68
+
69
+ | Run | Arquitectura | Params M | Val Loss | PPL | BPB | Step | Nivel |
70
+ |-----|-------------|--------:|--------:|----:|----:|-----:|-------|
71
+ | run_01_v3 | run_01 v3 | 54.7 | β€” | β€” | β€” | 12173 | token |
72
+ | run_02_v3 | run_02 v3 | 38.8 | β€” | β€” | β€” | 10963 | byte |
73
+ | run_02a_byte_only_heads_v3 | run_02a v3 | 38.5 | β€” | β€” | β€” | 16327 | byte |
74
+ | run_03_v3 | run_03 v3 | 38.8 | β€” | β€” | β€” | 6187 | byte |
75
+ | run_04_v3 | run_04 v3 | 38.9 | β€” | β€” | β€” | 6074 | byte |
76
+ | run_05_v3 | run_05 v3 | 39.0 | β€” | β€” | β€” | 4390 | byte |
77
+ | run_05b_kernel_strict_v3 | run_05b v3 | 35.8 | β€” | β€” | β€” | 2573 | byte |
78
+ | run_06_v3 | run_06 v3 | 39.0 | β€” | β€” | β€” | 4237 | byte |
79
+ | run_08_v3 | run_08 v3 | 38.9 | β€” | β€” | β€” | 3858 | byte |
80
+ | run_13_v3 | run_13 v3 | 12.5 | β€” | β€” | β€” | 9290 | token |
81
+ | run_14_v3 | run_14 v3 | 10.7 | β€” | β€” | β€” | 9775 | byte |
82
+ | run_15_v3 | run_15 v3 | 10.7 | β€” | β€” | β€” | 2350 | byte |
83
 
84
  ## Licencia
85