Update README: recalculated metrics 2026-06-12
Browse files
README.md
CHANGED
|
@@ -31,11 +31,55 @@ Code: [github.com/Cukinator/1.58bits](https://github.com/Cukinator/1.58bits)
|
|
| 31 |
|
| 32 |
| Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Ξ uniform | P/s | Bytes/s | Nivel |
|
| 33 |
|-----|-------------|--------:|------------:|---------:|----:|----:|----------:|----:|--------:|-------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
## Small runs β 10M (round 1, 2 tok/param)
|
| 36 |
|
| 37 |
| Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Nivel |
|
| 38 |
|-----|-------------|--------:|------------:|---------:|----:|----:|-------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
## Licencia
|
| 41 |
|
|
|
|
| 31 |
|
| 32 |
| Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Ξ uniform | P/s | Bytes/s | Nivel |
|
| 33 |
|-----|-------------|--------:|------------:|---------:|----:|----:|----------:|----:|--------:|-------|
|
| 34 |
+
| run_01 | Transformer + BPE 16K + FP16 | 54.7 | 218.9 | 4.6648 | 106.14 | β | β | 78.0 | 78 | token |
|
| 35 |
+
| run_02a_byte_only_heads | Transformer + Byte + 4 heads (no LBD) | 38.5 | 154.0 | 2.3079 | 10.05 | 3.3295 | -3.2373 | 85.9 | 344 | byte |
|
| 36 |
+
| run_02 | Transformer + Byte + LocalByteDecoder | 38.8 | 155.3 | 1.7154 | 5.56 | 2.4748 | -3.8298 | 86.1 | 344 | byte |
|
| 37 |
+
| run_03 | MLGRU + Byte + FP16 | 38.8 | 155.3 | 1.8697 | 6.49 | 2.6974 | -3.6755 | 59.9 | 240 | byte |
|
| 38 |
+
| run_04 | MLGRU + Byte + Ternary | 38.9 | 155.4 | 5.5671 | 261.68 | 8.0317 | 0.0220 | 8.0 | 32 | byte |
|
| 39 |
+
| run_05 | + FPResidual | 39.0 | 156.0 | 5.5516 | 257.65 | 8.0093 | 0.0064 | 7.6 | 30 | byte |
|
| 40 |
+
| run_05b_kernel_strict | MLGRU kernel-strict (no W_o) | 35.8 | 143.4 | 5.5940 | 268.82 | 8.0705 | 0.0489 | 8.7 | 35 | byte |
|
| 41 |
+
| run_06 | + Bolmo patch embedding | 39.0 | 156.0 | 5.5561 | 258.81 | 8.0157 | 0.0109 | 7.8 | 31 | byte |
|
| 42 |
+
| run_07 | + DeleteGate (CPU-1 completo) β | 39.0 | 156.1 | 5.5561 | 258.81 | 8.0157 | 0.0109 | 7.8 | 31 | byte |
|
| 43 |
+
| run_08 | Folded Transformer + Byte + Ternary | 38.9 | 155.4 | 5.5621 | 260.37 | 8.0244 | 0.0169 | 8.0 | 32 | byte |
|
| 44 |
+
| run_09 | + PFNet | 39.4 | 157.7 | 5.5330 | 252.91 | 7.9825 | -0.0122 | 7.3 | 29 | byte |
|
| 45 |
+
| run_10 | + decay per-canal aprendido | 39.4 | 157.7 | 5.5337 | 253.08 | 7.9835 | -0.0115 | 7.1 | 28 | byte |
|
| 46 |
|
| 47 |
## Small runs β 10M (round 1, 2 tok/param)
|
| 48 |
|
| 49 |
| Run | Arquitectura | Params M | RAM fp32 MB | Val Loss | PPL | BPB | Nivel |
|
| 50 |
|-----|-------------|--------:|------------:|---------:|----:|----:|-------|
|
| 51 |
+
| run_13 | Small CPU-1 + BPE 4K | 12.5 | 50.2 | 30.5375 | 18293109063475.55 | β | token |
|
| 52 |
+
| run_14 | Small CPU-1 byte (Qwen distill.) | 10.7 | 42.8 | 5.5755 | 263.88 | 8.0437 | byte |
|
| 53 |
+
| run_15 | Small CPU-1 byte + hidden distill. | 10.7 | 42.9 | 5.5755 | 263.88 | 8.0437 | byte |
|
| 54 |
+
| run_16 | Small CPU-1 raw bytes, no teacher | 10.7 | 42.8 | 5.5754 | 263.85 | 8.0436 | byte |
|
| 55 |
+
|
| 56 |
+
## Round 2 β re-runs a 15 tok/param
|
| 57 |
+
|
| 58 |
+
| Run | Arquitectura | Params M | Loss r1 | PPL r1 | Loss r2 | PPL r2 | Ξ loss | BPB r2 | Nivel |
|
| 59 |
+
|-----|-------------|--------:|--------:|-------:|--------:|-------:|-------:|-------:|-------|
|
| 60 |
+
| run_04_r2 | run_04 @ 15 tok/param | 38.9 | 5.5671 | 261.68 | β | β | β | β | byte |
|
| 61 |
+
| run_07_r2 | run_07 @ 15 tok/param | 39.0 | 5.5561 | 258.81 | β | β | β | β | byte |
|
| 62 |
+
| run_13_r2 | run_13 @ 15 tok/param | 12.5 | 30.5375 | 18293109063475.55 | 25.4264 | 110288596599.28 | -5.11118 | β | token |
|
| 63 |
+
| run_14_r2 | run_14 @ 15 tok/param | 10.7 | 5.5755 | 263.88 | 5.5755 | 263.88 | -0.00000 | 8.0437 | byte |
|
| 64 |
+
| run_15_r2 | run_15 @ 15 tok/param | 10.7 | 5.5755 | 263.88 | 5.5755 | 263.88 | -0.00000 | 8.0437 | byte |
|
| 65 |
+
| run_16_r2 | run_16 @ 15 tok/param | 10.7 | 5.5754 | 263.85 | 5.5754 | 263.85 | +0.00000 | 8.0436 | byte |
|
| 66 |
+
|
| 67 |
+
## Round 3 β v3 runs (bf16 AMP + lrΓ2 + 50 tok/p)
|
| 68 |
+
|
| 69 |
+
| Run | Arquitectura | Params M | Val Loss | PPL | BPB | Step | Nivel |
|
| 70 |
+
|-----|-------------|--------:|--------:|----:|----:|-----:|-------|
|
| 71 |
+
| run_01_v3 | run_01 v3 | 54.7 | β | β | β | 12173 | token |
|
| 72 |
+
| run_02_v3 | run_02 v3 | 38.8 | β | β | β | 10963 | byte |
|
| 73 |
+
| run_02a_byte_only_heads_v3 | run_02a v3 | 38.5 | β | β | β | 16327 | byte |
|
| 74 |
+
| run_03_v3 | run_03 v3 | 38.8 | β | β | β | 6187 | byte |
|
| 75 |
+
| run_04_v3 | run_04 v3 | 38.9 | β | β | β | 6074 | byte |
|
| 76 |
+
| run_05_v3 | run_05 v3 | 39.0 | β | β | β | 4390 | byte |
|
| 77 |
+
| run_05b_kernel_strict_v3 | run_05b v3 | 35.8 | β | β | β | 2573 | byte |
|
| 78 |
+
| run_06_v3 | run_06 v3 | 39.0 | β | β | β | 4237 | byte |
|
| 79 |
+
| run_08_v3 | run_08 v3 | 38.9 | β | β | β | 3858 | byte |
|
| 80 |
+
| run_13_v3 | run_13 v3 | 12.5 | β | β | β | 9290 | token |
|
| 81 |
+
| run_14_v3 | run_14 v3 | 10.7 | β | β | β | 9775 | byte |
|
| 82 |
+
| run_15_v3 | run_15 v3 | 10.7 | β | β | β | 2350 | byte |
|
| 83 |
|
| 84 |
## Licencia
|
| 85 |
|