Berna R8 — Cell-Based Language Model
Built on Berna R7 v3. Includes Bounded Forgetting experiments.
R8-v2 (Current, from scratch)
- Training: 10,200 steps, eng+math
- Architecture: 28L × 1536d, 1.2B params, 5,000 cell cap
- lr: 1.5e-4, accum: 4, measure_every: 500
- Best loss: ~4.0
- Cells: ~4,500 (capped at 5,000)
- Births: ~360
R8-v1 (First attempt, stopped)
- Best loss: 5.49 (plateaued)
- Cells: 5,000 (hit cap)
- Issue: lr=3e-4 unstable
H2 Experiment — Bounded Forgetting (CORRECTED)
- Base: R7 v3 (step_1400)
- Fine-tune: 2,000 steps on German
- Results:
Metric Value ENG before 4.078 ENG after 4.481 F_eng 0.403 DE before 4.741 DE after 2.678 - F_eng = 0.403 (10% forgetting, bounded)
Fixes over R7 v3
- K2/K5/K6 per-cell
- Plexus threshold 0.3
- Cell cap = 5,000 (anti-explosion)
- SmartFreezer active
License
CC BY-NC 4.0 — research use only.
Citation
@misc{berna_r8_2026, author = {Kamil, Muhamed}, title = {Berna R8}, year = {2026}}