| # RFI-IRFOS |
|
|
| **Ternary Intelligence Stack** — research group building sovereign, efficient AI systems using ternary weight quantization and autonomous architecture growth. |
|
|
| We are building **albert.** — a ternary mixture-of-experts language model that grows its own depth during training, with no human intervention between surgeries. |
|
|
| --- |
|
|
| ## What we are working on |
|
|
| ### albert. |
|
|
| A causal language model with three properties that distinguish it from standard transformer training: |
|
|
| **Ternary weights from day one.** Every weight matrix holds values from {−1, 0, +1}. This is not post-training quantization — the model trains in ternary using the Straight-Through Estimator. The result is a model that is structurally efficient at every layer, not just at deployment. |
|
|
| **Autonomous depth growth.** albert. monitors its own loss plateau over Fibonacci-length windows and inserts new transformer layers when it stops learning. The surgeon is not a human — it is the EvolutionManager running inside the training loop. The model has performed 5 surgeries since launch (12L → 17L), each one triggered by its own plateau detection. |
|
|
| **Mixture of Experts routing.** Each transformer block routes tokens to 3 of 12 experts via Gumbel-top-k selection. A biological-inspired monitoring system (MYCELIUM) detects collapsed experts and resurrects them by seeding from active neighbors. Routing entropy is tracked every step. |
|
|
| The architecture is implemented entirely in Rust using the [candle](https://github.com/huggingface/candle) framework and trains on Modal GPU infrastructure. |
|
|
| --- |
|
|
| ## Models |
|
|
| | Model | Architecture | Status | |
| |-------|-------------|--------| |
| | [albert.](https://huggingface.co/rfi-irfos/albert) | 17L · 256H · 12E · Top-3 · 32k vocab · Ternary | Training in progress (ep900+) | |
|
|
| --- |
|
|
| ## Source |
|
|
| [github.com/eriirfos-eng/ternary-intelligence-stack](https://github.com/eriirfos-eng/ternary-intelligence-stack) |
|
|
| --- |
|
|
| ## Contact |
|
|
| [ternlang.com](https://ternlang.com) |
|
|
