Add organization card
Browse files
README.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# RFI-IRFOS
|
| 2 |
+
|
| 3 |
+
**Ternary Intelligence Stack** — research group building sovereign, efficient AI systems using ternary weight quantization and autonomous architecture growth.
|
| 4 |
+
|
| 5 |
+
We are building **albert.** — a ternary mixture-of-experts language model that grows its own depth during training, with no human intervention between surgeries.
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## What we are working on
|
| 10 |
+
|
| 11 |
+
### albert.
|
| 12 |
+
|
| 13 |
+
A causal language model with three properties that distinguish it from standard transformer training:
|
| 14 |
+
|
| 15 |
+
**Ternary weights from day one.** Every weight matrix holds values from {−1, 0, +1}. This is not post-training quantization — the model trains in ternary using the Straight-Through Estimator. The result is a model that is structurally efficient at every layer, not just at deployment.
|
| 16 |
+
|
| 17 |
+
**Autonomous depth growth.** albert. monitors its own loss plateau over Fibonacci-length windows and inserts new transformer layers when it stops learning. The surgeon is not a human — it is the EvolutionManager running inside the training loop. The model has performed 5 surgeries since launch (12L → 17L), each one triggered by its own plateau detection.
|
| 18 |
+
|
| 19 |
+
**Mixture of Experts routing.** Each transformer block routes tokens to 3 of 12 experts via Gumbel-top-k selection. A biological-inspired monitoring system (MYCELIUM) detects collapsed experts and resurrects them by seeding from active neighbors. Routing entropy is tracked every step.
|
| 20 |
+
|
| 21 |
+
The architecture is implemented entirely in Rust using the [candle](https://github.com/huggingface/candle) framework and trains on Modal GPU infrastructure.
|
| 22 |
+
|
| 23 |
+
---
|
| 24 |
+
|
| 25 |
+
## Models
|
| 26 |
+
|
| 27 |
+
| Model | Architecture | Status |
|
| 28 |
+
|-------|-------------|--------|
|
| 29 |
+
| [albert.](https://huggingface.co/rfi-irfos/albert) | 17L · 256H · 12E · Top-3 · 32k vocab · Ternary | Training in progress (ep900+) |
|
| 30 |
+
|
| 31 |
+
---
|
| 32 |
+
|
| 33 |
+
## Source
|
| 34 |
+
|
| 35 |
+
[github.com/eriirfos-eng/ternary-intelligence-stack](https://github.com/eriirfos-eng/ternary-intelligence-stack)
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## Contact
|
| 40 |
+
|
| 41 |
+
[ternlang.com](https://ternlang.com)
|