rfi-irfos commited on
Commit
d851525
·
verified ·
1 Parent(s): a1d4e5e

Add organization card

Browse files
Files changed (1) hide show
  1. README.md +41 -0
README.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # RFI-IRFOS
2
+
3
+ **Ternary Intelligence Stack** — research group building sovereign, efficient AI systems using ternary weight quantization and autonomous architecture growth.
4
+
5
+ We are building **albert.** — a ternary mixture-of-experts language model that grows its own depth during training, with no human intervention between surgeries.
6
+
7
+ ---
8
+
9
+ ## What we are working on
10
+
11
+ ### albert.
12
+
13
+ A causal language model with three properties that distinguish it from standard transformer training:
14
+
15
+ **Ternary weights from day one.** Every weight matrix holds values from {−1, 0, +1}. This is not post-training quantization — the model trains in ternary using the Straight-Through Estimator. The result is a model that is structurally efficient at every layer, not just at deployment.
16
+
17
+ **Autonomous depth growth.** albert. monitors its own loss plateau over Fibonacci-length windows and inserts new transformer layers when it stops learning. The surgeon is not a human — it is the EvolutionManager running inside the training loop. The model has performed 5 surgeries since launch (12L → 17L), each one triggered by its own plateau detection.
18
+
19
+ **Mixture of Experts routing.** Each transformer block routes tokens to 3 of 12 experts via Gumbel-top-k selection. A biological-inspired monitoring system (MYCELIUM) detects collapsed experts and resurrects them by seeding from active neighbors. Routing entropy is tracked every step.
20
+
21
+ The architecture is implemented entirely in Rust using the [candle](https://github.com/huggingface/candle) framework and trains on Modal GPU infrastructure.
22
+
23
+ ---
24
+
25
+ ## Models
26
+
27
+ | Model | Architecture | Status |
28
+ |-------|-------------|--------|
29
+ | [albert.](https://huggingface.co/rfi-irfos/albert) | 17L · 256H · 12E · Top-3 · 32k vocab · Ternary | Training in progress (ep900+) |
30
+
31
+ ---
32
+
33
+ ## Source
34
+
35
+ [github.com/eriirfos-eng/ternary-intelligence-stack](https://github.com/eriirfos-eng/ternary-intelligence-stack)
36
+
37
+ ---
38
+
39
+ ## Contact
40
+
41
+ [ternlang.com](https://ternlang.com)