Website Status
Progress on our beta tester website
Beta Tester Website
In Development
Vocab Size
491
Architecture
FMN Transformer
Tokenization
Character-level
Availability
In Progress
Model Specifications (Subject to Change)
112
Model Dimension
6
Layers
4
Attention Heads
65K
Context Length
40
FMN Rank
120
Max Loops/Neuron
Feature Flags
Dynamic Routing
REINFORCE based
QK Normalization
Enabled
Gated Residuals
Enabled
Recurrent Mixer
Enabled
SwiGLU FFN
Disabled
Beta Access
We are working on a beta tester website so people can experience the pain of creating an AI with us.
View CompactAI on HuggingFace