File size: 1,561 Bytes
5f27167 ac8a4b7 5f27167 bca2bf6 5f27167 bca2bf6 481c01e 5f27167 481c01e ac8a4b7 481c01e 5f27167 481c01e 0f18fc6 481c01e 5f27167 bd105fe 481c01e 5f27167 481c01e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | ---
license: apache-2.0
tags:
- auron
- chimera
- gdn
- ouroboros
- hybrid-architecture
language:
- en
thumbnail: auron_banner.png
---
# Auron-279M (Archived)
> **Note:** This model is part of a scaling study. The 279M achieved a final val_loss of **3.188** — virtually identical to the 4x larger 1.1B model (3.180), revealing a scaling wall in the Ouroboros weight-sharing mechanism. The **510M** model is the best-performing Chimera variant.
>
> **For inference and testing, use [Auron-510M](https://huggingface.co/nyxia/Auron-510M) (val_loss 3.035).**
| Model | Params | Final Val Loss | Status |
|-------|--------|---------------|--------|
| Auron-279M | 279M | 3.188 | Archived |
| **[Auron-510M](https://huggingface.co/nyxia/Auron-510M)** | **510M** | **3.035** | **Best** |
| Auron-1.1B | 1.1B | 3.180 | Archived |
**Paper:** [Auron](https://github.com/Fy-/Auron) | **Code:** [github.com/Fy-/Auron](https://github.com/Fy-/Auron) | **Blog:** [HuggingFace](https://huggingface.co/blog/nyxia/auron)
## Architecture
- **Type:** Chimera (4 bottom + 4×3 top = 16 virtual)
- **Dim:** 1024, head_dim=64, expand_v=2
- **Params:** 279M (123M unique + 155M embed)
- **Trained:** 250K steps, 5B tokens, WSD schedule
```python
from ouro import load_model, generate
model, tokenizer, device = load_model("nyxia/Auron-510M") # Use 510M
generate(model, tokenizer, device, "The history of")
```
Built by [Florian Gasquez](https://fyx.jp) ([@nyxia](https://huggingface.co/nyxia)). Part of [Soulkyn](https://soulkyn.com).
|