---
license: mit
datasets:
- Amirjalaly/ooast_prompts
language:
- en
tags:
- DDM
- lightwieght
---
# TalosDDM

254,272 parameters
64 hidden size
2 layers
4 attention heads
2 KV heads
1,024 vocabulary
512 max sequence length
FP32
PyTorch 2.11.0+cu128
CUDA 12.8.1
NVIDIA Tesla T4

### Training Loss

Step 0 — 6.9177
Step 25 — 5.3568
Step 50 — 3.9134
Step 75 — 2.6434
Step 100 — 1.6195
Step 125 — 1.0475
Step 150 — 0.5186
Step 175 — 0.2652

### MoE Configuration

128 hidden size
3 layers
8 attention heads
4 KV heads
16 head dimension
2,048 vocabulary
16 experts
3 active experts
1 shared expert