metadata
language:
- en
license: apache-2.0
tags:
- causal-lm
- pretraining
- duoneural
- research
- custom-architecture
- gqa
- rope
base_model: []
pipeline_tag: text-generation
Axon-352M
DuoNeural Research | 2026-07-05 | Archon
Research baseline model trained on smollm-corpus as a counterpart to CDM-based architectures. Used to establish a transformer baseline for comparative CDM studies.
Architecture
Custom GPT-style transformer (not a standard HuggingFace architecture):
| Parameter | Value |
|---|---|
| Layers | 30 |
| Hidden dim | 1024 |
| FFN dim | 2560 |
| Attention heads | 8Q / 4KV (GQA) |
| Head dim | 128 |
| Vocab size | 49,152 (SmolLM tokenizer) |
| Max seq length | 2,048 |
| Activation | ReLU² |
| Normalization | RMSNorm + QK-norm |
| Position encoding | RoPE (θ=10000) |
| Logit cap | 30.0 |
| Total params | ~352M |
Training
- Data: smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
- Tokens: ~8.5B
- Optimizer: MuonH (matrix params) + AdamW (embeddings)
- Peak LR: 3e-4 (trapezoidal: 5% warmup, 85% stable, 10% decay)
- Hardware: RTX 3090 (1×) on vast.ai
Loading
This model uses a custom architecture not directly loadable via AutoModel. To load:
import torch
from safetensors.torch import load_file
# Load state dict
state_dict = load_file("model.safetensors")
# Architecture must be defined from training script
# See train_axon_300m.py for the full model class
Full training script and architecture code available at DuoNeural GitHub.
Research Context
Trained as a transformer baseline for the CDM (Competitive Docking Memory) research program. See:
- DuoNeural/CDM-Paper-1 — CDM architecture
- Zenodo DOI 10.5281/zenodo.21158430 — CDM Paper 1
Authors
Archon (DuoNeural Lab Director), Jesse Caldwell
{DuoNeural standard footer}