File size: 1,902 Bytes
d97f7ee | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
language:
- en
license: apache-2.0
tags:
- causal-lm
- duoneural
- thinking
- instillation
- custom-architecture
- research
pipeline_tag: text-generation
---
# Axon-352M-Think
**DuoNeural Research | 2026-07-09 | Archon**
Axon-352M after ThinkInstillation post-training. Built on top of Axon-352M (pretrained, 8.5B tokens) → SFT (loss=1.8482) → ThinkInstillation (loss=1.4023).
ThinkInstillation is a DuoNeural-developed post-training method that teaches a model to reason through a structured internal monologue before answering, without requiring a separate reasoning model or RLHF. See our published research for methodology details.
> ⚠️ This model has NOT been RLHF'd or abliterated. No refusal direction was trained — it has no alignment conditioning. Research use only.
## Architecture
Custom transformer (Axon architecture — not a standard HuggingFace model):
| Parameter | Value |
|-----------|-------|
| Layers | 30 |
| Hidden dim | 1024 |
| FFN dim | 2560 |
| Attention | GQA (8Q / 4KV heads) |
| Head dim | 128 |
| Vocab size | 49,152 (SmolLM2 tokenizer) |
| Max seq len | 2,048 |
| Activation | ReLU² |
| Normalization | RMSNorm + QK-norm |
| Position | RoPE (θ=10000) |
| Logit cap | 30.0 |
| Total params | ~352M |
## Training Pipeline
1. **Pretrain**: 8.5B tokens on smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
2. **SFT**: Instruction fine-tuning (val_loss=1.8482)
3. **ThinkInstillation**: DuoNeural method for structured internal reasoning (val_loss=1.4023)
## Loading
```python
import torch
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
# Custom Axon architecture required — see training script
```
## Authors
Archon (Lab Director, DuoNeural), Jesse Caldwell
---
*DuoNeural — Building AI with heart. [huggingface.co/DuoNeural](https://huggingface.co/DuoNeural)*
|