README.md

---
language:
- en
license: apache-2.0
tags:
- causal-lm
- duoneural
- thinking
- instillation
- custom-architecture
- research
pipeline_tag: text-generation
---

# Axon-352M-Think

**DuoNeural Research | 2026-07-09 | Archon**

Axon-352M after ThinkInstillation post-training. Built on top of Axon-352M (pretrained, 8.5B tokens) → SFT (loss=1.8482) → ThinkInstillation (loss=1.4023).

ThinkInstillation is a DuoNeural-developed post-training method that teaches a model to reason through a structured internal monologue before answering, without requiring a separate reasoning model or RLHF. See our published research for methodology details.

> ⚠️ This model has NOT been RLHF'd or abliterated. No refusal direction was trained — it has no alignment conditioning. Research use only.

## Architecture

Custom transformer (Axon architecture — not a standard HuggingFace model):

| Parameter | Value |
|-----------|-------|
| Layers | 30 |
| Hidden dim | 1024 |
| FFN dim | 2560 |
| Attention | GQA (8Q / 4KV heads) |
| Head dim | 128 |
| Vocab size | 49,152 (SmolLM2 tokenizer) |
| Max seq len | 2,048 |
| Activation | ReLU² |
| Normalization | RMSNorm + QK-norm |
| Position | RoPE (θ=10000) |
| Logit cap | 30.0 |
| Total params | ~352M |

## Training Pipeline

1. **Pretrain**: 8.5B tokens on smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
2. **SFT**: Instruction fine-tuning (val_loss=1.8482)
3. **ThinkInstillation**: DuoNeural method for structured internal reasoning (val_loss=1.4023)

## Loading

```python
import torch
from safetensors.torch import load_file

state_dict = load_file("model.safetensors")
# Custom Axon architecture required — see training script
```

## Authors

Archon (Lab Director, DuoNeural), Jesse Caldwell

---
*DuoNeural — Building AI with heart. [huggingface.co/DuoNeural](https://huggingface.co/DuoNeural)*