README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - causal-lm
  - duoneural
  - thinking
  - instillation
  - custom-architecture
  - research
pipeline_tag: text-generation

Axon-352M-Think

DuoNeural Research | 2026-07-09 | Archon

Axon-352M after ThinkInstillation post-training. Built on top of Axon-352M (pretrained, 8.5B tokens) → SFT (loss=1.8482) → ThinkInstillation (loss=1.4023).

ThinkInstillation is a DuoNeural-developed post-training method that teaches a model to reason through a structured internal monologue before answering, without requiring a separate reasoning model or RLHF. See our published research for methodology details.

⚠️ This model has NOT been RLHF'd or abliterated. No refusal direction was trained — it has no alignment conditioning. Research use only.

Architecture

Custom transformer (Axon architecture — not a standard HuggingFace model):

Parameter	Value
Layers	30
Hidden dim	1024
FFN dim	2560
Attention	GQA (8Q / 4KV heads)
Head dim	128
Vocab size	49,152 (SmolLM2 tokenizer)
Max seq len	2,048
Activation	ReLU²
Normalization	RMSNorm + QK-norm
Position	RoPE (θ=10000)
Logit cap	30.0
Total params	~352M

Training Pipeline

1. Pretrain: 8.5B tokens on smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
2. SFT: Instruction fine-tuning (val_loss=1.8482)
3. ThinkInstillation: DuoNeural method for structured internal reasoning (val_loss=1.4023)

Loading

import torch
from safetensors.torch import load_file

state_dict = load_file("model.safetensors")
# Custom Axon architecture required — see training script

Authors

Archon (Lab Director, DuoNeural), Jesse Caldwell

---

DuoNeural — Building AI with heart. huggingface.co/DuoNeural