license: mit
library_name: pytorch
tags:
- causal-lm
- decoder-only
- reasoning
- pretraining
- think
- thought
- reasoning
- code
Nano.Deep.Reasoner.11m
A small decoder-only causal language model trained from scratch.
Architecture
- Parameters: 11,229,120
- Context length: 1096
- Hidden size: 360
- Layers: 6
- Attention heads: 8
- Intermediate size: 1440
- Vocabulary size: 4,096
Reasoning tokens
<|input|><|think|><|thought|><|reasoning|><|answer|>
Objective
True causal next-token prediction.
For an input sequence:
x[0], x[1], x[2], ...
the model learns:
x[0] -> x[1]
x[1] -> x[2]
x[2] -> x[3]
and so on.
Context
Every individual training example is strictly limited to:
1095 content tokens + EOS
and padded to exactly 1096 positions.
No oversized example is intentionally split across separate training examples.
Example selection
Starting from Session 3, new examples are selected via a deterministic shuffled scan of the dataset (seeded, reproducible across runs) rather than raw sequential order, to avoid category/source concentration within a session. Sessions 1-2 (40,000 examples) were selected sequentially before this correction and remain part of the trained corpus.
Dataset
Plans11/Organized_PreTrain_1k_Context
Categories exposed by the dataset include:
- Think
- Thought
- Reasoning
- Instruct
- Chat
- Tool_Calling
- Code_Instruct
Resumability
Training state is persisted to Hugging Face, including:
- model.safetensors
- training_state.pt
- tokenizer files
- config.json
- progress.json
- seen_examples.jsonl
- training_metadata.json
Examples are tracked using SHA-256 content hashes.
The tokenizer becomes immutable after its initial creation.
Current progress
- Unique examples trained: 60,000
- Global optimizer steps: 1,441
- Last session: 20,000
This is an experimental small language model and is not guaranteed to produce factually or logically correct outputs.