| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - causal-lm |
| - decoder-only |
| - reasoning |
| - pretraining |
| - think |
| - thought |
| - reasoning |
| - code |
| --- |
| |
| # Nano.Deep.Reasoner.11m |
|
|
| A small decoder-only causal language model trained from scratch. |
|
|
| ## Architecture |
|
|
| - Parameters: 11,229,120 |
| - Context length: 1096 |
| - Hidden size: 360 |
| - Layers: 6 |
| - Attention heads: 8 |
| - Intermediate size: 1440 |
| - Vocabulary size: 4,096 |
|
|
| ## Reasoning tokens |
|
|
| - `<|input|>` |
| - `<|think|>` |
| - `<|thought|>` |
| - `<|reasoning|>` |
| - `<|answer|>` |
|
|
| ## Objective |
|
|
| True causal next-token prediction. |
|
|
| For an input sequence: |
|
|
| `x[0], x[1], x[2], ...` |
|
|
| the model learns: |
|
|
| `x[0] -> x[1]` |
|
|
| `x[1] -> x[2]` |
|
|
| `x[2] -> x[3]` |
|
|
| and so on. |
|
|
| ## Context |
|
|
| Every individual training example is strictly limited to: |
|
|
| `1095` content tokens + EOS |
|
|
| and padded to exactly `1096` positions. |
|
|
| No oversized example is intentionally split across separate training examples. |
|
|
| ## Example selection |
|
|
| Starting from Session 3, new examples are selected via a deterministic |
| shuffled scan of the dataset (seeded, reproducible across runs) rather than |
| raw sequential order, to avoid category/source concentration within a |
| session. Sessions 1-2 (40,000 examples) were selected sequentially before |
| this correction and remain part of the trained corpus. |
|
|
| ## Dataset |
|
|
| `Plans11/Organized_PreTrain_1k_Context` |
|
|
| Categories exposed by the dataset include: |
|
|
| - Think |
| - Thought |
| - Reasoning |
| - Instruct |
| - Chat |
| - Tool_Calling |
| - Code_Instruct |
|
|
| ## Resumability |
|
|
| Training state is persisted to Hugging Face, including: |
|
|
| - model.safetensors |
| - training_state.pt |
| - tokenizer files |
| - config.json |
| - progress.json |
| - seen_examples.jsonl |
| - training_metadata.json |
| |
| Examples are tracked using SHA-256 content hashes. |
| |
| The tokenizer becomes immutable after its initial creation. |
| |
| ## Current progress |
| |
| - Unique examples trained: 60,000 |
| - Global optimizer steps: 1,441 |
| - Last session: 20,000 |
| |
| This is an experimental small language model and is not guaranteed to |
| produce factually or logically correct outputs. |
| |
