---
license: mit
library_name: pytorch
tags:
- causal-lm
- decoder-only
- reasoning
- pretraining
- think
- thought
- reasoning
- code
---

# Nano.Deep.Reasoner.11m

A small decoder-only causal language model trained from scratch.

## Architecture

- Parameters: 11,229,120
- Context length: 1096
- Hidden size: 360
- Layers: 6
- Attention heads: 8
- Intermediate size: 1440
- Vocabulary size: 4,096

## Reasoning tokens

- `<|input|>`
- `<|think|>`
- `<|thought|>`
- `<|reasoning|>`
- `<|answer|>`

## Objective

True causal next-token prediction.

For an input sequence:

`x[0], x[1], x[2], ...`

the model learns:

`x[0] -> x[1]`

`x[1] -> x[2]`

`x[2] -> x[3]`

and so on.

## Context

Every individual training example is strictly limited to:

`1095` content tokens + EOS

and padded to exactly `1096` positions.

No oversized example is intentionally split across separate training examples.

## Example selection

Starting from Session 3, new examples are selected via a deterministic
shuffled scan of the dataset (seeded, reproducible across runs) rather than
raw sequential order, to avoid category/source concentration within a
session. Sessions 1-2 (40,000 examples) were selected sequentially before
this correction and remain part of the trained corpus.

## Dataset

`Plans11/Organized_PreTrain_1k_Context`

Categories exposed by the dataset include:

- Think
- Thought
- Reasoning
- Instruct
- Chat
- Tool_Calling
- Code_Instruct

## Resumability

Training state is persisted to Hugging Face, including:

- model.safetensors
- training_state.pt
- tokenizer files
- config.json
- progress.json
- seen_examples.jsonl
- training_metadata.json

Examples are tracked using SHA-256 content hashes.

The tokenizer becomes immutable after its initial creation.

## Current progress

- Unique examples trained: 60,000
- Global optimizer steps: 1,441
- Last session: 20,000

This is an experimental small language model and is not guaranteed to
produce factually or logically correct outputs.