---
license: mit
library_name: pytorch
tags:
- causal-lm
- decoder-only
- reasoning
- pretraining
- think
- thought
- reasoning
- code
---

# Mini.Deep.Thinker.11m

A small decoder-only causal language model trained from scratch with deep chain-of-thought reasoning capabilities.

## Architecture

- Parameters: 11,004,896
- Context length: 1096
- Hidden size: 352
- Layers: 7
- Attention heads: 8
- Intermediate size: 1152
- Vocabulary size: 4,096

## Reasoning tokens

- `<|input|>`
- `<|think|>`
- `<|thought|>`
- `<|reasoning|>`
- `<|answer|>`

## Objective

True causal next-token prediction with strict per-example context limits.

## Context

Every training example is strictly limited to 1095 content tokens + EOS, padded to exactly 1096 positions. No oversized example is split across separate training examples.

## Example selection

Deterministic shuffled scan with SHA-256 content hash deduplication. Each session selects 20,000 new, unseen examples.

## Dataset

`Plans11/Organized_PreTrain_1k_Context`

## Resumability

Training state is persisted to Hugging Face, including:
- model.safetensors
- training_state.pt
- tokenizer files (tokenizer.json, tokenizer_config.json)
- config.json
- progress.json
- seen_examples.jsonl
- training_metadata.json

## Current progress

- Unique examples trained: 40,000
- Global optimizer steps: 1,250
- Last session: 20,000 examples
- Best session loss: 1.88992

This is an experimental small language model and is not guaranteed to produce factually or logically correct outputs.