Nano.Deep.Reasoner.11m | 20,000 new examples | step 816

README.md

@@ -5,31 +5,28 @@ tags:
 - causal-lm
 - decoder-only
 - reasoning
-- chain-of-thought
+- pretraining
 - think
 - thought
 - reasoning
-- pretraining
+- code
 ---
 
 # Nano.Deep.Reasoner.11m
 
-A decoder-only language model trained completely from scratch.
+A small decoder-only causal language model trained from scratch.
 
 ## Architecture
 
 - Parameters: 11,229,120
-- Context length: 1096 tokens
+- Context length: 1096
 - Hidden size: 360
 - Layers: 6
 - Attention heads: 8
 - Intermediate size: 1440
-- Vocabulary: 4,096
-- Weight tying: enabled
+- Vocabulary size: 4,096
 
-## Core reasoning format
-
-The model vocabulary contains dedicated reasoning control tokens:
+## Reasoning tokens
 
 - `<|input|>`
 - `<|think|>`
@@ -37,33 +34,69 @@ The model vocabulary contains dedicated reasoning control tokens:
 - `<|reasoning|>`
 - `<|answer|>`
 
-These tokens are part of the model vocabulary and training representation.
+## Objective
+
+True causal next-token prediction.
+
+For an input sequence:
+
+`x[0], x[1], x[2], ...`
+
+the model learns:
+
+`x[0] -> x[1]`
+
+`x[1] -> x[2]`
+
+`x[2] -> x[3]`
+
+and so on.
+
+## Context
+
+Every individual training example is strictly limited to:
+
+`1095` content tokens + EOS
+
+and padded to exactly `1096` positions.
 
-## Training
+No oversized example is intentionally split across separate training examples.
 
-Dataset:
+## Dataset
 
 `Plans11/Organized_PreTrain_1k_Context`
 
-Training is performed in sessions of up to:
+Categories exposed by the dataset include:
 
-`20,000`
+- Think
+- Thought
+- Reasoning
+- Instruct
+- Chat
+- Tool_Calling
+- Code_Instruct
 
-new examples per session.
+## Resumability
 
-Examples are tracked using SHA-256 content hashes so that dataset
-reordering does not intentionally cause the same example to be trained twice.
+Training state is persisted to Hugging Face, including:
 
-Current progress:
+- model.safetensors
+- training_state.pt
+- tokenizer files
+- config.json
+- progress.json
+- seen_examples.jsonl
+- training_metadata.json
 
-- Unique examples trained: 20,000
-- Global optimizer steps: 191
+Examples are tracked using SHA-256 content hashes.
 
-## Important
+The tokenizer becomes immutable after its initial creation.
 
-This model is a small experimental reasoning language model.
-The reasoning-token architecture does not guarantee factual correctness,
-logical correctness, or reliable reasoning.
+## Current progress
 
-The model was trained with causal next-token prediction.
+- Unique examples trained: 40,000
+- Global optimizer steps: 816
+- Last session: 20,000
 
+This is an experimental small language model and is not guaranteed to
+produce factually or logically correct outputs.

config.json

@@ -28,6 +28,7 @@
     "dataset": "Plans11/Organized_PreTrain_1k_Context",
     "session_examples": 20000,
     "context_length": 1096,
+    "strict_max_example_tokens": 1095,
     "causal_next_token_prediction": true,
     "reasoning_tokens": [
       "<|think|>",

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ea21473b0a0a15bb675da9f2eb36ba87f844b388ec6877ed1ffa10d94fce946c
+oid sha256:7f9dc4e6787f4d1bf45204fe2544a66799c37ad80d173f3cb7fc600adb02910f
 size 50821608

progress.json

@@ -1,14 +1,16 @@
 {
-  "model_name": "Nano.Deep.Reasoner.11m",
-  "dataset_repo": "Plans11/Organized_PreTrain_1k_Context",
+  "model": "Nano.Deep.Reasoner.11m",
   "model_repo": "ITLL/Nano.Deep.Reasoner.11m",
+  "dataset_repo": "Plans11/Organized_PreTrain_1k_Context",
   "context_length": 1096,
-  "session_size": 20000,
-  "global_step": 191,
-  "unique_examples_completed": 20000,
-  "last_session_examples": 20000,
-  "remaining_estimate": 2693413,
-  "tokenizer_vocab_size": 4096,
+  "vocab_size": 4096,
   "parameter_count": 11229120,
-  "updated_at_unix": 1787132858.6197612
+  "global_step": 816,
+  "unique_examples_completed": 40000,
+  "last_session_examples": 20000,
+  "max_session_examples": 20000,
+  "timestamp": 1787134475.6632547,
+  "causal_next_token": true,
+  "strict_example_context": true,
+  "tokenizer_immutable": true
 }

training_metadata.json

@@ -3,10 +3,11 @@
   "architecture": "decoder-only",
   "parameters": 11229120,
   "context_length": 1096,
+  "strict_example_limit": 1095,
   "vocab_size": 4096,
   "dataset": "Plans11/Organized_PreTrain_1k_Context",
-  "global_step": 191,
-  "unique_examples_trained": 20000,
+  "global_step": 816,
+  "unique_examples_trained": 40000,
   "last_session_size": 20000,
   "learning_rate": 0.0003,
   "batch_size": 8,
@@ -18,5 +19,8 @@
     "<|thought|>",
     "<|reasoning|>",
     "<|answer|>"
-  ]
+  ],
+  "causal_objective": "next_token_prediction",
+  "tokenizer_immutable": true,
+  "content_hash_deduplication": true
 }

training_state.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a2c80b31fdca557676deadae88ae0e3e5f92df1ddfa967c92657a51b7bef7b7
-size 134839931
+oid sha256:5ce8251ce45a4da4d787297cc8173504f30550cf68462a16e9615fec20a88819
+size 134840827