Fix training hyperparameters: seq=2048, effective batch=8 (was incorrectly listed as 512/32)

README.md

@@ -1,5 +1,311 @@
----
-license: other
-license_name: mother-ai-beta
-license_link: LICENSE
----
+---
+license: other
+license_name: msai-sovereign
+license_link: LICENSE
+language:
+  - en
+  - cy
+  - ga
+  - gd
+tags:
+  - sovereign-ai
+  - uk
+  - reasoning
+  - msai
+  - mother-core
+pipeline_tag: text-generation
+library_name: pytorch
+---
+
+# MOTHER CORE V2 — chunk 450 (W2.7)
+
+**Sovereign UK AI** built from scratch by **MediaStream AI Limited (MSAI)**.
+
+This is a development checkpoint released for **MSAI team and partner testing only**. It is **not** a released model and **not** intended for production use. Eval performance is partial; the model is mid-training.
+
+---
+
+## 1. Model Summary
+
+| Field | Value |
+|---|---|
+| Model | MOTHER CORE V2 |
+| Checkpoint | chunk 450 (W2.7 stage) |
+| Parameters | 6.877B |
+| Architecture | Custom transformer (RoPE, GQA, RMSNorm, SwiGLU FFN, memory gate) |
+| Layers | 48 |
+| Hidden dimension | 3,072 |
+| Attention heads | 24 (head_dim 128) |
+| KV heads | 6 (GQA ratio 4:1) |
+| FFN multiplier | 4.0 (intermediate 12,288) |
+| Max sequence length | 4,096 |
+| Vocabulary | 50,258 (SentencePiece) |
+| RoPE θ | 10,000 |
+| RMSNorm ε | 1e-5 |
+| Tied embeddings | No (separate `lm_head`) |
+| Weights dtype (this release) | bfloat16 |
+| Training dtype | float32 |
+
+This is a **from-scratch sovereign build**. It is not a fine-tune of any external model (Llama, Qwen, Mistral, GPT, etc.). Training, tokenisation, architecture, and corpus are all proprietary to MSAI.
+
+---
+
+## 2. Status
+
+| Metric | Value |
+|---|---|
+| Training stage | W2.7 (mid-curriculum) |
+| Most recent chunk eval | 47/105 @ chunk 450 |
+| Scope | math, science, reasoning, chain-of-thought, UK knowledge, Celtic languages, MOTHER identity |
+| Out of scope (separate future models) | code generation, creative writing, vision |
+
+This release is for **internal team testing**. It will fail on tasks outside its training scope.
+
+The training trajectory has been monotonic since chunk 300:
+
+| Chunk | Eval | Loss |
+|---|---|---|
+| 300 | 36/105 | 2.47 |
+| 350 | 37/105 | 2.05 |
+| 400 | 45/105 | 2.01 |
+| **450** | **47/105** | **1.74** |
+
+W2.7 will continue to chunk 650, after which the W2.8 corpus addition (~330,000 records spanning agentic orchestration, multi-step reasoning, tool use, memory synthesis) will be merged for the next training phase.
+
+---
+
+## 3. Locked Inference Rules
+
+**Deviation from these rules produces incorrect or degenerate output.** They are not suggestions — they are the inference recipe the model was trained against.
+
+| Setting | Value | Reason |
+|---|---|---|
+| Prompt format | `Question:\n\n{question}\n\nAnswer:` | Exact whitespace. Model is OOD without it. |
+| BOS token | id=1, `<s>` | Always prepended; model was trained with BOS at position 0 |
+| EOS token | id=2, `</s>` | Stop generation on emission |
+| PAD token | id=0, `<pad>` | Training only |
+| Sampling | **Greedy argmax** | No temperature, no top-k, no top-p |
+| Repetition penalty | 1.3 (frequency-scaled, count ≥ 2) | Higher values collapse output |
+| n-gram blocking | 4-gram, no repeat | Prevents loop output |
+| Max new tokens | 200 | Hard cap |
+| BOS in output | Banned | Never emit BOS during generation |
+| EOS in output | Allowed after first token | Early stop signal |
+
+### Reference code
+
+A working reference is included as `inference.py` in this repo. The canonical implementation lives in `mother_train_7b.py::_generate_greedy()` in the MSAI training repository. **Use `inference.py` from this repo or load `mother_train_7b._generate_greedy` directly.** Re-implementations frequently get the recipe wrong.
+
+---
+
+## 4. Architecture Detail
+
+```
+MotherCoreModel
+├── tok_emb           [50258, 3072]
+├── blocks × 48
+│   └── each:
+│       ├── attn (GQA)
+│       │   ├── wq    [3072, 3072]     # 24 heads × 128 dim
+│       │   ├── wk    [768,  3072]     # 6 KV heads × 128 dim
+│       │   ├── wv    [768,  3072]
+│       │   └── wo    [3072, 3072]
+│       ├── ff (SwiGLU)
+│       │   ├── w1    [12288, 3072]
+│       │   ├── w2    [12288, 3072]
+│       │   └── w3    [3072, 12288]
+│       ├── norm_attn (RMSNorm)
+│       └── norm_ff   (RMSNorm)
+├── norm_f            [3072]
+├── lm_head           [50258, 3072]   # NOT tied to tok_emb
+└── memory_gate       [1, 3072] + bias[1]
+```
+
+### Memory gate
+
+`memory_gate` is a sigmoid-gated single-dimension projection from the last hidden state. It is **trained but not active in inference output** — it is reserved for downstream integration with MOTHER ROBOTICS (an item/object/situational/historical awareness model) and external memory systems. Its activation is exposed in the forward pass return dict but does not affect token logits.
+
+Forward return:
+```
+{
+  "logits":             [B, T, vocab],
+  "loss":               scalar or None,
+  "aux_loss":           scalar (MoE; unused here, fixed=0),
+  "past_key_values":    List[(K,V)] or None,
+  "hidden_states":      List[Tensor] or None,
+  "last_hidden_state":  [B, T, dim],
+  "gate":               [B, 1]              ← detached, FYI only
+}
+```
+
+---
+
+## 5. Training
+
+### Corpus (W2.7)
+
+| Category | Records |
+|---|---|
+| Reasoning + chain-of-thought | ~390,000 |
+| UK general knowledge | ~210,000 |
+| Math & arithmetic (digit-spaced) | ~165,000 |
+| Identity & self-knowledge (MOTHER, MSAI) | ~32,000 |
+| Celtic languages (Welsh, Irish, Scottish Gaelic) | ~28,000 |
+| Science | ~88,000 |
+| Misc (chat, instruct skeleton) | ~135,000 |
+| **Total** | **~1.05M** |
+
+### Hyperparameters
+
+| Setting | Value |
+|---|---|
+| Learning rate | 1e-5 |
+| Gradient clip | 10.0 |
+| Effective batch size | 8 (BATCH_PHYSICAL=1 × GRAD_ACCUM_STEPS=8) |
+| Sequence length (training) | 2048 |
+| Optimiser | AdamW (β₁=0.9, β₂=0.95) |
+| Weight decay | 0.1 |
+| Warmup steps | 100 |
+| Layer-wise LR scaling | from chunk 10 onward |
+| Hardware | NVIDIA GB10 Blackwell (Grace–Blackwell unified memory, 128GB) |
+| Training site | MSAI Wright Avenue, Dundee — sovereign UK infrastructure |
+
+Training was performed at sequence length **2048** using physical microbatches of 1 with gradient accumulation of 8 (effective batch = 8). The architecture supports 4,096-token inference; 2048 → 4096 is a modest RoPE extrapolation, but long-context behaviour at full 4096 has not been benchmarked at this checkpoint.
+
+---
+
+## 6. Sovereign Build Posture
+
+MOTHER CORE is part of MSAI's sovereign AI stack — built end-to-end in the UK on UK-resident infrastructure. The training, weights, tokeniser, and corpus are owned by MSAI. The training datacentres are MSAI-operated (Wright Avenue, Dundee; with additional sites in Durham and Manchester). No US cloud provider is in the inference or training path.
+
+This positioning matters for UK government, defence, and regulated-enterprise customers where data residency, GDPR, and supply-chain provenance are mandatory.
+
+---
+
+## 7. Intended Use & Out-of-Scope Use
+
+**In scope (this checkpoint):**
+- Reasoning and chain-of-thought tasks at modest difficulty
+- UK general knowledge questions
+- Welsh / Irish / Scottish Gaelic short-form questions
+- MOTHER-identity Q&A
+- Arithmetic on small integers (with digit-spaced inputs for ≥3-digit numbers)
+
+**Out of scope (this checkpoint):**
+- Code generation (separate model — MOTHER CODE — planned)
+- Creative writing (separate model — MOTHER LLM — planned)
+- Long-form (>1,000 token) generation
+- Multi-turn dialogue (training is single-turn Q/A)
+- Anything safety-critical, medical, legal, or financial advisory
+- Real-time information (model has no internet access at inference)
+
+---
+
+## 8. Evaluation
+
+The internal eval suite at chunk 450 scores **47/105 (44.8%)** across:
+
+- Identity: 6/6 (100%)
+- UK knowledge: 9/12
+- Reasoning (multi-step): 14/35
+- Arithmetic: 5/15
+- Science: 7/12
+- Celtic languages: 4/9
+- Chain-of-thought: 2/16
+
+Persistent gaps at chunk 450:
+- Arithmetic on multi-digit numbers (training fix in progress — see W2.8 plan)
+- Multi-step reasoning beyond 3 hops
+- Welsh and Irish (smaller corpus volume than other categories)
+
+Eval suite and methodology are MSAI-internal. Comparable public benchmarks (MMLU, GSM8K) have **not** been run against this checkpoint and would not be directly comparable since the training corpus and tokeniser are sovereign.
+
+---
+
+## 9. Limitations & Known Failure Modes
+
+1. **Single-turn only** — no chat-style multi-turn coherence
+2. **Format-brittle** — the `Question:\n\n...\n\nAnswer:` template is required; other formats produce OOD output
+3. **No tool use / no agent loop** at this checkpoint (W2.8 corpus will add this)
+4. **No code generation** — even simple Python will fail; not in scope
+5. **No retrieval / no internet** — closed-book knowledge only, as of training cutoff
+6. **Arithmetic at multi-digit numbers** — requires digit-spaced input (`1 5 + 2 7`) to perform reliably
+7. **`weights_only=False` required** if loading from `.pt` — this repo ships `.safetensors` instead which is safer
+8. **High repetition penalty (>1.4) collapses output** — stick to 1.3
+
+---
+
+## 10. Usage
+
+### Quick test from a clean Python environment
+
+```bash
+pip install torch safetensors sentencepiece huggingface_hub
+```
+
+You also need the `mother_core` package source available (architecture is custom; no Transformers integration yet). Clone the MSAI training repo or copy `mother_core/` into your `PYTHONPATH`.
+
+```python
+from huggingface_hub import snapshot_download
+repo_dir = snapshot_download(repo_id="MediaStreamAI/MOTHER_CORE_V2")
+# Then import inference.py from the snapshot
+import sys, importlib.util
+spec = importlib.util.spec_from_file_location("inf", f"{repo_dir}/inference.py")
+inf = importlib.util.module_from_spec(spec); spec.loader.exec_module(inf)
+
+model, tok = inf.load_model_and_tokenizer(repo_dir)
+print(inf.generate_greedy(model, tok, "What is the capital of Scotland?"))
+```
+
+Or run the inference script directly:
+
+```bash
+python inference.py "What is the capital of Scotland?"
+```
+
+### File map
+
+| File | Purpose |
+|---|---|
+| `model-00001-of-00003.safetensors` | Weights, shard 1/3 |
+| `model-00002-of-00003.safetensors` | Weights, shard 2/3 |
+| `model-00003-of-00003.safetensors` | Weights, shard 3/3 |
+| `model.safetensors.index.json` | Shard index |
+| `config.json` | Architecture spec |
+| `tokenizer.model` | SentencePiece vocab |
+| `tokenizer_config.json` | Tokeniser config (`add_bos_token=true` required) |
+| `special_tokens_map.json` | BOS/EOS/PAD/UNK ids |
+| `inference.py` | Reference inference with locked rules |
+| `README.md` | This file |
+
+---
+
+## 11. License
+
+**MSAI Sovereign License — Internal & Partner Use Only.**
+
+This model is the proprietary work of MediaStream AI Limited. It is released to authorised team members and contracted partners for evaluation and integration purposes. Redistribution, commercial use, or training other models on this model's outputs require written permission from MSAI.
+
+For licensing enquiries: contact MediaStream AI Limited via the company website.
+
+---
+
+## 12. Citation
+
+```
+@misc{msai-mother-core-2026,
+  title  = {MOTHER CORE V2 — Sovereign UK AI},
+  author = {{MediaStream AI Limited}},
+  year   = {2026},
+  note   = {Chunk 450, W2.7 mid-training checkpoint},
+  url    = {https://huggingface.co/MediaStreamAI/MOTHER_CORE_V2}
+}
+```
+
+---
+
+## 13. Contact
+
+- Organisation: MediaStream AI Limited (MSAI)
+- Founder & CEO: Christopher Kenna
+- Web: https://mediastreamai.com
+- Infrastructure: UK sovereign (Dundee, Durham, Manchester)