Model card: SFT v12.2 LoRA assistant release (step-12000) — content update, design unchanged

README.md

@@ -39,7 +39,25 @@ pipeline_tag: text-generation
 
 ---
 
-## Current release: NeuralAI the assistant — SFT v11 (LoRA step-12000)
+## Current release: NeuralAI the assistant — SFT v12.2 (LoRA step-12000)
+
+| Property | Value |
+| --- | --- |
+| **What it is** | Instruction-tuned assistant checkpoint with tool use (`search`, `fetch`, `read_file`) and working plain chat (greetings, identity, small talk), built from the NEA1 pretrain line |
+| **Base** | Pretrain step-100000. Same 100.1M architecture |
+| **Method** | LoRA rank 32 on attention + MLP with the base weights frozen; adapters merged into a standard checkpoint. Same recipe as the v9–v11 gate winners — v12.2 changes the data, not the method |
+| **SFT data** | 22,290 rows: the full v11 set (15,831) as replay plus 6,459 new rows in targeted fix banks — arithmetic drills, memory recall and honesty, two-part questions, plain-chat corrections, a redesigned date bank (1,220 rows: the date given in the conversation stated back in full — immediately, after a greeting, after distractors, and against event-date decoys — plus tomorrow/yesterday rows and no-date honesty rows), and capital-decoy reinforcement with Australia weighted. Every checkable answer is machine-verified |
+| **Training** | LoRA on Kaggle T4, adapter LR peak 1e-4 cosine, 12,000 steps (final train loss 2.8408); step-12000 selected by the gate below |
+| **Golden-40 tool-use gate** | **35.0/40 (87.5%) — PASS** (bar 60%). Tool calls 38/40, grounded answers 32/40. Report: `evals/eval_sftv12_2lora_step12000_golden40.json` |
+| **Fluency guardrail** | PPL **48.1** on the 50-doc guardrail set vs **47.59** for the step-100000 base on the same set. This is a different, smaller set than the 2000-doc set used below — the numbers are not comparable across sets |
+| **Plain chat + conversation** | v12.2's headline is dates. Date questions were the previous build's blocker (3/8 on the date8 gate set); the redesigned date bank takes date8 to **7/8** with both no-bracket honesty probes passing — the model states the date given in the conversation back in full, and says so when none was given instead of inventing one. The one remaining miss is relative arithmetic across a month boundary ("tomorrow" after August 31 repeats August 31 instead of September 1), a v13 item. Everything else holds: multiturn12 **11/12** with zero parroting and the Australia follow-up back to Canberra, memory6 **6/6**, casual7 **7/7**, and the smoke checks pass — the locked greeting verbatim ("Hey, I'm NeuralAI — how can I help you today?"), "who created you?" naming De'Andrew Preston Harris, and "what is 2 + 2?" answering "2 + 2 = 4." (reports: `evals/eval_sftv12_2lora_step12000_multiturn12.json`, `evals/eval_sftv12_2lora_step12000_date8.json`, `evals/eval_sftv12_2lora_step12000_smoke.json`) |
+| **Weights in this repo** | `sft_v12_2_lora_step12000/` (safetensors f32, config, tokenizer with chat template) — the same release also sits at the repo root, so the repository loads directly |
+| **Serving** | Production since 2026-10-22: served from NeuralAI's own infrastructure; Render `nae1-inference` runs the same bf16 copy (`sft_v12_2_lora_step12000_bf16/`) as the fallback |
+| **Chat template** | Ships in `sft_v12_2_lora_step12000/tokenizer_config.json`. Turns are `User: ...` / `NeuralAI: ...`; the generation prompt ends `NeuralAI:` with no trailing space (see the SFT section for why) |
+
+---
+
+## Previous release: SFT v11 (LoRA step-12000)
 
 | Property | Value |
 | --- | --- |
@@ -51,8 +69,8 @@ pipeline_tag: text-generation
 | **Golden-40 tool-use gate** | **35.5/40 (88.8%) — PASS** (bar 60%). Tool calls 39/40, grounded answers 32/40. Report: `evals/eval_sftv11lora_step12000_golden40.json` |
 | **Fluency guardrail** | PPL **48.08** on the 50-doc guardrail set vs **47.59** for the step-100000 base on the same set (`evals/eval_sftv11lora_step12000_ppl50_n512.json`). This is a different, smaller set than the 2000-doc set used below — the numbers are not comparable across sets |
 | **Plain chat + conversation** | v11's headline is real conversation. A new gate set (multiturn12) passes **11/12 with zero cross-topic parroting**: independent questions inside one conversation stay on their own topics, follow-ups and recall-with-distractors work, and tool use survives inside longer transcripts (report: `evals/eval_sftv11lora_step12000_multiturn12.json`). The locked greeting is restored verbatim — "hello" returns "Hey, I'm NeuralAI — how can I help you today?" word-for-word — and v10's fixes hold: "who created you?" names De'Andrew Preston Harris and "what is 2 + 2?" answers "2 + 2 = 4." (smoke report: `evals/eval_sftv11lora_step12000_smoke.json`) |
-| **Weights in this repo** | `sft_v11_lora_step12000/` (safetensors f32, config, tokenizer with chat template) — the same release also sits at the repo root, so the repository loads directly |
-| **Serving** | Production since 2026-10-22: Modal `neuralai-nae1` (CPU containers) serves the bf16 copy `sft_v11_lora_step12000_bf16/` (native PyTorch); Render `nae1-inference` runs the same copy as the fallback |
+| **Weights in this repo** | `sft_v11_lora_step12000/` (safetensors f32, config, tokenizer with chat template) |
+| **Serving** | Served in production on 2026-10-22 until the v12.2 swap the same day |
 | **Chat template** | Ships in `sft_v11_lora_step12000/tokenizer_config.json`. Turns are `User: ...` / `NeuralAI: ...`; the generation prompt ends `NeuralAI:` with no trailing space (see the SFT section for why) |
 
 ---
@@ -103,7 +121,7 @@ pipeline_tag: text-generation
 | **Training** | 80,000 steps. Resumed 60,000 → 80,000 on Kaggle T4, peak LR 6e-4, cosine to a 10% floor |
 | **Held-out perplexity** | **54.60** native f32 (2000 docs, 1,454,064 tokens, `n_ctx=512`) |
 | **GGUF f32 / Q4_K_M** | **55.19 / 55.88** on the same 2000 docs (llama.cpp scoring, token counts differ slightly, see below) |
-| **Weights in this repo** | `ckpt_v7_step80000_bf16/` (safetensors, former serving copy — production moved to the SFT line on 2026-10-21: v8, then v9, then v10, then v11 on 2026-10-22) + `tokenizer_v2/`. GGUF Q4_K_M: `nae1-step80000-Q4_K_M.gguf` |
+| **Weights in this repo** | `ckpt_v7_step80000_bf16/` (safetensors, former serving copy — production moved to the SFT line on 2026-10-21: v8, then v9, v10, v11, and v12.2) + `tokenizer_v2/`. GGUF Q4_K_M: `nae1-step80000-Q4_K_M.gguf` |
 | **License** | Apache 2.0 |
 
 > **Lineage note.** The v1 run (439-token vocab, 32k steps) is kept below for history. The v2 run rebuilt the tokenizer at 32k and continued training to step 80000. These are different lineages. Do not compare the v1 numbers (for example 54.16) with the v2 numbers. The v2 eval is on the 2000-doc held-out set (`evals/`).
@@ -170,13 +188,14 @@ The gate is golden-40: 40 held-out tasks whose facts exist only inside each task
 | v9 full-FT step-12000 | 33.5/40 | 38/40 | 29/40 | 52.48 | Fail — fluency bar (≤52) missed by 0.48; plain-chat facts not learned |
 | v9 LoRA step-12000 | 33.5/40 | 39/40 | 28/40 | 47.96 | Pass — released 2026-10-21, superseded by v10 the same day |
 | v10 LoRA step-12000 | 34.0/40 | 39/40 | 29/40 | 48.01 | Pass — released 2026-10-21, superseded by v11 on 2026-10-22 |
-| **v11 LoRA step-12000 (released)** | **35.5/40** | **39/40** | **32/40** | **48.08** | **Pass — selected (adds multiturn12 11/12)** |
+| v11 LoRA step-12000 | 35.5/40 | 39/40 | 32/40 | 48.08 | Pass — released 2026-10-22, superseded by v12.2 the same day (adds multiturn12 11/12) |
+| **v12.2 LoRA step-12000 (released)** | **35.0/40** | **38/40** | **32/40** | **48.1** | **Pass — selected (date8 7/8, memory6 6/6, casual7 7/7)** |
 
 **Root cause of the v5–v7 zero calls, proven at token level.** The training kernel's assistant-span mask started after the space in `NeuralAI: `, but the ByteLevel tokenizer fuses that space and the following `»` into a single token (id 4776) — so the first token of every tool call was excluded from the loss in every run, and the models learned the call body conditional on a first token that was never trained. v8 fixes the span, carries a pre-flight assertion (junction token supervised in 7,161 of 7,161 tool traces, confirmed in the training log), and ends the generation prompt at `NeuralAI:` so it tokenizes as a true prefix of the training format. The shipped chat template uses the same junction.
 
 **Tool protocol.** The assistant emits a call line — `» search("query")`, `» fetch("url")`, or `» read_file("path")`. The host executes the call, returns the result block into the transcript, and the assistant answers from the returned text.
 
-**Known limitations.** General prose is still rough at 100M parameters. v11 closed v10's two open items — the multi-turn topic collapse and the greeting drift — at no fluency cost (48.08 against the base's 47.59 on the guardrail set), and the public chat site runs full multi-turn conversations again. The remaining rough edges are factual rather than structural: an occasional wrong fact inside a follow-up (one gate probe answered Australia's capital as Sydney), garbled phrasing on some definition questions, and repetition loops on open-ended prompts at greedy decoding.
+**Known limitations.** General prose is still rough at 100M parameters. v12.2 fixed the date handling that blocked the previous build (date8 3/8 → 7/8, both no-bracket honesty probes clean) and restored the Australia-capital follow-up to Canberra, at no fluency cost (48.1 against the base's 47.59 on the guardrail set). What remains: relative date arithmetic across a month boundary ("tomorrow" after August 31 still repeats August 31 instead of September 1), garbled or vacuous phrasing on some definition and follow-up questions, repetition loops on open-ended prompts at greedy decoding, and live-tool search turns that do not always follow the call-then-answer shape — the search behavior is a genuine model gap, targeted in v13.
 
 ---
 
@@ -192,15 +211,17 @@ out = llm("The quick brown fox", max_tokens=32, temperature=0.0)
 print(out["choices"][0]["text"])
 ```
 
-The GGUF quickstart above loads the pretrained base (step-80000), which has no chat template. For the assistant, use `sft_v11_lora_step12000/` with the chat template in its `tokenizer_config.json`.
+The GGUF quickstart above loads the pretrained base (step-80000), which has no chat template. For the assistant, use `sft_v12_2_lora_step12000/` with the chat template in its `tokenizer_config.json`.
 
 ---
 
 ## Files
 
-- `model.safetensors` + `config.json` + `tokenizer.json` + `tokenizer_config.json` (repo root): the current v11 assistant release, identical to `sft_v11_lora_step12000/`
-- `sft_v11_lora_step12000/`: current assistant release (SFT v11, LoRA merged) — safetensors f32 + tokenizer with chat template
-- `sft_v11_lora_step12000_bf16/`: bf16 serving copy of the current assistant release (what production runs)
+- `model.safetensors` + `config.json` + `tokenizer.json` + `tokenizer_config.json` (repo root): the current v12.2 assistant release, identical to `sft_v12_2_lora_step12000/`
+- `sft_v12_2_lora_step12000/`: current assistant release (SFT v12.2, LoRA merged) — safetensors f32 + tokenizer with chat template
+- `sft_v12_2_lora_step12000_bf16/`: bf16 serving copy of the current assistant release (what production runs)
+- `sft_v11_lora_step12000/`: previous assistant release (SFT v11, LoRA merged) — safetensors f32 + tokenizer with chat template
+- `sft_v11_lora_step12000_bf16/`: bf16 serving copy of the v11 release
 - `sft_v10_lora_step12000/`: previous assistant release (SFT v10, LoRA merged) — safetensors f32 + tokenizer with chat template
 - `sft_v10_lora_step12000_bf16/`: bf16 serving copy of the v10 release
 - `sft_v9_lora_step12000/`: previous assistant release (SFT v9, LoRA merged) — safetensors f32 + tokenizer with chat template