DedeProGames
/

LowOnMind-5M

@@ -27,6 +27,9 @@ share an **identical tokenizer, dataset, token budget (200M) and schedule
 shape**, so validation loss, bits-per-character and benchmark results are
 directly comparable across the series.
 ## Architecture
 | | 300k | 1M | 5M |
@@ -81,9 +84,10 @@ with automatic re-expansion, residual projections initialized at
 | hardware | Tesla T4 |
 | wall clock | 27 min |
-At 41 tokens per parameter this run is the closest of
-the three to the Chinchilla-optimal ratio of roughly 20, though still well above
-it. Train and validation loss tracked each other throughout — no overfitting.
 ## Results
@@ -97,43 +101,75 @@ Perplexity is not comparable across tokenizers, but it is comparable across
 these three models because they share one. Bits per character
 (loss / ln 2 / 2.35 chars-per-token) is the portable figure.
-Deltas: -0.4080 nats from LowOnMind-1M
-(5.0x the parameters),
--0.7154 nats from LowOnMind-300k
-(16.6x).
 ### Real-word rate
 With a 1024-token byte-level vocabulary, no long word exists as a single token —
 the model has to assemble every one of them from fragments. The fraction of
-emitted words that are real English words measures this directly.
 | | rate |
 |---|---:|
 | LowOnMind-1M | 98.0% |
-| LowOnMind-5M | **96.3%** |
 | FineWeb-Edu itself (same lexicon) | 98.4% |
-Measured over 64 unconditional samples
-(5,398 words), using the same reference lexicon as
-LowOnMind-1M (words appearing at least 5 times in a 20k-document sample of the
-training corpus), so the rows are directly comparable.
-LowOnMind-1M already reached 98.0% against a
-98.4% ceiling, so this metric had
-almost no headroom left before this model was trained. It is reported for
-continuity, not as a place where 5x the parameters could have shown much.
-Most frequent non-words: `shima`, `meikimo`, `amiravicis`, `illuminator`, `hymenola`, `phillipsburg`, `perleti`, `almanine`
-## BananaMind Base Bench 1.1
-Not yet evaluated. The two smaller models score
-[Elo 833 / 26.6% accuracy](https://huggingface.co/DedeProGames/LowOnMind-300k)
-and [Elo 843 / 28.9%](https://huggingface.co/DedeProGames/LowOnMind-1M) against
-a chance floor of Elo 805 and 25.0% accuracy. Neither clears significance
-against chance overall; only the `language_completion` category does, in both.
-Results for this model will be added once the benchmark has been run.
 ## Usage
@@ -156,8 +192,9 @@ KV cache and recomputes the full window at each generation step.
 At ~5M parameters this is still a research artifact, not a usable model. Expect
 fluent local syntax and register-appropriate structure, but **no reliable
 coherence across a paragraph**, no dependable factual knowledge, and no ability
-to track state across a passage. The 1024-token vocabulary caps absolute quality
-below what this parameter count could otherwise reach.
 The 512-token context and absent KV cache also make it unsuitable for any real
-workload.

 shape**, so validation loss, bits-per-character and benchmark results are
 directly comparable across the series.
+It is also the first model in the family whose benchmark performance is
+statistically distinguishable from chance.
 ## Architecture
 | | 300k | 1M | 5M |
 | hardware | Tesla T4 |
 | wall clock | 27 min |
+At 41 tokens per parameter this run is the closest of the three to the
+Chinchilla-optimal ratio of roughly 20 — about 2x above it, against 10x for
+LowOnMind-1M and 34x for LowOnMind-300k. Train and validation loss tracked
+each other throughout; no overfitting.
 ## Results
 these three models because they share one. Bits per character
 (loss / ln 2 / 2.35 chars-per-token) is the portable figure.
+Deltas: -0.4080 nats from LowOnMind-1M (5.0x the parameters),
+-0.7154 nats from LowOnMind-300k (16.6x).
 ### Real-word rate
 With a 1024-token byte-level vocabulary, no long word exists as a single token —
 the model has to assemble every one of them from fragments. The fraction of
+emitted words that are real English words was introduced to measure this.
 | | rate |
 |---|---:|
 | LowOnMind-1M | 98.0% |
+| LowOnMind-5M | 96.3% |
 | FineWeb-Edu itself (same lexicon) | 98.4% |
+Measured over 64 unconditional samples (5,398 words), using the same reference
+lexicon as LowOnMind-1M: words appearing at least 5 times in a 20k-document
+sample of the training corpus.
+**This number went down, and it should not be read as degraded spelling.** The
+drop is statistically real (z = 5.38, not sampling noise), but inspecting the
+non-words shows what happened: `illuminator` is an ordinary English word,
+`phillipsburg` is a US town, `shima` is a common element of Japanese place
+names. They are counted as errors only because they fall below the reference
+lexicon's frequency-5 threshold. The remainder (`hymenola`, `almanine`,
+`perleti`, `amiravicis`) skew toward proper-noun and Latinate-technical
+morphology rather than the malformed common words the metric was built to catch
+— LowOnMind-300k produced things like `landship` and `parsetic`, failures of a
+different kind.
+**The metric has a floor problem as well as a ceiling problem.** As a model
+improves it emits rarer real vocabulary — names, places, technical terms — which
+a frequency-thresholded lexicon scores as wrong. So the measured rate can fall
+while actual quality rises. Comparing against a full dictionary with proper-noun
+handling, rather than a corpus-frequency cutoff, would be the fix. The 96.3%
+figure is reported as-measured for continuity, but it should not be used to rank
+these models.
+## BananaMind Base Bench 1.1
+Evaluated on [BananaMind/BananaMind-Base-Bench-1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1),
+the same 350-item English continuation-likelihood benchmark used across the
+family, with identical scoring: context and each of the four continuations
+tokenized separately with `add_special_tokens=False`, no BOS, selection by
+highest mean conditional token log-probability.
+Run validity: dataset SHA-256 matched, full schema validation passed, no context
+required truncation against the 512-token window.
+| Category | 300k | 1M | 5M | z vs chance (5M) | Elo (5M) |
+|---|---:|---:|---:|---:|---:|
+| language_completion | 46.0% | 52.0% | **62.0%** | **+6.04** | **1008** |
+| world_knowledge | 22.0% | 22.0% | **38.0%** | +2.12 | 881 |
+| context_tracking | 14.0% | 24.0% | 32.0% | +1.14 | 851 |
+| quantitative | 32.0% | 28.0% | 28.0% | +0.49 | 872 |
+| logical_reasoning | 24.0% | 28.0% | 26.0% | +0.16 | 900 |
+| commonsense | 34.0% | 28.0% | 24.0% | -0.16 | 758 |
+| code_completion | 14.0% | 20.0% | 16.0% | -1.47 | 805 |
+| | 300k | 1M | 5M |
+|---|---:|---:|---:|
+| Overall Elo | 833 | 843 | **863** |
+| Chance-level Elo (this grid) | 805 | 805 | 805 |
+| Raw accuracy | 26.6% | 28.9% | **32.3%** |
+| 95% CI | [22.0, 31.2] | [24.2, 33.6] | **[27.4, 37.2]** |
+| z vs. chance | +0.69 | +1.68 | **+3.15** |
+| significant vs. chance | no | no | **yes** |
+Difficulty split: easy 30.8%, medium 33.3%, hard 32.8%.
 ## Usage
 At ~5M parameters this is still a research artifact, not a usable model. Expect
 fluent local syntax and register-appropriate structure, but **no reliable
 coherence across a paragraph**, no dependable factual knowledge, and no ability
+to track state across a passage. Benchmark accuracy of 32.3% is above chance and
+far below usefulness. The 1024-token vocabulary caps absolute quality below what
+this parameter count could otherwise reach.
 The 512-token context and absent KV cache also make it unsuitable for any real
+workload.