README.md

---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- lowonmind
- tiny-lm
- pretrained-from-scratch
- scaling-limits
---

# LowOnMind-1M

A decoder-only language model with **985,152 parameters**, pretrained from
scratch on **200M tokens** of
`HuggingFaceFW/fineweb-edu` (sample-10BT).

This is the 3.3x scale-up of
[DedeProGames/LowOnMind-300k](https://huggingface.co/DedeProGames/LowOnMind-300k),
run as a **controlled experiment**: identical tokenizer, identical dataset,
identical token budget, identical schedule, and the same hidden/layers aspect
ratio. Parameter count is the only variable, so the two models are directly
comparable on validation loss and on downstream benchmarks.

## Architecture

| | LowOnMind-300k | LowOnMind-1M |
|---|---|---|
| parameters | 296,960 | **985,152** |
| hidden_size | 64 | 96 |
| intermediate_size | 136 (2.12x) | 256 (2.67x) |
| num_hidden_layers | 6 | 9 |
| heads (q / kv) | 4 / 2 | 6 / 2 |
| head_dim | 16 | 16 |
| aspect ratio | 10.7 | 10.7 |
| vocab_size | 1024 | 1024 (same tokenizer) |
| context | 512 | 512 |
| tokens seen | 200M | 200M |
| tokens/param | 673 | 203 |

Architecture details, inherited from
[DynamicMind-Mini](https://huggingface.co/DedeProGames/DynamicMind-Mini) with
modifications: GQA, SwiGLU, RMSNorm, tied embeddings, **QK-Norm** per head,
**precomputed RoPE** with automatic re-expansion, and residual projections
initialized at `std / sqrt(2 * num_layers)`.

## Training

| | |
|---|---|
| data | `HuggingFaceFW/fineweb-edu`, sample-10BT |
| tokens | 200M (6,103 steps x 32,768) |
| sequence length | 512 |
| batch size | 64 |
| optimizer | AdamW, betas (0.9, 0.95), wd 0.1 |
| lr | 1.5e-03 peak, cosine to 1.5e-04, 250 warmup |
| grad clip | 1.0 |
| precision | float16 + GradScaler |
| hardware | Tesla T4 |
| wall clock | 11 min |

At 203 tokens per parameter this run still sits far past
the Chinchilla-optimal ratio. Train and validation loss tracked each other for
the entire run — this model is parameter-limited, not data-limited.

## Results

| metric | LowOnMind-300k | LowOnMind-1M | delta |
|---|---:|---:|---:|
| validation loss | 3.2982 | **2.9908** | -0.3074 |
| validation perplexity | 27.06 | **19.90** | -7.16 |
| bits per character | 2.030 | **1.836** | -0.194 |

Perplexity is not comparable across tokenizers, but it *is* comparable between
these two models because they share one. Bits per character (loss / ln 2 /
2.35 chars-per-token) is the portable figure.

### Real-word rate

With a 1024-token byte-level vocabulary, no long word exists as a single token —
the model has to assemble every one of them from fragments. The fraction of
emitted words that are real English words measures this directly, and neither
validation loss nor a multiple-choice benchmark captures it.

| | rate |
|---|---:|
| LowOnMind-1M | **98.0%** |
| FineWeb-Edu itself (same lexicon) | 98.4% |

Measured over 64 unconditional samples
(5,647 words). The reference lexicon is every
lowercase word appearing at least 5 times in a 20k-document sample of the
training corpus, so the corpus row is the practical ceiling rather than 100%.
At 98.0% against a 98.4% ceiling, the lexicon is essentially saturated — the
300k → 1M scale-up bought almost all of its capacity in spelling and word
formation, not in anything downstream of it.

Most frequent non-words: `sculieness`, `lockholm`, `scul`, `purpled`, `paradigmar`, `prefection`, `frushing`, `fullly`

## BananaMind Base Bench 1.1

Evaluated on [BananaMind/BananaMind-Base-Bench-1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1),
the same 350-item English continuation-likelihood benchmark used for
LowOnMind-300k, with identical scoring: context and each of the four
continuations tokenized separately with `add_special_tokens=False`, no BOS,
selection by highest mean conditional token log-probability.

Run validity: dataset SHA-256 matched, full schema validation passed, and no
context required truncation against the 512-token window.

| Category | LowOnMind-1M | LowOnMind-300k | delta | Elo (1M) |
|---|---:|---:|---:|---:|
| language_completion | **52.0%** | 46.0% | +6.0pp | 937 |
| logical_reasoning | 28.0% | 24.0% | +4.0pp | 925 |
| context_tracking | 24.0% | 14.0% | +10.0pp | 770 |
| code_completion | 20.0% | 14.0% | +6.0pp | 851 |
| world_knowledge | 22.0% | 22.0% | +0.0pp | 752 |
| quantitative | 28.0% | 32.0% | -4.0pp | 913 |
| commonsense | 28.0% | 34.0% | -6.0pp | 786 |

| | LowOnMind-1M | LowOnMind-300k |
|---|---:|---:|
| Overall Elo | 843 | 833 |
| Chance-level Elo (this item grid) | 805 | 805 |
| Raw accuracy | 28.9% | 26.6% |
| 95% CI | [24.1%, 33.6%] | [22.0%, 31.2%] |
| z vs. chance | +1.67 | +0.69 |

## Sample output

Prompts sampled at temperature 0.8, top_p 0.95. Unlike the 300k model, output
now shifts register with the prompt — a date-led prompt produces
bibliography-style formatting, a directive prompt produces policy-style prose —
and invented non-words (see the real-word-rate section) are markedly rarer than
at 300k parameters. Meaning and referential continuity are still absent.

```
prompt: "The "
-> The veil-group, where is used to solve solar passwords and vegetables. It
is often very important to reduce their treatments, including checkouts,
labs, maintenance, mental groups, chemicals and maintenance costs. These
literally confirmed elements are the most commonly used in the elementary
front, solar energy, a

prompt: "Photosynthesis is "
-> Photosynthesis is essential to prevent disease infection. These types of
disease include:
• Transformation of Brook Scientific World Organization
• Effectiveness of a diagnosis is commonly diagnosed with diagnosis. The
Cultural Disorder is an important factor in diagnosis. Analysis of this
purpose can be considered.

prompt: "In 1969, "
-> In 1969, 8, 1999, Michael Kerzin, Anderson, 1997, Scotland, 1993, 1,359,
p. 547-414
Our war was chosen in the West of Europe, Papua Party, Ottoman Economy,
Pluto Mountains, and Evans, 1854, 1946, Maine, 1888-81, p

prompt: "Students should "
-> Students should identify their learners and adults.
The United States has free quotations in the world that can help them with
this curriculum and finding problem concerning these funding.
One major solutions for malnutrition in South East and Africa is a way to
limit half its own loop. This is the best way to ensure that the drills of
the state and
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-1M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/LowOnMind-1M", trust_remote_code=True)

ids = tok("The ", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))
```

`trust_remote_code=True` is required — the architecture ships as custom modeling
code in the repository. `use_cache=False` is required: this implementation has no
KV cache and recomputes the full window at each generation step.

## Limitations

At ~1M parameters this model learns spelling, morphology, and short-range syntax.
It **does not produce coherent text**, has no reliable factual knowledge, and
cannot track state across a passage. It exists to measure the lower end of the
scaling curve, not to be used.

The 512-token context, 1024-token vocabulary, and absent KV cache make it
unsuitable for any real workload.