README.md

---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- lowonmind
- tiny-lm
- pretrained-from-scratch
- scaling-limits
---

# LowOnMind-300k

A decoder-only language model with **296,960 parameters**, pretrained from scratch on
**200M tokens** of `HuggingFaceFW/fineweb-edu` (sample-10BT).

It is an extreme variant of [DedeProGames/DynamicMind-Mini](https://huggingface.co/DedeProGames/DynamicMind-Mini)
(8.9M params), scaled down roughly 30x. The name is the point: this model is not
built to be good. It is built to find out what a third of a million parameters
actually buys you.

A 3.3x scale-up trained under identical conditions — same tokenizer, same data,
same token budget, same aspect ratio — is published as
[DedeProGames/LowOnMind-1M](https://huggingface.co/DedeProGames/LowOnMind-1M).
The two models form a controlled pair; see [Scaling comparison](#scaling-comparison-vs-lowonmind-1m)
below.

## Architecture

| | value |
|---|---|
| hidden_size | 64 |
| intermediate_size | 136 (2.12x hidden) |
| num_hidden_layers | 6 |
| heads (q / kv) | 4 / 2 (GQA, head_dim 16) |
| vocab_size | 1024 (own byte-level BPE) |
| context | 512 |
| embeddings | tied |
| total params | 296,960 |

Parameter split: 65,536 in the tied embedding, 231,360 across the six blocks,
64 in the final norm.

### Changes from DynamicMind-Mini

- **QK-Norm** — per-head RMSNorm on queries and keys before RoPE. Costs 32 params
  per block and is what makes an LR of 2e-3 survivable at this scale.
- **Precomputed RoPE** — cos/sin are built once and reused instead of being
  recomputed every forward, and the cache re-expands automatically for sequences
  longer than `max_position_embeddings`.
- **Scaled residual init** — `o_proj` and `down_proj` initialize at
  `std / sqrt(2 * num_layers)` instead of a flat 0.02.
- **1024-token vocabulary** — required to fit the parameter budget. At 8192 tokens
  the embedding alone would have been 8192 x hidden, larger than the entire model.
  The cost is compression: roughly 2.35 characters per token.

## Training

| | |
|---|---|
| data | `HuggingFaceFW/fineweb-edu`, sample-10BT |
| tokens | 200M (6,103 steps x 32,768) |
| tokens/param | 673 |
| sequence length | 512 |
| batch size | 64 |
| optimizer | AdamW, betas (0.9, 0.95), wd 0.1 |
| lr | 2e-3 peak, cosine to 2e-4, 250 warmup |
| grad clip | 1.0 |
| hardware | 1x Tesla T4 |
| wall clock | 43 min |

The token budget is deliberately small in absolute terms, but at 673 tokens per
parameter this run sits about 30x past the Chinchilla-optimal ratio. The model is
parameter-starved, not data-starved: train and validation loss stayed within
0.05 of each other for the entire run, so there is no overfitting to speak of.

## Language modelling results

| metric | value |
|---|---|
| validation loss | **3.2982** nats/token |
| validation perplexity | **27.06** |
| bits per character | **2.03** |

Perplexity is not comparable across tokenizers — with a 1024-token vocabulary,
27.06 means something very different than it would at 50k. Bits per character
(loss / ln 2 / 2.35 chars-per-token) is the portable number.

## BananaMind Base Bench 1.1

Evaluated on [BananaMind/BananaMind-Base-Bench-1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1),
a 350-item English continuation-likelihood benchmark for base models. Scoring follows
the official specification: context and each of the four continuations tokenized
separately with `add_special_tokens=False`, no BOS, selection by highest mean
conditional token log-probability.

Run validity: dataset SHA-256 matched, full schema validation passed, and no
context required truncation against the 512-token window.

| Category | Acc | Weighted acc | Elo |
|---|---:|---:|---:|
| language_completion | **46.0%** | 52.5% | 921 |
| commonsense | 34.0% | 35.6% | 848 |
| quantitative | 32.0% | 35.0% | 938 |
| logical_reasoning | 24.0% | 25.4% | 905 |
| world_knowledge | 22.0% | 21.6% | 728 |
| code_completion | 14.0% | 15.0% | 789 |
| context_tracking | 14.0% | 15.9% | 707 |

| | Overall |
|---|---:|
| **Overall Elo** | **833** |
| Chance-level Elo (this item grid) | 805 |
| Raw accuracy | 26.6% |
| 95% CI | [22.0%, 31.2%] |
| z vs. chance | +0.69 |
| Weighted accuracy | 27.9% |
| Chance | 25.0% |

## Sample output

Prompt: `"The "`, temperature 0.8, top_p 0.95.

```
The vehicle has been shown that the verb is a chief.
No:
- Even a good certain steps of the day, it is good and then a statement.
- It is a number of positive checking gain in the south of this power.
The risk of the main pottery, the straight guidelines of the brain's deployment.
- She is a little fruitful vegetables.
- It's an increasingly a brain, and this is, as a perfect men on the storage.
- Exodus of Donah, a conference, it is an aware of the message.
- The main straw of the problem.
- You're potential that the moments can be written in the majority of the paradigm.
- New York
```

This sample is more informative than the perplexity number. With a 1024-token
byte-level vocabulary, words like "increasingly", "guidelines" and "deployment"
do not exist as tokens — the model assembled each from four or five fragments in
the correct order without misspelling one. That is orthography learned as a
process, not as a lookup table. It also picked up document structure from
FineWeb-Edu: bullet markers, a `No:` header, capitalization after periods,
apostrophes in contractions. Even the invented word, "Donah", is invented with
English phonotactics.

What it does not have is meaning beyond a span of about four tokens. "an aware of
the message", "a little fruitful vegetables", "an increasingly a brain" — each
three-to-four-word fragment is grammatical and the seams between them are not.
That is the signature of ~2 bits per character: locally perfect grammar, zero
tracking.

## Scaling comparison vs LowOnMind-1M

[DedeProGames/LowOnMind-1M](https://huggingface.co/DedeProGames/LowOnMind-1M) is
the same family at 985,152 parameters (3.32x), trained with the identical
tokenizer, dataset, token budget (200M), schedule shape, and hidden/layers aspect
ratio (10.7). Parameter count is the only thing that changed, so the comparison
below isolates its effect.

| metric | 300k | 1M | delta |
|---|---:|---:|---:|
| validation loss | 3.2982 | 2.9908 | -0.3074 |
| bits per character | 2.03 | 1.836 | -0.194 |
| real-word rate (generated text) | not measured | 98.0% (ceiling 98.4%) | — |
| BananaMind overall accuracy | 26.6% | 28.9% | +2.3pp (not significant, z=+0.67) |
| BananaMind `language_completion` | 46.0% (+3.43σ vs chance) | 52.0% (+4.41σ vs chance) | +6.0pp (not significant on its own, z≈0.6) |

**The clean result is on the intrinsic LM metrics.** Validation loss and bits per
character moved by a large, unambiguous margin — those are averaged over
millions of validation tokens, so the estimate is tight. 3.3x the parameters
bought a real, measurable drop in both.

**The benchmark result is not clean, and that's informative on its own.**
350 four-way items is not enough statistical power to confirm a 2.3-point
accuracy gain; the unpaired z-test (z = 0.67) falls well short of significance,
and LowOnMind-1M's own confidence interval still overlaps chance. The one
category that clears significance against chance in *both* models is
`language_completion` — consistent with the real-word-rate result and with the
qualitative difference in the two models' sample outputs, where LowOnMind-1M
shows register-appropriate structure (bibliography-style formatting after a
date-led prompt, policy-style prose after a directive one) that this model does
not. That agreement across three independent measurements is worth more than
any single number here, including the Elo score.

**Read the two models together, not the 300k score alone**, if the question is
"does scale help at this size." It does, clearly, on the metrics with enough
samples to say so, and the benchmark is simply underpowered to confirm or deny
it on its own.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-300k")
model = AutoModelForCausalLM.from_pretrained(
    "DedeProGames/LowOnMind-300k", trust_remote_code=True
)

ids = tok("The ", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))
```

`trust_remote_code=True` is required — the architecture ships as custom modeling
code in the repository. `use_cache=False` is required: this implementation has no
KV cache, and generation recomputes the full window each step.

## Limitations

At ~300k parameters this model learns spelling, spacing, morphology and short
n-grams, but **does not produce coherent text**. It has no factual knowledge, no
reasoning ability, and no capacity to track state across a passage. It should not
be used for anything except research into the lower end of the scaling curve.