Sane-47M

A 46.7M-parameter English language model trained from scratch — architecture, tokenizer, training code and behaviour data are all original. No existing weights were fine-tuned, distilled or merged.

It is a research artifact and a small, honest chat model: it knows who it is, follows a chat format, and — most usefully — reads a supplied context and answers from it rather than from memory. At 47M parameters its own world knowledge is thin, so retrieval is the intended way to use it.

Parameters	46.74 M (tied embeddings)
Architecture	RoPE · RMSNorm · SwiGLU · pre-norm, 12 layers, d_model 512, 8 heads, FFN 1408
Context	1024 tokens
Tokenizer	own byte-level BPE, vocabulary 16 000
Training tokens	≈ 4.4 B (≈ 3.5 passes over a 1.25 B-token corpus)
Hardware	Kaggle 2×T4, then TPU v5e-8
Final step	30 000
Language	English only
Precision	fp16 weights (~105 MiB)

Quickstart

Everything needed is in this repository.

pip install torch tokenizers huggingface_hub
hf download sekund0chka/sane-47m --local-dir sane-47m
cd sane-47m
python sane_chat.py --cpu --ckpt sane_final_fp16.pt --tok tokenizer.json

Runs on CPU. Commands inside the chat: /ctx <text> supplies a retrieval context, /reset clears history, /q quits.

Prompt format

The model was trained with four control tokens and no system prompt — the persona lives in the weights.

<|user|>
{question}
<|sane|>
{answer}<|end|>

With retrieval, the context goes before the question:

<|context|>
Search results: The Ridgeloom V4 weighs 34 kg and sells for 690 euros.
<|user|>
How much does the Ridgeloom V4 weigh?
<|sane|>

Sampling defaults that work well: temperature 0.7, top_k 50, top_p 0.92, repetition_penalty 1.12. For factual questions over a context, drop the temperature to 0.3 or use greedy decoding.

What it does and does not do

Works:

• knows what it is and does not claim to be human;
• reads <|context|> and copies the right value out of it, including noisy search-result text with distractors, and says so when the answer is not in the passage;
• holds a short multi-turn conversation within the 1024-token window;
• simple string and format tasks.

Does not work:

• facts without a context are unreliable. This is a capacity wall, not a bug to be fine-tuned away — 47M parameters trained on 4.4 B tokens cannot store much;
• arithmetic beyond the trivial;
• long free-form generation drifts;
• anything outside English.

Evaluation

No standardised benchmark numbers were recorded for this model — it was evaluated with a held-out behavioural probe only. Its successor, Sane-118M, was measured and scores roughly at GPT-2-small level; Sane-47M sits below it.

If you benchmark it, please open a discussion — the numbers are welcome.

Training data

Pre-training and instruction data (see ATTRIBUTION.md in this repository):

Source	License
HuggingFaceFW/fineweb-edu (sample-10BT)	ODC-By 1.0
HuggingFaceTB/smol-smoltalk	Apache-2.0

The behaviour pool — identity, retrieval-reading, refusal-of-roleplay — is procedurally generated by the author and contains no third-party text.

ODC-By requires attribution; ATTRIBUTION.md is that attribution and must travel with the model if you redistribute it.

Bias, risks and limitations

The model inherits whatever biases exist in web text filtered for educational quality. It has no safety tuning, no refusal training for harmful requests, and no guardrails: treat its output as untrusted text. It has no memory between sessions and no access to the internet — anything time-sensitive must be supplied in <|context|>.

License

Released under the Sane Community License 1.0 (see LICENSE). In short: free to use, modify and redistribute, including commercially, until the product built on it earns more than 1 000 USD per month — beyond that, contact the licensor. Attribution notice required: "Powered by Sane-47M."

Citation

@misc{sane47m,
  title  = {Sane-47M: a small language model trained from scratch},
  author = {S.C.},
  year   = {2026},
  url    = {https://huggingface.co/sekund0chka/sane-47m}
}