Lumen-118M-Base 💡

📢 Update: Thanks for the feedback! I’ve started pretraining Lumen-v2-130M from scratch on a completely clean 6B token dataset (no ClimbMix). Stay tuned! 🚀

Lumen-118M-Base is a compact Causal Language Model pretrained from scratch on a curated high-density educational data blend, on a single consumer GPU (RTX 5060 8GB) and a 3.88B token budget.

---

📊 Leaderboard & Benchmark Performance

Measured 0-shot with length-normalized likelihood, following the Open SLM Leaderboard protocol. The table is the board's own published data, snapshot 2026-08-12, with the Intelligence Index recomputed from it using the board's formula.

#	Model	Params	Int Index	HellaSwag	ARC-Easy	ARC-Chall	PIQA	ArithMark-3
1	SmolLM2-135M (HuggingFace)	135M	27.13	43.22	58.63	29.69	68.44	39.20
2	SmolLM-135M (HuggingFace)	135M	25.74	42.70	56.31	29.01	68.28	36.80
3	GPT-X2.5-135M (Axiomic Labs)	135M	25.17	40.57	51.81	29.18	69.42	38.40
4	MobileLLM-R1-140M-base (Meta)	140M	24.64	33.84	49.92	24.74	63.22	65.70
5	GPT-X2-125M (Axiomic Labs)	125M	23.36	40.41	51.47	27.82	67.30	37.20
6	BananaMind-2-Pro-Preview	138M	23.04	39.83	51.01	27.13	66.76	38.90
7	🔥 Lumen-118M-Base (ours)	117.5M	20.32	33.74	49.20	25.77	66.65	39.50
8	GPT-X-125M (Axiomic Labs)	125M	19.94	36.57	50.76	26.62	64.96	35.60
9	Supra2-100M-Base (SupraLabs)	101M	19.41	35.98	47.81	24.83	65.40	36.90
10	Supra2-100M-Instruct (SupraLabs)	101M	18.48	35.89	44.44	24.74	64.36	38.20
12	Museko-125M (TobiasLogic)	123M	16.88	33.26	48.06	25.68	63.22	34.60

Lumen's row is a local run on the released checkpoint over the full test sets — HellaSwag n=10042, ARC-Easy n=2376, ARC-Challenge n=1172, PIQA n=1838, ArithMark-3 n=1000 — not yet a board submission. Against that snapshot's 125 scored models these scores place 7th of 126.

The board displays ARC as a single combined column (for Lumen, 37.48 = the mean of Easy and Challenge, normalized after averaging); both halves are broken out above.

On the token budget & dataset notes

• Token Efficiency: SmolLM2-135M, the current leader, was trained on 2T tokens. Lumen reaches 75% of its index on 0.19% of the tokens, in ~62 GPU-hours on one 8GB consumer card.

Most other entries do not publish a token budget, so the comparison stops there rather than being extended by guesswork.

---

📌 Architecture & Specs

Parameter	Value
Architecture	Causal Decoder-Only Transformer (GQA)
Inference Parameters	117,477,466
Layers	8
Hidden Size	768
Feed-forward Size	3072 (squared-ReLU activation)
Attention Heads	6 query / 2 KV (GQA 3:1), head dim 128
Positional Encoding	RoPE, base 100,000
Normalization	Parameter-free RMSNorm (pre-norm), plus QK-norm
Context Window	2048 tokens
Vocabulary Size	32,768 (custom byte-level BPE)
Pretraining Tokens	3.88B (7,402 steps × 524,288 tokens)
Training Hardware	1 × NVIDIA RTX 5060 8GB (single consumer GPU)
Training Time	~62 GPU-hours (30.3 s/step, ~17.3k tokens/s average)
Precision	bfloat16
Validation bits/byte	0.8953
Data Blend	ClimbMix* + Procedural Math + FineWiki + Cosmopedia

The model is a standard pre-norm decoder plus three small additions, all of which are implemented in modeling_lumen.py and documented in configuration_lumen.py:

• Local convolutional mixing — a causal depthwise convolution (kernel 3) added residually to the normed input of each attention and feed-forward sublayer.
• Gated value embeddings — the two deepest odd layers mix a per-token value embedding into the attention values through a per-head, input-dependent gate.
• Residual routing — per-layer learned scalars rescale the residual stream and re-inject the initial embedding, plus a mid-depth "backout" term subtracted before the final norm, and a gated blend of the previous token's embedding at the input.

A multi-token-prediction head was used as an auxiliary training objective only. It takes no part in a forward pass and is not included in the released weights, so the 117.5M figure above is the true inference cost.

---

💻 Quickstart

The model ships its own modeling code, so trust_remote_code=True is required. Needs transformers >= 4.57 and torch >= 2.5 (the cache and attention APIs it builds on).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kefir090/Lumen-118M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The solar system consists of"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.7,
        top_p=0.85,
        repetition_penalty=1.15,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This is a base model with no instruction tuning: it continues text, it does not follow instructions or hold a conversation.

---

🔧 Finetuning

Nothing special is needed — the model is a normal AutoModelForCausalLM and works with Trainer, TRL and PEFT.

from peft import LoraConfig, get_peft_model

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

Full finetuning, gradient checkpointing (model.gradient_checkpointing_enable()) and resize_token_embeddings are supported as well.

Things worth knowing before you train

• The tokenizer prepends <|bos|> automatically. Every pretraining document began with it, so leaving it in place keeps finetuning consistent with pretraining. <|bos|> also serves as the end-of-text and padding token.
• Chat special tokens are already in the vocabulary but were never trained: <|user_start|>, <|user_end|>, <|assistant_start|>, <|assistant_end|>, <|python_start|>, <|python_end|>, <|output_start|>, <|output_end|>. They are available for SFT without resizing the embedding table.
• Logits are soft-capped as 15 · tanh(logits / 15), which is how the model was trained. It is applied inside forward, so both the returned logits and the labels loss already include it. Reported benchmark numbers assume it stays on.
• inputs_embeds is not supported. Value embeddings and the input-side smearing gate are keyed on token identity, so the model needs input_ids. Soft prompting and prompt tuning will not work; LoRA and full finetuning are unaffected.
• Padding: use right-padding for training and left-padding for batched generation. Padded positions are masked from attention, but the convolutional mixing and the smearing gate are positional, so they will read a neighbouring pad token at the boundary.
• Precision: bfloat16 reproduces training exactly (the trunk ran in bfloat16 with fp32 master weights). float32 works and is marginally more stable for finetuning. Note that in bfloat16, cached generation is not bit-identical to a full forward pass — the usual consequence of different attention kernels, not a defect.

---

📁 Files

File	Purpose
model.safetensors	Weights, bfloat16, 235 MB
config.json	Architecture configuration
configuration_lumen.py	LumenConfig, with every field documented
modeling_lumen.py	LumenModel / LumenForCausalLM
generation_config.json	Default sampling settings
tokenizer.json	Fast tokenizer (byte-level BPE, 32,768)

The exported model was checked against the original training implementation: on identical inputs the logits match bit for bit in bfloat16, and cached incremental decoding matches a full forward pass to 2.5e-05 in float32.

---

⚠️ Limitations

• English only.
• 3.88B pretraining tokens is a small budget; factual recall is weak and confident-sounding errors are common.
• ARC-Challenge sits near chance level (25%), so multi-step reasoning is not a strength.
• No safety tuning, no alignment, no filtering beyond what the source corpora carry.
• 2048-token context; there is no length extrapolation.

---

📜 License

This model and its weights are released under the Apache 2.0 License. ```