Lune-58M-Base
🤗 Lune-58M (chat) | 📊 Evaluation | 🛠️ Quickstart
Model Summary
Lune-58M-Base is an experimental base model for Russian and English: a 57.8M-parameter Qwen3.5-style hybrid language model (Gated DeltaNet + gated attention, 3:1) with an optional 25.9M-parameter ViT distilled from the Qwen3.8-27B vision tower (83.7M parameters in total). It works as a text-only base model, and as a vision-language base model when the vision encoder is loaded. The whole model was trained on a single Apple M2 Max (language model) and a GTX 1060 (vision teacher features).
This is a base model: it continues text and describes or transcribes images. For chat, reasoning and tool use see Lune-58M.
- Architecture:
Qwen3_5ForConditionalGeneration(loads with stock transformers, no custom code) - Parameters: 57.8M language model (incl. 16.8M tied embeddings) + 25.9M vision encoder = 83.7M total
- Modes: text only (language model alone) or multimodal (with the vision encoder)
- Layers: 12:
[GDN, GDN, GDN, Attention] × 3, hidden 512, SwiGLU 1408 - Attention: 4 query / 2 KV heads × 128, QK-norm, partial MRoPE (25 %), sigmoid output gate
- Linear attention: Gated DeltaNet, 4 heads × 128, causal conv 4
- Vision encoder: ViT 12 × 384, 6 heads, patch 16, 2×2 merge → 512, 2D RoPE, images and video
- Context length: 16,384 tokens
- Vocabulary: 32,768 byte-level BPE (ru + en)
- Languages: Russian, English
- Knowledge cut-off: English: March 2026 · Russian: September 2026
- Precision: bfloat16
- License: Apache 2.0
Quickstart
pip install -U transformers torch torchvision pillow
The image and video examples use two sample files from this repository's assets folder, made for this card: a
receipt rendered in PT Sans and a 10-second slide show of five numbers.
Text continuation
The checkpoint holds both parts. Runtimes that load only the language model run Lune as a 57.8M text model, and the vision encoder is then simply not used.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ZolMik2004/Lune-58M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto") # language model only: 57.8M
# every pretraining document started with <|endoftext|>, so the prompt does too
inputs = tokenizer(tokenizer.bos_token + "Столица Франции —", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.7)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Image → text
from huggingface_hub import hf_hub_download
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "ZolMik2004/Lune-58M-Base"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto")
# the base model continues an image with its caption or the text written in it
image = Image.open(hf_hub_download(model_id, "assets/receipt.png"))
prompt = "<|endoftext|><|vision_start|><|image_pad|><|vision_end|>\n"
inputs = processor(text=prompt, images=image, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.2) # plain greedy can repeat a line
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Video → text
Video decoding in transformers needs torchcodec and FFmpeg (pip install torchcodec). The processor takes 2 frames
per second, encodes them in pairs and puts a timestamp before each pair.
# processor and model from the image example above
prompt = "<|endoftext|><|vision_start|><|video_pad|><|vision_end|>\n"
inputs = processor(text=prompt, videos=hf_hub_download(model_id, "assets/slides.mp4"), return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The base model saw no video in training. Given a video, it reads the text in some of the frames, as it does with a single image. Questions about a video are trained only in Lune-58M.
MLX (Apple Silicon)
pip install -U mlx-lm mlx-vlm
# text: the language model alone
from mlx_lm import generate, load
from mlx_lm.sample_utils import make_sampler
model, tokenizer = load("ZolMik2004/Lune-58M-Base")
prompt = tokenizer.encode(tokenizer.bos_token + "Столица Франции —", add_special_tokens=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, sampler=make_sampler(temp=0.7)))
# image: with the vision encoder
from huggingface_hub import hf_hub_download
from mlx_vlm import generate, load
model, processor = load("ZolMik2004/Lune-58M-Base")
image = hf_hub_download("ZolMik2004/Lune-58M-Base", "assets/receipt.png")
prompt = "<|endoftext|><|vision_start|><|image_pad|><|vision_end|>\n"
print(generate(model, processor, prompt, image=[image], max_tokens=64, repetition_penalty=1.2).text)
Checked with mlx-lm 0.32.2 and mlx-vlm 0.7.11. Both in float32 on the same tokens, an image included, their next-token distributions differ from transformers by a mean KL divergence of 0.00005.
Evaluation
Zero-shot, prompts and metrics of lm-evaluation-harness 0.4.13: normalized accuracy for HellaSwag, ARC and PIQA, accuracy for the rest. Params: the language model (text benchmarks run on it alone).
The comparison models run in F16 with llama.cpp on a phone (Snapdragon 8 Elite Gen 5). lm-evaluation-harness builds and tokenizes every request as it does for transformers and scores the answers, only the forward pass happens on the phone. On spot checks the log-probabilities stay within 0.001 per token of transformers in FP32, so the scores are the ones a transformers run gives. nanowhale is evaluated with transformers, llama.cpp does not run it yet.
Text — English
| Model | Params | HellaSwag | ARC-e | ARC-c | PIQA | WinoGrande | LAMBADA | MMLU |
|---|---|---|---|---|---|---|---|---|
| Lune-58M-Base | 58M | 28.7 | 41.6 | 22.6 | 57.3 | 50.9 | 21.1 | 23.3 |
| nanowhale-100m-base | 110M | 27.1 | 37.5 | 22.4 | 57.3 | 52.5 | 9.1 | 22.9 |
| SmolLM2-135M | 135M | 43.1 | 58.5 | 29.5 | 68.5 | 53.0 | 42.9 | 24.4 |
| LFM2.5-230M-Base | 230M | 39.4 | 59.5 | 35.2 | 63.5 | 54.1 | 31.4 | 40.1 |
| gemma-3-270m | 268M | 41.4 | 57.1 | 28.0 | 68.8 | 53.7 | 43.3 | 26.3 |
| SmolLM2-360M | 362M | 56.3 | 68.1 | 38.4 | 72.0 | 58.9 | 53.8 | 26.3 |
| Qwen3.5-0.8B-Base | 752M | 54.8 | 67.3 | 40.4 | 71.3 | 60.1 | 51.2 | 52.9 |
Text — Russian and multilingual
xStoryCloze ru: story endings translated by people. ARC-c ru: ARC-Challenge in the machine translation of Okapi, normalized accuracy like ARC in English.
| Model | Params | Belebele ru | Belebele en | XNLI ru | XNLI en | xStoryCloze ru | ARC-c ru |
|---|---|---|---|---|---|---|---|
| Lune-58M-Base | 58M | 25.6 | 24.7 | 42.4 | 43.3 | 51.2 | 23.8 |
| nanowhale-100m-base | 110M | 23.0 | 23.3 | 33.3 | 36.3 | 46.0 | 25.8 |
| SmolLM2-135M | 135M | 22.6 | 23.7 | 33.4 | 49.7 | 48.4 | 25.1 |
| LFM2.5-230M-Base | 230M | 32.3 | 38.0 | 42.0 | 47.4 | 51.0 | 23.5 |
| gemma-3-270m | 268M | 24.1 | 24.9 | 43.3 | 49.8 | 54.7 | 24.3 |
| SmolLM2-360M | 362M | 25.1 | 25.7 | 35.5 | 53.3 | 48.1 | 24.9 |
| Qwen3.5-0.8B-Base | 752M | 58.8 | 68.8 | 43.2 | 49.9 | 58.2 | 33.2 |
Fresh knowledge (2025–2026)
Closed-book questions about notable tech events of 2025–2026, written from the fresh news layer: 304 questions, 154 in Russian and 150 in English. Each has four answers, the model picks the most likely one, so guessing gives 25 %. Models trained before 2025 can only guess. So does Lune: the fresh layer taught it the vocabulary and topics of 2025–2026 news, not the facts themselves. For facts, use it with a search tool.
| Model | Overall | ru | en | 2025 H1 | 2025 H2 | 2026 H1 | 2026 H2 |
|---|---|---|---|---|---|---|---|
| Lune-58M-Base | 18.8 | 16.9 | 20.7 | 18.2 | 17.2 | 18.1 | 21.0 |
Vision (base)
OCR: 1,000 held-out rendered images per language (the multimodal stage never saw them), character error rate over the whole set, lower is better. The errors of one image count at most its own length, so a failed image costs 100 % of it. Captions: 1,000 COCO 2017 validation images, CIDEr-D. Both base models get the image with no instruction and continue the text, as in their pretraining (with a question in its chat template SmolVLM-256M-Base writes unrelated text).
| Model | Params | OCR CER ru ↓ | OCR CER en ↓ | COCO CIDEr-D |
|---|---|---|---|---|
| Lune-58M-Base | 84M | 64.5 % | 63.5 % | 17.5 |
| SmolVLM-256M-Base | 256M | 97.2 % | 83.8 % | 48.7 |
Text quality after adding vision
Validation loss of the text-only base before and after the vision encoder was stitched on: adding vision must not cost the text model its quality.
| Source | Text-only base | Lune-58M-Base | Change |
|---|---|---|---|
| fineweb_edu_en | 2.926 | 2.935 | +0.30 % |
| fineweb2hq_ru | 3.105 | 3.107 | +0.06 % |
| finemath_en | 2.084 | 2.090 | +0.29 % |
| wiki_en2026 | 2.908 | 2.914 | +0.22 % |
| wiki_ru2026 | 2.454 | 2.454 | +0.02 % |
| fresh_en | 1.297 | 1.281 | -1.26 % |
| fresh_ru | 2.080 | 2.092 | +0.59 % |
Training
Stage 1 — text pretraining
- Tokens: 5.0B (+ fresh layer, annealed)
- Hardware: 1 × Apple M2 Max (30-core GPU), MLX
- Duration: 6 days (144 hours)
- Optimizer: Muon (hidden matrices) + AdamW (embeddings, norms, GDN parameters)
- Schedule: WSD: warmup 1 %, plateau, linear decay over the last 20 %
- Sequence length: 2,048, the last 10 % of tokens at 16,384
- Batch: 131k tokens / step, 38,146 steps
- Precision: fp32 master weights, bf16 compute
- Final loss: 2.87 (training loss over the last 500 steps, at 16,384 context)
Data mix:
| Source | Language | Tokens |
|---|---|---|
| FineWeb-Edu | en | 2.00B |
| FineWeb2-HQ | ru | 2.25B |
| FineMath | en | 0.25B |
| Wikipedia ru 2026-10-01 | ru | 0.30B |
| Wikipedia en 2026-04-01 | en | 0.20B |
| Fresh tech news 2025–2026 (Common Crawl, filtered) | ru + en | 0.017B |
AI-generated style is down-weighted (kept with probability 0.3), not removed. Wikipedia ru is our plain-text conversion of the full 2026-10-01 dump (published as a dataset). Pretraining used a slice of it: every technology article plus a random 20 % of the rest.
Stage 2 — vision encoder distillation
- Teacher: vision tower of Qwen3.8-27B (411M), frozen
- Student: 12 × 384 ViT (25.9M)
- Targets: PCA-256 of teacher layers 9 / 18 / 27 (int8)
- Images: 889k (CC3M photos, FineVision documents / charts / screenshots / scene text, 130k rendered ru/en OCR images)
- Hardware: 1 × GTX 1060 6 GB (fp32)
- Final cosine to teacher: 0.927 / 0.881 / 0.749 at layers 9 / 18 / 27, on held-out images
Stage 3 — multimodal stitching
- 3a: merger only, image–text pairs (captions, OCR transcriptions, document Q&A)
- 3b: all weights, low LR, 50 % text replay + KL to the text-only model
- 3c: 300 more steps like 3b, with the end of a text after an image as a training target. Before it the model went on repeating the text after reading it. After it, it predicts the end in 73 % of cases (2 % before).
- Steps / tokens: 3a 2,000 steps, 3b 3,000 steps, 3c 300 steps, 32k tokens per step (174M tokens in total)
Limitations
- A model this small knows little and makes factual mistakes.
- Small print OCR and complex visual reasoning are beyond it.
- Base model: no instruction tuning, no safety tuning.
- It knows the topics of 2025–2026 news, not their facts: for anything recent, give it a search tool.
License
The model weights and code are released under the Apache 2.0 license. The training datasets are distributed under their own licenses (see the links above).
Citation
@misc{lune58mbase2026,
title = {Lune-58M-Base: a small Russian and English base model with an optional vision encoder},
author = {ZolMik2004},
year = {2026},
url = {https://huggingface.co/ZolMik2004/Lune-58M-Base}
}
- Downloads last month
- 22