flame-27m-instruct

A 27.1M-parameter English instruction-tuned model: flame-27m-base (pretrained on FineWeb-Edu-dedup + Cosmopedia-v2 + ClimbMix + FineMath) fine-tuned on a ~1.18M conversation instruction mixture. ~5× smaller than SmolLM-135M-Instruct.

• Architecture: Llama-style decoder — hidden 512, 8 layers, 8 heads / 2 KV heads (GQA), intermediate 1280, RoPE (θ=1e6), context 2048, vocab 12000 (English BPE).
• Training: base → SFT (5 epochs, EMA weights) on SmolTalk + Tulu-3-Persona-IF + No-Robots + WildChat (English, non-toxic).

Usage (chat)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ifx-pse-sys-ml/flame-27m-instruct", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("ifx-pse-sys-ml/flame-27m-instruct")

messages = [{"role": "user", "content": "Tell me about the moon in one sentence."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

The decoder also accepts inputs_embeds (exactly one of input_ids / inputs_embeds), so a vision projector can splice visual tokens in — usable as a small VLM text backbone. A raw PyTorch checkpoint (pytorch_model.pth) is included alongside the safetensors weights.

Generation parameters

The model ships its own generate() (loaded via trust_remote_code). Two things to know:

• It does not read generation_config.json. A bare model.generate(ids) uses the built-in defaults — temperature 0.85, top_p 0.85, top_k 50, repetition_penalty 1.0 — so pass the settings you want explicitly.
• Greedy is do_sample=False, not temperature=0 (which divides by zero).

Measured presets, from a 28-prompt VLM-style sweep over 13 decoding configs:

Use	Settings	Measured
Answering about a given scene / image (VQA-style)	do_sample=False, repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False	Best overall: 59% grounded accuracy, no looping, shortest answers
Captions / open-ended description	do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False	Varied and loop-free

Parameter	Default	What it does / when to change
repetition_penalty	1.0	Penalises tokens already seen. 1.3 stops this model's multi-sentence looping.
no_repeat_ngram_size	0	Blocks any n-gram from repeating (3 works well). Never blocks the end-of-turn token.
penalize_prompt	True	Set False for anything grounded in the prompt. By default (Hugging Face semantics) both penalties also cover the prompt, which pushes the model away from copying the answer out of the context ("a red car", "Answer yes or no"). Limiting them to the reply raised grounded accuracy from 47% → 59% (greedy) and 27% → 39% (sampled).
temperature	0.85	Lower = more focused. Must be > 0.
top_p / top_k	0.85 / 50	Nucleus / top-k cutoffs.
max_new_tokens	8192 (capped at the 2048-token context)	Always set it — the model is at its best in short replies.

# VQA-style: answer from the context, deterministically
out = model.generate(ids, max_new_tokens=64, do_sample=False,
                     repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)

# Captions / open-ended
out = model.generate(ids, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)

Benchmarks

Accuracy (%) via lm-evaluation-harness 0.4, same harness and shots for every model, so columns are directly comparable.

Benchmark	chance	flame-27m-instruct	SmolLM-135M-Instruct
hellaswag	25	29.2	41.9
arc_easy	25	37.3	43.9
arc_challenge	25	22.6	27.4
piqa	50	59.3	67.0
winogrande	50	51.7	51.3
openbookqa	25	27.4	33.6
commonsense_qa	20	19.7	20.3
mmlu	25	24.9	24.4
average	—	34.0	38.7

These academic benchmarks measure base knowledge, which SFT cannot add. The richer instruction mixture was chosen to improve instruction-following and response quality, which it does (held-out assistant-token loss 1.34 → 1.23 vs a SmolTalk-only SFT, and cleaner format adherence) — at a small cost on the knowledge probes above. Benchmarks are the wrong lens for an instruct model's quality; they are shown only for comparability with the base and SmolLM.

Honest limitations

At 27M parameters this is near random chance on knowledge/reasoning benchmarks; the gap to SmolLM-135M is capacity, not data or tuning. Instruction-tuning adds response format, not facts. It follows simple instructions but cannot reliably satisfy hard multi-constraint prompts (IFEval ≈ 0) — that capability is bound by the 27M base, not the SFT data. A research/prototyping instruct model and a lightweight decoder, not a knowledge model. English only. Trained with the Nexus codebase.