Typakos-140M-it

The name "Typakos" comes from the Greek "Τυπάκος," meaning "small dude," a nod to the model's small (140M) parameter count.

Typakos-140M-it is the instruction-tuned, preference-aligned chat model built on top of alexliap/typakos-140m-base, a 140M-parameter bilingual (Greek/English) base language model. It's produced by a two-stage post-training pipeline on the base checkpoint: supervised fine-tuning (SFT) for instruction-following and chat formatting, followed by Direct Preference Optimization (DPO) for preference alignment. Both stages train all the parameters.

The full training pipeline, code, and configs live in scripts/typakos_140m/ on GitHub.

Model Details

Same architecture as the base model, with the vocabulary grown by 3 chat-template special tokens (added in the SFT stage, see below).

Model Configuration	Value
Layers	18
Hidden size	768
Intermediate size	1792
Attention heads	12 (query) / 6 (KV, grouped-query attention)
Head dim	64
Context length	2048
Vocab size	50,261 (base 50,258 + 3 chat-template special tokens)
Tied embeddings	yes
Precision	bf16
Parameters	~145M total, ~106M non-embedding

Tokenizer & Chat Template

Starts from the base model's byte-level BPE tokenizer (50,000 merges + 256 byte tokens + <|begin_of_text|>/<|end_of_text|>). The SFT stage adds three Llama-3-style special tokens (<|start_header_id|>, <|end_header_id|>, <|eot_id|>) and resizes the (tied) embedding matrix to match, with the new rows mean/covariance-initialized rather than random.

The chat template (adapted from TRL's llama3_training.jinja) renders each turn as:

<|start_header_id|>ROLE<|end_header_id|>

CONTENT<|eot_id|>

and wraps assistant turns in {% generation %} / {% endgeneration %} markers, which the SFT stage uses to mask the training loss to assistant tokens only (assistant_only_loss=True).

Training Pipeline

Stage 0: Pretraining

Trained from scratch, full-parameter, on ~9.33B tokens split roughly 50/50 English/Greek. See alexliap/typakos-140m-base for the full pretraining data mix and procedure.

Stage 1: Supervised Fine-Tuning (SFT)

Teaches the base checkpoint the chat format and instruction-following, via trl.SFTTrainer.

Data: alexliap/typakos_sft_dataset: 7 filtered subsets from 4 upstream Hub sources, kept only if well-formed (an optional single leading system turn, then alternating user/assistant turns ending on assistant) and token-bounded to fit the 2048-token context, then shuffled and split 90/10 train/validation.

source	upstream dataset	language	rows (final)
dolci_el	openeurollm/Dolci-Instruct-SFT-translated	el	448,718
eu_instruct_el	openeurollm/EU-Instruct-Synthetic	el	137,988
aya_el	CohereLabs/aya_dataset (Greek subset)	el	623
aya_en	CohereLabs/aya_dataset (English subset)	en	3,938
smol_constraints	HuggingFaceTB/smoltalk (smol-constraints)	en	34,423
smol_magpie_ultra	HuggingFaceTB/smoltalk (smol-magpie-ultra)	en	361,514
smol_rewrite	HuggingFaceTB/smoltalk (smol-rewrite)	en	53,342
TOTAL			1,040,546

~ 901M tokens total (~ 394M Greek, ~ 507M English), split into 936,491 train / 104,055 validation conversations (shuffle seed 0, 10% held out).

Training Configuration	Value
Trainer	trl.SFTTrainer
Batch size	16/device
Gradient accumulation	1
Optimizer	AdamW, lr 2e-5, betas (0.9, 0.95), eps 1e-10, weight_decay 0.01
LR schedule	Cosine, warmup 10% of steps
Epochs	1.0
Loss masking	Assistant turns only (assistant_only_loss)
Precision	bf16

Stage 2: Direct Preference Optimization (DPO)

Aligns the SFT model to preference pairs via trl.DPOTrainer, starting from Stage 1's last checkpoint.

Data: openeurollm/Dolci-Instruct-DPO-translated (el + en configs): concatenates both language configs, shuffles (seed 0), and splits off 10% as validation. Rows are prompt/chosen/rejected conversational triples.

split	rows
train	423,954
validation	47,107

Training Configuration	Value
Trainer	trl.DPOTrainer
Batch size	8/device
Gradient accumulation	1
Optimizer	AdamW, lr 5e-6, betas (0.9, 0.95), eps 1e-10, weight_decay 0.01
LR schedule	Constant with warmup, 500 warmup steps
Loss	sigmoid (standard DPO), beta 0.1
Epochs	1.0
Precision	bf16

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "alexliap/typakos-140m-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)

output = model.generate(input_ids, max_new_tokens=200, do_sample=False, temperature=0.7)
print(tokenizer.decode(output[0][input_ids.shape[-1] :], skip_special_tokens=True))

Example Outputs

Greedy-decoded (do_sample=False) chat completions from the final DPO checkpoint. Chosen to show both the model's capability and its main failure mode: without a repetition penalty, greedy decoding tends to lock onto a phrase and loop it.

English -- What is the capital of France?

The capital of France is Paris, which is the capital of France, and it is the capital of the French Empire. The capital is the capital of the French Empire, which is the capital of France. [...loops]

Greek -- Ποια είναι η πρωτεύουσα της Ελλάδας;

Η πρωτεύουσα της Ελλάδας είναι η Αθήνα, η οποία είναι η πρωτεύουσα της Ελλάδας. Η πρωτεύουσα της Ελλάδας είναι η Αθήνα, η οποία είναι η πρωτεύουσα της Ελλάδας. [...loops]

Greek -- Τι είναι η τεχνητή νοημοσύνη;

Η τεχνητή νοημοσύνη (AI) είναι μια μορφή τεχνητής νοημοσύνης που χρησιμοποιείται για την ανάλυση και την ανάλυση δεδομένων. Είναι μια μορφή που χρησιμοποιείται για την ανάλυση δεδομένων, την ανάλυση δεδομένων και την ανάλυση δεδομένων. [...loops]

Both examples get the underlying fact right (Paris, Athens) and stay grammatical throughout, but repeat themselves rather than moving on once they lock onto a clause. Sampling avoids the repetition but trades it for occasional factual drift; see Limitations.

Evaluation

Evaluated 0-shot with EleutherAI lm-evaluation-harness, except BBH (3-shot) and GSM8K (5-shot). The model's own chat template was applied for IFEval; the remaining tasks were run as raw completion/multiple-choice scoring without the chat template, matching Open LLM Leaderboard-style methodology.

Benchmark	Typakos-140M-it	SmolLM2-135M-Instruct
IFEval (avg of 4 sub-metrics)	15.6	29.9
HellaSwag (acc_norm)	26.7	40.9
ARC (avg acc_norm)	28.4	37.3
PIQA (acc_norm)	55.8	66.3
MMLU cloze (acc)	23.5	29.3
BBH 3-shot (acc_norm)	28.6	28.2
GSM8K 5-shot (exact_match)	1.1	1.4

BBH here uses leaderboard_bbh, a multiple-choice reformulation, rather than the original free-form chain-of-thought task: a model this size cannot yet produce coherent multi-step reasoning text, so this number reflects loglikelihood-based answer ranking, not demonstrated step-by-step reasoning.

Greek Suite

Evaluated 0-shot on the ilsp_greek task suite (65 sub-tasks, including all 57 mmlu_greek subject splits), without the chat template. No SmolLM2 reference exists for these tasks, so they are reported standalone:

Task	Typakos-140M-it
arc_challenge_greek (acc)	19.6
hellaswag_greek (acc)	26.6
mmlu_greek (acc, 57 subj.)	22.9
mgsm_direct_greek (exact_match)	1.6
winogrande_greek (acc)	50.9
truthfulqa_greek_mc1 (acc)	24.2
truthfulqa_greek_mc2 (acc)	44.3
medical_mcqa_greek (acc)	16.0
mcqa_greek_asep (acc)	18.7

medical_mcqa_greek and mcqa_greek_asep have no English-benchmark equivalent and are reported for completeness only. winogrande_greek uses a different task formulation than the English winogrande row above (full-candidate-sentence plausibility scoring rather than fill-in-the-blank), so the two are not directly comparable.

Limitations

• Small (140M parameter) model; expect base-rate reasoning/knowledge limitations consistent with its scale and ~9.33B-token pretraining budget (see the base model card's evaluation table).
• Greedy decoding tends to loop once it locks onto a phrase (see Example Outputs); sampling avoids this at some cost to factual reliability.

License

MIT