Lythri

Hugging Face | ModelScope | Technical Report

Introduction

Lythri-Wink-2B-pt is the pre-trained base model behind Lythri-Wink-2B. It has 2.52B total parameters and 1.48B activated parameters.

Lythri-Wink-2B-pt is structurally sheared from google/gemma-4-E2B and recovered through teacher distillation on approximately 813M tokens at an 8K training context length. It is intended for continued pre-training, supervised fine-tuning and researchβ€”not direct assistant-style conversation.

Architecture

Gemma4-E2B Lythri-Wink-2B-pt
Total parameters 4.63B 2.52B
Activated parameters 2.28B 1.48B
Decoder layers 35 31
PLE dimension 256 128

The FFN width is reduced to approximately 60% of the original model. The language-model hidden size and attention-head structure are retained.

Distillation Recovery

At the final validation point before export (step 1500):

Metric Value
Validation KL 0.3241
Student CE 2.552
Teacher CE 2.270

The released checkpoint was exported at step 1550, corresponding to approximately 812.6M training tokens. Convert two cross entropy in bits we get 3.275 and 3.682, which is a 97.24% recovery in bit level. However the error would accumulate through out the context, so real performance is much worse than that.

Base-Model Benchmarks

Benchmark Gemma 3 1B PT Lythri-Wink-2B-pt
Knowledge
MMLU26.5038.50
ARC-C38.4040.10
ARC-E73.0065.99
Reasoning
HellaSwag62.3057.45
Commonsense
PIQA73.8071.60
BoolQ63.2064.98
SocialIQA48.9042.43
WinoGrande58.2058.33

Lythri-Wink-2B-pt is evaluated without a chat template and with a BOS token prepended, Lythri-Wink-2B-pt acheives similar intelligence compare Gemma3-1B-pt, while holding a larger knowlege scale and a more permissive license.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_path = "Lythri/Lythri-Wink-2B-pt"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=True,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Your Own Model

If You dislike our instruct model Lythri/Lythri-Wink-2B, you could train a friend from our pt model! First, prepare a JSONL dataset with one text field per sample:

{"text": "Your training text goes here."}

Here's an example about the code:

from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    DataCollatorForLanguageModeling,
    Trainer,
    TrainingArguments,
)
import torch

model_id = "Lythri/Lythri-Wink-2B-pt"

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.add_bos_token = True
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.config.use_cache = False

dataset = load_dataset("json", data_files="train.jsonl", split="train")
dataset = dataset.map(
    lambda x: tokenizer(
        x["text"],
        truncation=True,
        max_length=2048,
        add_special_tokens=True,
    ),
    remove_columns=dataset.column_names,
)

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./wink-2b-output",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=16,
        learning_rate=2e-5,
        num_train_epochs=1,
        bf16=True,
        gradient_checkpointing=True,
        logging_steps=10,
        save_steps=200,
    ),
    train_dataset=dataset,
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

trainer.train()
trainer.save_model("./wink-2b-output")
tokenizer.save_pretrained("./wink-2b-output")

Important: Keep the BOS token enabled when evaluating or fine-tuning this model.

Intended Use

  • Continued pre-training
  • Domain adaptation
  • Supervised fine-tuning
  • Research on structured pruning and distillation

For direct conversation, use Lythri/Lythri-Wink-2B.

Limitations

  • This is a base model and is not instruction-tuned.
  • The model inherits limitations and biases from its base model, teacher and training data.
  • It can produce inaccurate or inappropriate content.

License

Lythri-Wink-2B-pt is built on Gemma 4 and released under the Apache License 2.0.

Citation

@techreport{li2026lythri,
  title       = {Lythri Technical Report},
  author      = {Li, Jiawen},
  year        = {2026},
  institution = {Zenodo},
  doi         = {10.5281/zenodo.23179311},
  url         = {https://doi.org/10.5281/zenodo.23179311}
}

Support

The whole training process is self-funded. If you like our model, please click a free like β€” that means a lot to me as a student!

Downloads last month
507
Safetensors
Model size
3B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Lythri/Lythri-Wink-2B-pt

Finetunes
1 model
Quantizations
2 models

Collection including Lythri/Lythri-Wink-2B-pt