Librarian-Instruct-130m

A 125.6M parameter causal language model fine-tuned on DailyDialog for conversational text generation. Built on top of librarian-base-130m using LoRA (rank 8), then merged into a single clean checkpoint.

Model Specs

Property	Value
Parameters	125,553,408
Layers	12
Heads	12
Embedding dim	768
Context length	1024 tokens
Vocabulary	16,000 (custom BPE)
Base perplexity	6.19 (WikiText-103 val)
Fine-tuning method	LoRA (rank 8, alpha 16) → merged
Fine-tuning data	DailyDialog (train split, ~87k examples)

What Changed from the Base Model

The base librarian-base-130m was trained purely for text continuation on WikiText-103 and TinyStories — it has no concept of conversation. This instruct variant was fine-tuned with LoRA on DailyDialog using a masked loss (gradients only on response tokens, not the prompt context), which nudges the model toward producing natural conversational continuations.

It can:

• Continue dialogue and conversational exchanges in English
• Serve as a starting point for further instruction tuning
• Generate coherent short responses to everyday prompts

It cannot:

• Reliably follow instructions or answer factual questions
• Maintain long multi-turn context
• Replace an RLHF-tuned assistant model

Usage

Clone Librarian-Instruct-130m and run locally. Dependencies: torch, tokenizers.

pip install torch tokenizers

Quickstart — no flags needed if files are in the same directory

# Sampling (recommended)
python generate.py --prompt "Hey, how was your day?"

# Greedy decoding
python generate.py --prompt "Hey, how was your day?" --greedy

All options

python generate.py --prompt "..." --max_new 128 --temp 0.8 --top_p 0.9 --device cpu

Flag	Default	Description
--prompt	(required)	Input prompt string
--max_new	128	Maximum new tokens to generate
--greedy	off	Use greedy decoding instead of sampling
--temp	0.8	Sampling temperature (ignored with --greedy)
--top_p	0.9	Nucleus sampling cutoff (ignored with --greedy)
--checkpoint	Librarian-Instruct-130m.pt	Path to model checkpoint
--model_cfg	model_130M.json	Path to architecture config
--tokenizer	tokenizer.json	Path to tokenizer
--device	cuda if available, else cpu	Compute device

Python API

import sys, json, torch
from tokenizers import Tokenizer

sys.path.insert(0, ".")
from src.model.gpt import GPT
from configs.model_config import ModelConfig

# load model
with open("model_130M.json") as f:
    model = GPT(ModelConfig(**json.load(f)))

ckpt = torch.load("Librarian-Instruct-130m.pt", map_location="cpu", weights_only=True)
model.load_state_dict(ckpt["model"])
model.eval()

# tokenize + generate
tokenizer = Tokenizer.from_file("tokenizer.json")
bos_id    = tokenizer.token_to_id("<bos>")
eos_id    = tokenizer.token_to_id("<eos>")

prompt    = "Hey, how was your day?"
input_ids = torch.tensor([[bos_id] + tokenizer.encode(prompt).ids])

with torch.no_grad():
    for _ in range(128):
        logits  = model(input_ids)
        next_id = logits[0, -1].argmax().item()
        if next_id == eos_id:
            break
        input_ids = torch.cat([input_ids, torch.tensor([[next_id]])], dim=1)

response_ids = input_ids[0, len(tokenizer.encode(prompt).ids) + 1:].tolist()
print(tokenizer.decode(response_ids))

Generation Tips

Conversational prompts work better than single-word inputs:

# Better — gives the model context to continue
"Hey, how was your day?"
"I was thinking about going to the park tomorrow."
"What do you think about the weather lately?"

# Too short — likely to drift
"hi"
"hello"

Lower temperature (0.7–0.8) and top_p (0.9) give more coherent responses. Greedy decoding tends to produce repetitive outputs for chat.

Training Details

Setting	Value
Method	LoRA → merged
LoRA rank	8
LoRA alpha	16
LoRA dropout	0.05
Learning rate	3e-4 (cosine decay to 3e-5)
Warmup steps	200
Total steps	5,000
Batch size	16 × grad accum 4 = 64 effective
Optimizer	AdamW (β₁=0.9, β₂=0.95)
Mixed precision	bf16
Loss	Masked cross-entropy (completion tokens only)
Dataset	daily_dialog

Files

File	Description
Librarian-Instruct-130m.pt	Merged PyTorch checkpoint (base + LoRA, ready for inference)
model_130M.json	Model architecture config
tokenizer.json	Custom 16k BPE tokenizer
tokenizer_config.json	Tokenizer config
generate.py	Self-contained inference script
src/model/gpt.py	GPT model implementation
src/model/lora.py	LoRA implementation
configs/model_config.py	ModelConfig dataclass

Model Series

Model	Status
librarian-base-130m	✅ Released
Librarian-Instruct-130m	✅ Released (this model)
librarian-base-390m	🔜 Coming soon

License

MIT