---

LissaBrat-Base

1. Introduction

103 million parameters of pure pony language, still only 206 MB on disk. This is the pretrained base checkpoint: a plain next-token model, not a chat assistant.

• Trained from scratch. No distilled weights and no borrowed backbone: a Llama architecture with a 16,384-token vocabulary, pretrained on a fandom corpus.
• A foundation, not a product. It has no dialogue format and no instruction tuning. For conversations and roleplay use LissaBrat-Instruct.
• Small enough to embed and to fine-tune. 103M parameters retrain in minutes on a single consumer GPU.
• Trained by one person on one consumer GPU. No cluster, no budget, no institutional support.

2. Model Overview

field	value
Type	Causal Language Model
Training stage	Pretraining from scratch
Architecture	Llama
Parameters	102,760,704 (103M)
Layers	8
Hidden size	1,024
Attention heads	32 (4 key/value heads)
Head dimension	32
Intermediate size	2,730
Context length	1,024 tokens
Vocabulary	16,384 (byte-level BPE)
Tied embeddings	yes
Precision	bfloat16
License	Apache 2.0

3. The Lissa family

model	parameters	weights	best for
LissaNano-Base	32M	64 MB	continued pretraining, research
LissaNano-Instruct	32M	64 MB	phones, single-board computers, embedded use
LissaBrat-Base	103M	206 MB	continued pretraining, research
LissaBrat-Instruct	103M	206 MB	laptops, longer and more coherent text
Lissa 1.5B Instruct	1.45B	2.9 GB	a full roleplay partner
Lissa 1.5B Base	1.45B	2.9 GB	pretraining, research

4. Requirements

pip install torch transformers

No custom kernels and no exotic dependencies: this is a plain Llama checkpoint and loads with any recent transformers.

5. Quick start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CastIronMind/LissaBrat-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
model.eval()

prompt = "Twilight Sparkle opened the library door and"
ids = tok.encode(prompt, add_special_tokens=False)

out = model.generate(torch.tensor([ids]),
                     max_new_tokens=120, do_sample=True,
                     temperature=0.8, top_p=0.9, repetition_penalty=1.1,
                     pad_token_id=0, eos_token_id=1)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

6. Text completion

This is a base language model: feed it plain text and it continues it. There is no chat template and no role markers.

The special tokens 2 <system>, 3 <user> and 4 <assistant> exist in the vocabulary but are not part of the pretraining format, so do not use them here. If you need dialogue, use the Instruct checkpoint instead.

7. Data

Pretraining. Fandom prose and forums from the My Little Pony community, about 1.5 billion unique tokens, two epochs.

This checkpoint is the result of that pretraining stage only.

8. Evaluation

A frozen sample of 313 texts from Fimfarchive, about 10 million tokens, held out from training and identical for every model in the family.

metric	value
Perplexity	25.83
Bits per character	1.1357

9. Hardware and cost

item	value
Training hardware	one consumer GPU
Authors	one person
Weights on disk	206 MB (bfloat16)
License	Apache 2.0

10. Intended use

• continued pretraining on your own fandom or domain corpus
• a compact backbone for classification, tagging or embedding experiments
• research on small language models and tokenizer design
• teaching: a complete, reproducible pipeline from scratch to a working checkpoint on a single GPU

11. License

Apache License 2.0. Use it, fine-tune it, quantize it, embed it, and publish your own versions.