| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| pretty_name: LissaNano-Instruct |
| tags: |
| - text-generation |
| - llama |
| - my-little-pony |
| - mlp |
| - fanfiction |
| - small-language-model |
| - from-scratch |
| - lissa |
| - roleplay |
| - rp |
| - character |
| - edge-deployment |
| model_card_authors: |
| - CastIronMind |
|
|
| --- |
| |
| <div align="center"> |
| <img src="logo.png" width="60%" alt="LissaNano-Instruct" /> |
| </div> |
|
|
| <hr> |
|
|
| <div align="center" style="line-height: 1;"> |
| <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct/blob/main/LICENSE" target="_blank" style="margin: 2px;"> |
| <img alt="License" src="https://img.shields.io/badge/license-Apache%202.0-blue" style="display: inline-block; vertical-align: middle;" /> |
| </a> |
| <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;"> |
| <img alt="Parameters" src="https://img.shields.io/badge/parameters-32M-2ea44f" style="display: inline-block; vertical-align: middle;" /> |
| </a> |
| <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;"> |
| <img alt="Context" src="https://img.shields.io/badge/context-1%2C024%20tokens-orange" style="display: inline-block; vertical-align: middle;" /> |
| </a> |
| <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;"> |
| <img alt="Vocabulary" src="https://img.shields.io/badge/vocab-16%2C384-blue" style="display: inline-block; vertical-align: middle;" /> |
| </a> |
| <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;"> |
| <img alt="Weights" src="https://img.shields.io/badge/weights-64 MB-lightgrey" style="display: inline-block; vertical-align: middle;" /> |
| </a> |
| </div> |
| |
| # LissaNano-Instruct |
|
|
| ## 1. Introduction |
|
|
| **The smallest My Little Pony language model we could find: 32 million parameters and 64 MB of weights.** This is the supervised checkpoint, tuned for character-driven conversation. |
|
|
| - **Trained from scratch.** No distilled weights and no borrowed backbone: a Llama architecture with a 16,384-token vocabulary, pretrained on a fandom corpus and then supervised on dialogue. |
| - **Built for character work.** The supervised stage uses roleplay and persona data, so the model spends its capacity on voice, character and scene. |
| - **Dialogue format is required.** The model expects the exact `<system>` / `<user>` / `<assistant>` layout described in section 6. |
| - **Trained by one person on one consumer GPU.** No cluster, no budget, no institutional support. |
|
|
| ## 2. Model Overview |
|
|
| | field | value | |
| |---|---| |
| | Type | Causal Language Model | |
| | Training stage | Pretraining from scratch, then supervised fine-tuning | |
| | Architecture | Llama | |
| | Parameters | 31,990,272 (32M) | |
| | Layers | 8 | |
| | Hidden size | 512 | |
| | Attention heads | 16 (2 key/value heads) | |
| | Head dimension | 32 | |
| | Intermediate size | 1,536 | |
| | Context length | 1,024 tokens | |
| | Vocabulary | 16,384 (byte-level BPE) | |
| | Tied embeddings | yes | |
| | Precision | bfloat16 | |
| | License | Apache 2.0 | |
|
|
| ## 3. The Lissa family |
|
|
| | model | parameters | weights | best for | |
| |---|---:|---:|---| |
| | [LissaNano-Base](https://huggingface.co/CastIronMind/LissaNano-Base) | 32M | 64 MB | continued pretraining, research | |
| | [LissaNano-Instruct](https://huggingface.co/CastIronMind/LissaNano-Instruct) | 32M | 64 MB | phones, single-board computers, embedded use | |
| | [LissaBrat-Base](https://huggingface.co/CastIronMind/LissaBrat-Base) | 103M | 206 MB | continued pretraining, research | |
| | [LissaBrat-Instruct](https://huggingface.co/CastIronMind/LissaBrat-Instruct) | 103M | 206 MB | laptops, longer and more coherent text | |
| | [Lissa 1.5B Instruct](https://huggingface.co/CastIronMind/Lissa-1.5B-Instruct) | 1.45B | 2.9 GB | a full roleplay partner | |
| | [Lissa 1.5B Base](https://huggingface.co/CastIronMind/Lissa-1.5B-Base) | 1.45B | 2.9 GB | pretraining, research | |
|
|
| **GGUF quantizations:** [LissaNano-Instruct-GGUF](https://huggingface.co/CastIronMind/LissaNano-Instruct-GGUF) — 15 files from Q8_0 down to IQ2, each with measured perplexity against this bf16 checkpoint. |
| |
| **Demo:** [Lissa Playground](https://huggingface.co/spaces/CastIronMind/Lissa-Playground) — compare quantizations side by side in the browser, no server. |
| |
| ## 4. Requirements |
| |
| ``` |
| pip install torch transformers |
| ``` |
| |
| No custom kernels and no exotic dependencies: this is a plain Llama checkpoint and loads with any recent `transformers`. |
| |
| ## 5. Quick start |
| |
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model_id = "CastIronMind/LissaNano-Instruct" |
| tok = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16) |
| model.eval() |
|
|
| SYS, USR, AST, EOS = 2, 3, 4, 1 |
|
|
| system = "You are Twilight Sparkle, a studious purple unicorn who loves books." |
| user = "Twilight, I found a strange book in the library." |
|
|
| ids = [SYS] + tok.encode(system, add_special_tokens=False) \ |
| + [USR] + tok.encode(user, add_special_tokens=False) + [AST] |
| x = torch.tensor([ids], dtype=torch.long) |
|
|
| out = model.generate(x, max_new_tokens=120, do_sample=True, |
| temperature=0.8, top_p=0.9, repetition_penalty=1.1, |
| pad_token_id=0, eos_token_id=EOS) |
| print(tok.decode(out[0][len(ids):], skip_special_tokens=True)) |
| ``` |
| |
| ## 6. Prompt format |
| |
| The model was supervised on exactly this layout: |
| |
| ``` |
| <system>{persona}<user>{message}<assistant>{reply}<eos> |
| ``` |
| |
| Tags are concatenated without spaces. A non-empty system message is required: every supervised example carried one. |
| |
| **Special tokens.** `0 <pad>`, `1 <eos>`, `2 <system>`, `3 <user>`, `4 <assistant>`. |
| |
| **Sampling.** Temperature 0.8, top_p 0.9, repetition penalty 1.1. |
|
|
| ## 7. Data |
|
|
| **Pretraining.** Fandom prose and forums from the My Little Pony community, about 1 billion unique tokens, two epochs. |
|
|
| **Supervised fine-tuning.** Roleplay and persona dialogues with the `<system>` / `<user>` / `<assistant>` layout, two epochs. |
|
|
| ## 8. Evaluation |
|
|
| This checkpoint is tuned for dialogue, so held-out perplexity is reported on the base model card, where it is measured on the same frozen set for every model in the family: see [LissaNano-Base](https://huggingface.co/CastIronMind/LissaNano-Base). |
|
|
| What the supervised stage targets: character voice, in-persona dialogue, and stable formatting across turns. |
|
|
| ## 9. Hardware and cost |
|
|
| | item | value | |
| |---|---| |
| | Training hardware | one consumer GPU | |
| | Authors | one person | |
| | Weights on disk | 64 MB (bfloat16) | |
| | License | Apache 2.0 | |
|
|
| ## 10. Intended use |
|
|
| - roleplay bots and character agents in chat applications |
| - embedded and offline deployments: games, visual novels, toys |
| - experiments with character consistency in small models |
| - further fine-tuning on a specific character; the model retrains in minutes |
|
|
| ## 11. License |
|
|
| Apache License 2.0. Use it, fine-tune it, quantize it, embed it, and publish your own versions. |
|
|
|
|
|
|
|
|
| - ПАЦАНЫ, это малышок, она генерирует текст, и это иногда даже похоже на смысл! |
|
|
