File size: 5,515 Bytes
feb6b86
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- educational
- tiny-model
- from-scratch
- byte-bpe
- decoder-only
language:
- en
metrics:
- perplexity
base_model: []
---

# fable-mini

A **3,285,504-parameter** decoder-only transformer, trained from scratch as an
educational miniature of the "Fable" assistant style (see the companion GitHub
repo [gghsff07-sudo/my_claude](https://github.com/gghsff07-sudo/my_claude) for
the full study, design, training and evaluation write-ups).

**This is a small educational model, not a real assistant, and not equivalent
to Fable 5.2 or any production LLM.** It was built to demonstrate a complete,
honest, from-scratch pipeline on a CPU budget. It did not learn to answer
questions or produce coherent sentences (see "Evaluation" below).

## Model description

- Architecture: pre-LN decoder-only transformer — 4 layers, hidden size 256,
  4 attention heads (head dim 64), FFN 768, GELU, learned absolute positions,
  bias-free fused QKV, tied input/output embeddings. Context length 512.
- Tokenizer: byte-level BPE (vocab 2,048 = 256 bytes + 1789 learned merges +
  3 specials), trained from scratch on the training split only.
- Weights: run2 final checkpoint (step 749), fp32 safetensors. The tied LM head
  weight is stored once (`tok_emb.weight`) and re-tied on load.
- Generation: `model.generate()` supports `temperature`, `top_p`, `top_k`,
  `max_new_tokens`, `repetition_penalty` with a KV cache.

Special tokens: `<|pad|>`=256, `<|eos|>`=257, `<|doc|>`=258 (document separator,
doubling as BOS). No `<|unk|>` — byte-level coverage means every byte has an id.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("<user>/fable-mini", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("<user>/fable-mini", trust_remote_code=True)

messages = [
    {"role": "system", "content": "You are a concise, warm assistant."},
    {"role": "user", "content": "Hi!"},
]
inputs = tok.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
)

out = model.generate(
    inputs.input_ids,
    max_new_tokens=64,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.2,
    pad_token_id=256,
    eos_token_id=257,
)
text = tok.decode(out[0][inputs.input_ids.shape[1]:])
print(text)
```

The model's chat format (what the chat template renders) is plain-text turns:

```
<system>You are a concise, warm assistant.</system>
User: Hi!
Assistant: ...
```

## Training data

Trained on an 18-document in-repo corpus (90/10 document-level split, seed 42):

- The "Claude Fable 5.1" reference document (~270 KB)
- 5 research/design documents written during the project
- 11 short synthetic Fable-style dialogue files (User:/Assistant: turns)

run2 upweighted the dialogue files 6× to reinforce the chat register. The total
corpus is a few hundred KB of English text — far too small for general language
ability, by design. No web scrape, no external datasets.

Training: AdamW, peak lr 3e-4, 100-step warmup, cosine decay to 10%, gradient
clip 1.0, batch 8×512 tokens, CPU only, ~750 steps in total across two runs
(600 plain + 150 resumed with dialogue upweighting).

## Evaluation (honest numbers)

| Split | Loss | Perplexity |
| --- | --- | --- |
| train | 3.6209 | 37.37 |
| val | 4.1600 | 64.07 |

Perplexity is **on this corpus only** — a corpus-fit number, not a
general-language number. Caveats, stated plainly:

- run2's val split shares genre and phrasing patterns with the upweighted
  training dialogues, so val ppl 64.07 flatters the model somewhat.
- The train/val gap (3.62 vs 4.16) shows substantial memorization.
- Sample generations are register-imitation, not coherent answers. A verbatim
  example (temperature 0.8): *"I's this is story is that't leave as the
  one-15" and you to beeny and hurborry is usually search — segoy, or I'm the
  user't render it't being matter how you need to were food and a ca- Clalated
  — your holarning are mode?"*
- The chat scaffold (User:/Assistant: turns, `<|eos|>` stops) works
  mechanically — short turns, clean stops — but the content is word soup.

The full evaluation with sample tables is [EVAL.md in the GitHub repo]
(https://github.com/gghsff07-sudo/my_claude/blob/main/EVAL.md).

## Intended use

Education and research: studying a complete from-scratch LLM pipeline
(tokenizer training, transformer implementation, training loop, evaluation,
Hugging Face packaging) at a size that trains in minutes on a laptop CPU.
Good for: unit-testing HF integration code, experimenting with decoding
strategies, teaching.

## Limitations and safety

- **Not a chatbot.** Output is incoherent word salad imitating the training
  register. It cannot answer questions, follow instructions, or hold a topic.
- **No safety behavior exists.** The model has no refusal capability, no
  calibration, and no alignment training. Do not use it for anything real.
- **Not equivalent to Fable 5.2** or any production assistant; the name refers
  to the style study, not a capability claim.
- English-only training data; 512-token context; 2K vocab.

## Acknowledging the source style

The "Fable" reference documents analyzed during this project are Anthropic-style
assistant-writing material; this model is an independent educational miniature
with no connection to, or endorsement by, Anthropic.