dev-sandakelum commited on
Commit
feb6b86
·
verified ·
1 Parent(s): 840399b

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - educational
7
+ - tiny-model
8
+ - from-scratch
9
+ - byte-bpe
10
+ - decoder-only
11
+ language:
12
+ - en
13
+ metrics:
14
+ - perplexity
15
+ base_model: []
16
+ ---
17
+
18
+ # fable-mini
19
+
20
+ A **3,285,504-parameter** decoder-only transformer, trained from scratch as an
21
+ educational miniature of the "Fable" assistant style (see the companion GitHub
22
+ repo [gghsff07-sudo/my_claude](https://github.com/gghsff07-sudo/my_claude) for
23
+ the full study, design, training and evaluation write-ups).
24
+
25
+ **This is a small educational model, not a real assistant, and not equivalent
26
+ to Fable 5.2 or any production LLM.** It was built to demonstrate a complete,
27
+ honest, from-scratch pipeline on a CPU budget. It did not learn to answer
28
+ questions or produce coherent sentences (see "Evaluation" below).
29
+
30
+ ## Model description
31
+
32
+ - Architecture: pre-LN decoder-only transformer — 4 layers, hidden size 256,
33
+ 4 attention heads (head dim 64), FFN 768, GELU, learned absolute positions,
34
+ bias-free fused QKV, tied input/output embeddings. Context length 512.
35
+ - Tokenizer: byte-level BPE (vocab 2,048 = 256 bytes + 1789 learned merges +
36
+ 3 specials), trained from scratch on the training split only.
37
+ - Weights: run2 final checkpoint (step 749), fp32 safetensors. The tied LM head
38
+ weight is stored once (`tok_emb.weight`) and re-tied on load.
39
+ - Generation: `model.generate()` supports `temperature`, `top_p`, `top_k`,
40
+ `max_new_tokens`, `repetition_penalty` with a KV cache.
41
+
42
+ Special tokens: `<|pad|>`=256, `<|eos|>`=257, `<|doc|>`=258 (document separator,
43
+ doubling as BOS). No `<|unk|>` — byte-level coverage means every byte has an id.
44
+
45
+ ## Usage
46
+
47
+ ```python
48
+ from transformers import AutoModelForCausalLM, AutoTokenizer
49
+
50
+ tok = AutoTokenizer.from_pretrained("<user>/fable-mini", trust_remote_code=True)
51
+ model = AutoModelForCausalLM.from_pretrained("<user>/fable-mini", trust_remote_code=True)
52
+
53
+ messages = [
54
+ {"role": "system", "content": "You are a concise, warm assistant."},
55
+ {"role": "user", "content": "Hi!"},
56
+ ]
57
+ inputs = tok.apply_chat_template(
58
+ messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
59
+ )
60
+
61
+ out = model.generate(
62
+ inputs.input_ids,
63
+ max_new_tokens=64,
64
+ do_sample=True,
65
+ temperature=0.8,
66
+ top_p=0.9,
67
+ top_k=50,
68
+ repetition_penalty=1.2,
69
+ pad_token_id=256,
70
+ eos_token_id=257,
71
+ )
72
+ text = tok.decode(out[0][inputs.input_ids.shape[1]:])
73
+ print(text)
74
+ ```
75
+
76
+ The model's chat format (what the chat template renders) is plain-text turns:
77
+
78
+ ```
79
+ <system>You are a concise, warm assistant.</system>
80
+ User: Hi!
81
+ Assistant: ...
82
+ ```
83
+
84
+ ## Training data
85
+
86
+ Trained on an 18-document in-repo corpus (90/10 document-level split, seed 42):
87
+
88
+ - The "Claude Fable 5.1" reference document (~270 KB)
89
+ - 5 research/design documents written during the project
90
+ - 11 short synthetic Fable-style dialogue files (User:/Assistant: turns)
91
+
92
+ run2 upweighted the dialogue files 6× to reinforce the chat register. The total
93
+ corpus is a few hundred KB of English text — far too small for general language
94
+ ability, by design. No web scrape, no external datasets.
95
+
96
+ Training: AdamW, peak lr 3e-4, 100-step warmup, cosine decay to 10%, gradient
97
+ clip 1.0, batch 8×512 tokens, CPU only, ~750 steps in total across two runs
98
+ (600 plain + 150 resumed with dialogue upweighting).
99
+
100
+ ## Evaluation (honest numbers)
101
+
102
+ | Split | Loss | Perplexity |
103
+ | --- | --- | --- |
104
+ | train | 3.6209 | 37.37 |
105
+ | val | 4.1600 | 64.07 |
106
+
107
+ Perplexity is **on this corpus only** — a corpus-fit number, not a
108
+ general-language number. Caveats, stated plainly:
109
+
110
+ - run2's val split shares genre and phrasing patterns with the upweighted
111
+ training dialogues, so val ppl 64.07 flatters the model somewhat.
112
+ - The train/val gap (3.62 vs 4.16) shows substantial memorization.
113
+ - Sample generations are register-imitation, not coherent answers. A verbatim
114
+ example (temperature 0.8): *"I's this is story is that't leave as the
115
+ one-15" and you to beeny and hurborry is usually search — segoy, or I'm the
116
+ user't render it't being matter how you need to were food and a ca- Clalated
117
+ — your holarning are mode?"*
118
+ - The chat scaffold (User:/Assistant: turns, `<|eos|>` stops) works
119
+ mechanically — short turns, clean stops — but the content is word soup.
120
+
121
+ The full evaluation with sample tables is [EVAL.md in the GitHub repo]
122
+ (https://github.com/gghsff07-sudo/my_claude/blob/main/EVAL.md).
123
+
124
+ ## Intended use
125
+
126
+ Education and research: studying a complete from-scratch LLM pipeline
127
+ (tokenizer training, transformer implementation, training loop, evaluation,
128
+ Hugging Face packaging) at a size that trains in minutes on a laptop CPU.
129
+ Good for: unit-testing HF integration code, experimenting with decoding
130
+ strategies, teaching.
131
+
132
+ ## Limitations and safety
133
+
134
+ - **Not a chatbot.** Output is incoherent word salad imitating the training
135
+ register. It cannot answer questions, follow instructions, or hold a topic.
136
+ - **No safety behavior exists.** The model has no refusal capability, no
137
+ calibration, and no alignment training. Do not use it for anything real.
138
+ - **Not equivalent to Fable 5.2** or any production assistant; the name refers
139
+ to the style study, not a capability claim.
140
+ - English-only training data; 512-token context; 2K vocab.
141
+
142
+ ## Acknowledging the source style
143
+
144
+ The "Fable" reference documents analyzed during this project are Anthropic-style
145
+ assistant-writing material; this model is an independent educational miniature
146
+ with no connection to, or endorsement by, Anthropic.
chat_template.jinja ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if messages[0]['role'] == 'system' %}
2
+ <system>{{ messages[0]['content'] }}</system>
3
+ {% set start = 1 %}
4
+ {%- else %}
5
+ {% set start = 0 %}
6
+ {%- endif %}
7
+ {%- for message in messages[start:] %}
8
+ {{ message['role'] | capitalize }}: {{ message['content'] }}
9
+ {% endfor %}
10
+ {%- if add_generation_prompt %}
11
+ Assistant:
12
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_map": {
3
+ "AutoConfig": "configuration_fable.FableConfig",
4
+ "AutoModelForCausalLM": "modeling_fable.FableLMHeadModel"
5
+ },
6
+ "bos_token_id": 258,
7
+ "dropout": 0.0,
8
+ "eos_token_id": 257,
9
+ "hidden_size": 256,
10
+ "intermediate_size": 768,
11
+ "max_position_embeddings": 512,
12
+ "model_type": "fable",
13
+ "num_attention_heads": 4,
14
+ "num_hidden_layers": 4,
15
+ "pad_token_id": 256,
16
+ "tie_word_embeddings": true,
17
+ "transformers_version": "5.23.0",
18
+ "vocab_size": 2048
19
+ }
configuration_fable.py ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """FableConfig — Hugging Face configuration for the `fable` miniature.
2
+
3
+ Mirrors `fable_lm.model.ModelConfig` (vocab_size=2048, context_len=512, n_layers=4,
4
+ d_model=256, n_heads=4, d_ff=768) using standard HF configuration names so the
5
+ model is loadable with `AutoConfig` / `trust_remote_code`.
6
+
7
+ This file is intentionally self-contained (no `fable_lm` import): it is copied
8
+ verbatim into the Hub repo, where only the files next to `config.json` exist.
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ from transformers import PretrainedConfig
14
+
15
+
16
+ class FableConfig(PretrainedConfig):
17
+ model_type = "fable"
18
+
19
+ def __init__(
20
+ self,
21
+ vocab_size: int = 2048,
22
+ max_position_embeddings: int = 512,
23
+ num_hidden_layers: int = 4,
24
+ hidden_size: int = 256,
25
+ num_attention_heads: int = 4,
26
+ intermediate_size: int = 768,
27
+ dropout: float = 0.0,
28
+ pad_token_id: int = 256, # <|pad|>
29
+ bos_token_id: int = 258, # <|doc|> doubles as BOS (document separator)
30
+ eos_token_id: int = 257, # <|eos|>
31
+ **kwargs,
32
+ ):
33
+ # architecture hyperparameters (names follow HF conventions)
34
+ self.vocab_size = vocab_size
35
+ self.max_position_embeddings = max_position_embeddings
36
+ self.num_hidden_layers = num_hidden_layers
37
+ self.hidden_size = hidden_size
38
+ self.num_attention_heads = num_attention_heads
39
+ self.intermediate_size = intermediate_size
40
+ self.dropout = dropout
41
+ # token embeddings are tied into the LM head (FableLM: head.weight = tok_emb.weight)
42
+ self.tie_word_embeddings = True
43
+ super().__init__(
44
+ pad_token_id=pad_token_id,
45
+ bos_token_id=bos_token_id,
46
+ eos_token_id=eos_token_id,
47
+ **kwargs,
48
+ )
49
+
50
+ # ---- read-only aliases for the original repo-style names ----------------
51
+ # Properties (not attributes) so they never appear in to_dict()/config.json;
52
+ # serialization always uses the canonical HF names above.
53
+
54
+ @property
55
+ def context_len(self) -> int:
56
+ return self.max_position_embeddings
57
+
58
+ @property
59
+ def n_layers(self) -> int:
60
+ return self.num_hidden_layers
61
+
62
+ @property
63
+ def d_model(self) -> int:
64
+ return self.hidden_size
65
+
66
+ @property
67
+ def n_heads(self) -> int:
68
+ return self.num_attention_heads
69
+
70
+ @property
71
+ def d_ff(self) -> int:
72
+ return self.intermediate_size
73
+
74
+
75
+ try:
76
+ # Local (non-remote-code) use: `import hf_fable` makes AutoConfig aware of
77
+ # "fable" without trust_remote_code. exist_ok: the module may also be
78
+ # imported under a second name (hf_fable.configuration_fable vs a plain
79
+ # configuration_fable copied next to config.json).
80
+ from transformers import AutoConfig
81
+
82
+ AutoConfig.register("fable", FableConfig, exist_ok=True)
83
+ except Exception: # pragma: no cover - registration must never break import
84
+ pass
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 258,
3
+ "do_sample": true,
4
+ "eos_token_id": 257,
5
+ "max_new_tokens": 128,
6
+ "pad_token_id": 256,
7
+ "repetition_penalty": 1.2,
8
+ "temperature": 0.8,
9
+ "top_k": 50,
10
+ "top_p": 0.9,
11
+ "transformers_version": "5.23.0"
12
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:226685b476d7381b907c4605368f46d7e39e4eec2df6e7d0fa1908c7ef2861e6
3
+ size 13145912
modeling_fable.py ADDED
@@ -0,0 +1,258 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """FableLMHeadModel — Hugging Face wrapper for the Fable miniature (model_type "fable").
2
+
3
+ Self-contained copy of the `fable_lm.FableLM` architecture (HF remote-code files
4
+ must not import the repo package):
5
+ pre-LN decoder-only transformer, learned absolute positions, GELU FF,
6
+ fused bias-free QKV, tied input/output embeddings.
7
+ Identical module tree and tensor names, so a checkpoint state dict loads
8
+ unchanged:
9
+ tok_emb.weight, pos_emb.weight,
10
+ blocks.{i}.ln1.{weight,bias}, blocks.{i}.attn.qkv.weight, blocks.{i}.attn.proj.weight,
11
+ blocks.{i}.ln2.{weight,bias}, blocks.{i}.ff.fc.{weight,bias}, blocks.{i}.ff.proj.{weight,bias},
12
+ ln_f.{weight,bias}, head.weight (tied to tok_emb.weight)
13
+
14
+ forward(input_ids, attention_mask, labels) -> CausalLMOutputWithPast, with an
15
+ optional DynamicCache KV cache for generate(). Supports temperature, top_p,
16
+ top_k, max_new_tokens and repetition_penalty through GenerationMixin.
17
+ """
18
+
19
+ from __future__ import annotations
20
+
21
+ import math
22
+ from typing import Optional, Tuple
23
+
24
+ import torch
25
+ import torch.nn as nn
26
+ import torch.nn.functional as F
27
+ from transformers import GenerationMixin, PreTrainedModel
28
+ from transformers.cache_utils import Cache, DynamicCache
29
+ from transformers.modeling_outputs import CausalLMOutputWithPast
30
+
31
+ # transformers re-exports its own nn.init wrappers (they take a `generator`
32
+ # kwarg and advance the global RNG differently when it is explicitly None).
33
+ # Importing torch's originals by module path guarantees the same init draws
34
+ # as fable_lm.model.FableLM, so same-seed constructors are bit-identical.
35
+ _nn_init_normal = torch.nn.init.normal_
36
+ _nn_init_zeros = torch.nn.init.zeros_
37
+
38
+ from .configuration_fable import FableConfig
39
+
40
+
41
+ class CausalSelfAttention(nn.Module):
42
+ """Fused-QKV multi-head self-attention, bias-free, with KV-cache support."""
43
+
44
+ def __init__(self, cfg: FableConfig):
45
+ super().__init__()
46
+ assert cfg.hidden_size % cfg.num_attention_heads == 0
47
+ self.n_heads = cfg.num_attention_heads
48
+ self.d_head = cfg.hidden_size // cfg.num_attention_heads
49
+ self.qkv = nn.Linear(cfg.hidden_size, 3 * cfg.hidden_size, bias=False)
50
+ self.proj = nn.Linear(cfg.hidden_size, cfg.hidden_size, bias=False)
51
+
52
+ def forward(self, x, attn_mask_4d, past_key_values=None, layer_idx=0, use_cache=True):
53
+ B, T, C = x.shape
54
+ qkv = self.qkv(x) # (B, T, 3C)
55
+ q, k, v = qkv.split(C, dim=2)
56
+ q = q.view(B, T, self.n_heads, self.d_head).transpose(1, 2) # (B, nh, T, dh)
57
+ k = k.view(B, T, self.n_heads, self.d_head).transpose(1, 2)
58
+ v = v.view(B, T, self.n_heads, self.d_head).transpose(1, 2)
59
+
60
+ if past_key_values is not None:
61
+ k, v = past_key_values.update(k, v, layer_idx)
62
+
63
+ att = (q @ k.transpose(-2, -1)) / math.sqrt(self.d_head) # (B, nh, T, kv)
64
+ if attn_mask_4d is not None:
65
+ att = att.masked_fill(~attn_mask_4d, float("-inf"))
66
+ att = F.softmax(att, dim=-1)
67
+ y = att @ v # (B, nh, T, dh)
68
+ y = y.transpose(1, 2).contiguous().view(B, T, C)
69
+ return self.proj(y)
70
+
71
+
72
+ class FeedForward(nn.Module):
73
+ def __init__(self, cfg: FableConfig):
74
+ super().__init__()
75
+ self.fc = nn.Linear(cfg.hidden_size, cfg.intermediate_size)
76
+ self.proj = nn.Linear(cfg.intermediate_size, cfg.hidden_size)
77
+
78
+ def forward(self, x):
79
+ return self.proj(F.gelu(self.fc(x)))
80
+
81
+
82
+ class Block(nn.Module):
83
+ """Pre-LN block: x + attn(ln1(x)), then x + ff(ln2(x)) — as in fable_lm."""
84
+
85
+ def __init__(self, cfg: FableConfig):
86
+ super().__init__()
87
+ self.ln1 = nn.LayerNorm(cfg.hidden_size)
88
+ self.attn = CausalSelfAttention(cfg)
89
+ self.ln2 = nn.LayerNorm(cfg.hidden_size)
90
+ self.ff = FeedForward(cfg)
91
+
92
+ def forward(self, x, attn_mask_4d, past_key_values=None, layer_idx=0, use_cache=True):
93
+ x = x + self.attn(self.ln1(x), attn_mask_4d, past_key_values, layer_idx, use_cache)
94
+ x = x + self.ff(self.ln2(x))
95
+ return x
96
+
97
+
98
+ class FablePreTrainedModel(PreTrainedModel):
99
+ config_class = FableConfig
100
+ base_model_prefix = "fable"
101
+ supports_gradient_checkpointing = False
102
+ _tied_weights_keys = {"head.weight": "tok_emb.weight"} # 5.x: {target: source}
103
+
104
+ def _init_weights(self, module):
105
+ """Match fable_lm.FableLM._init: normal(0, 0.02), zero biases.
106
+
107
+ Uses torch's original nn.init functions (not transformers' re-exports)
108
+ so the random stream matches fable_lm exactly for a given seed.
109
+ """
110
+ if isinstance(module, nn.Linear):
111
+ _nn_init_normal(module.weight, mean=0.0, std=0.02)
112
+ if module.bias is not None:
113
+ _nn_init_zeros(module.bias)
114
+ elif isinstance(module, nn.Embedding):
115
+ _nn_init_normal(module.weight, mean=0.0, std=0.02)
116
+
117
+
118
+ class FableLMHeadModel(FablePreTrainedModel, GenerationMixin):
119
+ def __init__(self, config: FableConfig):
120
+ super().__init__(config)
121
+ self.tok_emb = nn.Embedding(config.vocab_size, config.hidden_size)
122
+ self.pos_emb = nn.Embedding(config.max_position_embeddings, config.hidden_size)
123
+ self.blocks = nn.ModuleList([Block(config) for _ in range(config.num_hidden_layers)])
124
+ self.ln_f = nn.LayerNorm(config.hidden_size)
125
+ # tied output head: logits = ln_f(x) @ tok_emb.weight.T
126
+ self.head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
127
+ # Tie BEFORE post_init, exactly like fable_lm.FableLM ties before its
128
+ # apply(_init): then the init draw for `head` overwrites the shared
129
+ # tok_emb storage, and same-seed construction is bit-identical to the
130
+ # original. post_init()'s tie_weights() then re-ties a no-op.
131
+ self.head.weight = self.tok_emb.weight
132
+ self.post_init()
133
+
134
+ # ---- embeddings plumbing (needed by PreTrainedModel.tie_weights) ----
135
+
136
+ def get_input_embeddings(self):
137
+ return self.tok_emb
138
+
139
+ def set_input_embeddings(self, value):
140
+ self.tok_emb = value
141
+
142
+ def get_output_embeddings(self):
143
+ return self.head
144
+
145
+ def set_output_embeddings(self, value):
146
+ self.head = value
147
+
148
+ # ---- mask helper ----
149
+
150
+ @staticmethod
151
+ def _make_attention_mask(
152
+ batch_size: int,
153
+ q_len: int,
154
+ kv_len: int,
155
+ past_len: int,
156
+ attention_mask: Optional[torch.Tensor],
157
+ device: torch.device,
158
+ ) -> Optional[torch.Tensor]:
159
+ """Combined causal + padding mask as a bool (B, 1, T, kv) keep-mask.
160
+
161
+ None (no padding) reduces to the pure causal mask, identical to
162
+ fable_lm's `masked_fill(triu(diagonal=1), -inf)` when past_len == 0.
163
+ """
164
+ causal = torch.ones(q_len, kv_len, dtype=torch.bool, device=device).tril(diagonal=past_len)
165
+ keep = causal[None, None, :, :] # (1, 1, T, kv)
166
+ if attention_mask is not None:
167
+ pad = attention_mask.to(device=device, dtype=torch.bool)
168
+ if pad.ndim == 2: # (B, kv) -> (B, 1, 1, kv)
169
+ pad = pad[:, None, None, :]
170
+ keep = keep & pad
171
+ elif past_len == 0 and q_len == kv_len:
172
+ # exactly the original mask (triu diagonal=1 inverted) — skip the
173
+ # extra AND for bit-identical parity with fable_lm
174
+ return keep
175
+ return keep
176
+
177
+ def forward(
178
+ self,
179
+ input_ids: Optional[torch.Tensor] = None,
180
+ attention_mask: Optional[torch.Tensor] = None,
181
+ position_ids: Optional[torch.Tensor] = None,
182
+ past_key_values: Optional[Cache] = None,
183
+ inputs_embeds: Optional[torch.Tensor] = None,
184
+ labels: Optional[torch.Tensor] = None,
185
+ use_cache: Optional[bool] = None,
186
+ output_attentions: Optional[bool] = None,
187
+ output_hidden_states: Optional[bool] = None,
188
+ return_dict: Optional[bool] = None,
189
+ **kwargs,
190
+ ) -> CausalLMOutputWithPast:
191
+ """Causal LM forward. `labels` are shifted internally (HF convention):
192
+ loss = CE(logits[:, :-1], labels[:, 1:])."""
193
+ if input_ids is None and inputs_embeds is None:
194
+ raise ValueError("input_ids or inputs_embeds required")
195
+ use_cache = True if use_cache is None else use_cache
196
+
197
+ if inputs_embeds is not None:
198
+ x = inputs_embeds
199
+ B, T = x.shape[:2]
200
+ else:
201
+ x = self.tok_emb(input_ids)
202
+ B, T = input_ids.shape
203
+
204
+ past_len = past_key_values.get_seq_length() if isinstance(past_key_values, Cache) else 0
205
+ if T + past_len > self.config.max_position_embeddings:
206
+ raise ValueError(
207
+ f"sequence length {T + past_len} exceeds max_position_embeddings "
208
+ f"{self.config.max_position_embeddings}"
209
+ )
210
+
211
+ if position_ids is None:
212
+ position_ids = torch.arange(past_len, past_len + T, device=x.device)
213
+ x = x + self.pos_emb(position_ids) # (B, T, C); pos_emb(ids) broadcasts from (T, C)
214
+
215
+ attn_mask_4d = self._make_attention_mask(
216
+ B, T, T + past_len, past_len, attention_mask, x.device
217
+ )
218
+
219
+ if use_cache and past_key_values is None:
220
+ past_key_values = DynamicCache()
221
+
222
+ for i, blk in enumerate(self.blocks):
223
+ x = blk(x, attn_mask_4d, past_key_values, i, use_cache)
224
+ x = self.ln_f(x)
225
+ logits = self.head(x)
226
+
227
+ loss = None
228
+ if labels is not None:
229
+ shift_logits = logits[..., :-1, :].contiguous()
230
+ shift_labels = labels[..., 1:].contiguous()
231
+ loss = F.cross_entropy(
232
+ shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1), ignore_index=-100
233
+ )
234
+
235
+ return CausalLMOutputWithPast(
236
+ loss=loss,
237
+ logits=logits,
238
+ past_key_values=past_key_values if use_cache else None,
239
+ hidden_states=None,
240
+ attentions=None,
241
+ )
242
+
243
+ def num_params(self, non_embedding: bool = False) -> int:
244
+ """fable_lm-compatible param count (tied head counted once)."""
245
+ n = sum(p.numel() for p in self.parameters())
246
+ if non_embedding:
247
+ n -= self.pos_emb.weight.numel()
248
+ return n
249
+
250
+
251
+ try:
252
+ # Local (non-remote-code) use: registering with the Auto classes makes
253
+ # from_pretrained work on a local hf_model/ dir without trust_remote_code.
254
+ from transformers import AutoModelForCausalLM
255
+
256
+ AutoModelForCausalLM.register(FableConfig, FableLMHeadModel, exist_ok=True)
257
+ except Exception: # pragma: no cover
258
+ pass
special_tokens_map.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "pad_token": {
3
+ "content": "<|pad|>"
4
+ },
5
+ "eos_token": {
6
+ "content": "<|eos|>"
7
+ },
8
+ "bos_token": {
9
+ "content": "<|doc|>"
10
+ },
11
+ "unk_token": null
12
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<|doc|>",
4
+ "eos_token": "<|eos|>",
5
+ "model_max_length": 512,
6
+ "pad_token": "<|pad|>",
7
+ "tokenizer_class": "TokenizersBackend",
8
+ "unk_token": null
9
+ }