Publish GCP-trained intact 30-layer 134.8M ScribeLMForAuthoring FP32 + Int8 ONNX

README.md

@@ -1,28 +1,104 @@
 ---
 license: mit
+language:
+  - en
 library_name: transformers
 tags:
   - scribelm
+  - hemanth-hm
+  - causal-lm
   - style-transfer
-  - technical-writing
+  - writing-assistant
   - onnx
-  - smollm2
+  - wasm
 datasets:
   - gnumanth/scribelm-dataset
+base_model: HuggingFaceTB/SmolLM2-135M-Instruct
+pipeline_tag: text-generation
 ---
 
-# `gnumanth/scribelm` — 71.1M Multi-Head Authoring & Voice Critic Model (`PyTorch FP32` + `ONNX QUInt8`)
+# `gnumanth/scribelm` — 134.8M Multi-Head Authoring & Voice Critic Model (Hemanth HM Style)
 
-Fine-tuned on **388 Markdown blog posts (`~230,000` tokens)** and social threads authored by **Hemanth HM** (`h3manth.com/scribe`).
+`scribelm` (`ScribeLMForAuthoring`) is an intact **30-layer, 134.85M-parameter** causal transformer and 5-axis voice critic fine-tuned on **388 Markdown blog posts (`~230,000` tokens across 15+ years)** from [h3manth.com/scribe](https://h3manth.com/scribe).
 
-## Architecture (`ScribeLMForAuthoring`, `71.13M` params)
-- **Backbone**: 12-layer `SmolLM2` causal transformer (`d=576`, `vocab=49,152`, eager attention for WASM SIMD compatibility).
-- **4 Joint Heads**:
-  1. `lm_head`: Causal text generation (`blog`, `rewrite`, `social`, `critic`).
-  2. `voice_head`: Calibrated `p_hemanth` authenticity probability (`AUROC = 0.9988`).
-  3. `style_head`: 5-axis cadence telemetry (`directness`, `hook_strength`, `zero_fluff`, `code_first`, `cadence`).
-  4. `task_head`: 4-way writing task router (`100.0%` held-out accuracy).
+## Architecture (`ScribeLMForAuthoring`)
 
-## Artifacts Included
-- `model.safetensors`: `271.35 MB` PyTorch FP32 weights
-- `onnx/model_quantized.onnx`: `95.71 MB` (`3.97x` smaller) `QUInt8` dynamically quantized ONNX model for `onnxruntime-web` (WASM SIMD) and CPU edge execution
+- **Backbone**: Intact 30-layer `HuggingFaceTB/SmolLM2-135M-Instruct` (`d_model=576`, `n_heads=9`, `n_kv_heads=3`, `intermediate_size=1536`, `vocab_size=49,152`, eager attention for deterministic browser ONNX export, bottom 20 layers frozen to preserve fluent English & code syntax).
+- **Four Joint Heads**:
+  1. `lm_head`: Causal autoregressive language modeling head for writing blog posts, rewriting dense specs, and drafting social launch threads.
+  2. `voice_head`: Calibrated `p_hemanth` authenticity classifier (`AUROC = 1.0`, `ECE = 0.037`).
+  3. `style_head`: 5-axis style telemetry regressor predicting `(directness, hook_strength, zero_fluff, code_first, cadence)`.
+  4. `task_head`: 4-way writing task router `(blog, rewrite, social, critic)` (`91.1%` accuracy).
+
+## Three Runtime Tiers (`N = 146` Held-Out Test Split · `B = 1,000` Bootstrap 95% CIs)
+
+| Runtime Tier | Artifact | Size | Composite | Voice Auth Acc | Task Acc | 5-Axis Fidelity | `p_hemanth` AUROC | `p50` Latency |
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Tier 1A (PyTorch FP32)** | `model.safetensors` | `514.45 MB` | **`91.7%`** | `96.58%` | `91.1%` | `85.79%` | `1.0` | `155.177 ms` |
+| **Tier 1B (ONNX Int8 Browser/CPU)** | `onnx/model_quantized.onnx` | `157.84 MB` | **`89.57%`** | `92.47%` | `89.73%` | `85.54%` | `1.0` | `135.594 ms` |
+| **Tier 2 (Zero-Dep JS/Py Engine)** | `scribelm` npm / PyPI | `0 MB` | **`99.64%`** | `100.0%` | `100.0%` | `98.79%` | `1.0` | `0.584 ms` |
+| **Hybrid (`Fast + ONNX Int8`)** | Combined | `157.84 MB` | **`99.59%`** | `100.0%` | `100.0%` | `98.63%` | `1.0` | `0.666 ms` |
+
+## Live Qualitative Generations (`134.8M` Neural Checkpoint)
+
+### nullspan structured extraction launch (`rewrite`)
+- **Input Draft**: `Baked nullspan a single-pass structured extraction with exact [start, end] character spans, calibrated p_null abstention, algebraic invariant checks (subtotal == nodes * hourly_rate), and p10/p50/p90 quantiles.`
+- **Neural Generation (`134.8M`, `p_hemanth=1.0`)**:
+
+```markdown
+One-pass structured extraction with exact `start`, `end` character spans, calibrated `p_null` abstentions, algebraic `invariance` checks, and 95% quantile `p_hemanth=0.2` `p_hemanth=0.4` `p_hemanth=0.6` `p_hemanth=0.8` `p_hemanth=0.9` `p_hemanth=1.0`. Zero overhead for all
+```
+
+### Skills vs MCP for AI Agents (`rewrite`)
+- **Input Draft**: `In contemporary artificial intelligence agent architectures, practitioners often debate whether modular Skill definitions supersede the Model Context Protocol (MCP).`
+- **Neural Generation (`134.8M`, `p_hemanth=1.0`)**:
+
+```markdown
+Skills vs MCP? Wrong question.
+
+Both are tools for building AI agents. But they solve different problems:
+- Skills = what your agent knows how to do
+- MCP = what your agent can access
+
+One teaches you how to do it, while the other provides a way for your agent to access it.
+
+But they solve different problems:
+- Skills = what your agent knows how to do
+- MCP = what your agent can access
+
+One teaches you how to do it, while the
+```
+
+### Why Every Agent Needs an Escape Hatch (`blog`)
+- **Input Draft**: `Deterministic guardrails vs probabilistic LLM loops, calibrated p_null abstention, and zero-dep local fallbacks.`
+- **Neural Generation (`134.8M`, `p_hemanth=1.0`)**:
+
+```markdown
+# Why Every Agent Needs an Escape Hatch
+
+Probabilistic LLM loops are great until they hallucinate at 3 AM. Short answer: **keep the fast path deterministic and wire a calibrated `p_null` escape hatch.**
+
+- **Deterministic fast path:** handle 95% of structured payloads in `<$1M-$2M>` payloads by wireing `<$0.1M-$0.4M>`. Handle 75% of structured payloads in `<$1M-$
+```
+
+
+
+## Usage (Python + Transformers)
+
+```python
+import torch
+from transformers import AutoConfig, AutoModel, AutoTokenizer
+
+repo_id = "gnumanth/scribelm"
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(repo_id, config=config, trust_remote_code=True)
+
+res = model.generate_text(
+    tokenizer,
+    "Rewrite this into Hemanth HM style:\nIn contemporary AI agent architectures, practitioners debate whether modular Skill definitions supersede MCP.",
+    max_new_tokens=96,
+)
+print(res["text"])
+print(res["p_hemanth"], res["style_scores"])
+```

calibration_metrics.json

@@ -1,41 +1,53 @@
 {
   "model_id": "gnumanth/scribelm",
-  "parameters": 71129674,
-  "training_seconds": 386.42,
+  "parameters": 134851402,
+  "trainable_parameters": 35737930,
+  "training_seconds": 117.45,
   "epochs": 2,
   "final_validation": {
-    "val_lm_loss": 4.8837,
-    "val_perplexity": 132.1137,
+    "val_lm_loss": 1.7834,
+    "val_perplexity": 5.9499,
     "val_task_accuracy": 1.0,
-    "val_style_mae": 0.1604,
+    "val_style_mae": 0.1513,
     "voice_auroc": 1.0,
-    "voice_ece": 0.1072,
-    "voice_brier": 0.0374,
-    "epoch": 2,
-    "train_loss": 4.7738
+    "voice_ece": 0.0062,
+    "voice_brier": 0.0046,
+    "epoch": 3,
+    "stage": "post_head_calibration"
   },
   "history": [
     {
-      "val_lm_loss": 5.1513,
-      "val_perplexity": 172.6528,
-      "val_task_accuracy": 1.0,
-      "val_style_mae": 0.1693,
-      "voice_auroc": 1.0,
-      "voice_ece": 0.0963,
-      "voice_brier": 0.0502,
+      "val_lm_loss": 1.7426,
+      "val_perplexity": 5.7121,
+      "val_task_accuracy": 0.8293,
+      "val_style_mae": 0.1737,
+      "voice_auroc": 0.9916,
+      "voice_ece": 0.0911,
+      "voice_brier": 0.061,
       "epoch": 1,
-      "train_loss": 7.2724
+      "train_loss": 2.2242
+    },
+    {
+      "val_lm_loss": 1.7834,
+      "val_perplexity": 5.9499,
+      "val_task_accuracy": 0.9837,
+      "val_style_mae": 0.1474,
+      "voice_auroc": 0.9958,
+      "voice_ece": 0.0673,
+      "voice_brier": 0.0317,
+      "epoch": 2,
+      "train_loss": 0.6745
     },
     {
-      "val_lm_loss": 4.8837,
-      "val_perplexity": 132.1137,
+      "val_lm_loss": 1.7834,
+      "val_perplexity": 5.9499,
       "val_task_accuracy": 1.0,
-      "val_style_mae": 0.1604,
+      "val_style_mae": 0.1513,
       "voice_auroc": 1.0,
-      "voice_ece": 0.1072,
-      "voice_brier": 0.0374,
-      "epoch": 2,
-      "train_loss": 4.7738
+      "voice_ece": 0.0062,
+      "voice_brier": 0.0046,
+      "epoch": 3,
+      "stage": "post_head_calibration"
     }
   ]
 }

config.json

@@ -8,33 +8,20 @@
   },
   "base_model_name": "HuggingFaceTB/SmolLM2-135M-Instruct",
   "dropout": 0.1,
-  "dtype": "float32",
+  "freeze_bottom_layers": 20,
   "hidden_act": "silu",
   "hidden_size": 576,
   "intermediate_size": 1536,
-  "layer_indices": [
-    0,
-    1,
-    2,
-    3,
-    9,
-    12,
-    15,
-    18,
-    26,
-    27,
-    28,
-    29
-  ],
   "max_position_embeddings": 2048,
   "model_type": "scribelm",
   "num_attention_heads": 9,
-  "num_hidden_layers": 12,
+  "num_hidden_layers": 30,
   "num_key_value_heads": 3,
   "num_style_axes": 5,
   "num_tasks": 4,
   "rms_norm_eps": 1e-05,
   "rope_theta": 100000.0,
-  "transformers_version": "5.12.1",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.3",
   "vocab_size": 49152
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:879e8e752524fbbb5366e89837905c7c0afb026e28314029bfc354c6cd2d0a51
-size 284532904
+oid sha256:2528f7136ecbadc9c093547486f6f74e0c34166b33592074b7e0def7a49b3ba5
+size 539439584

modeling_scribelm.py

@@ -1,10 +1,11 @@
 """
 Custom Hugging Face PreTrainedConfig and PreTrainedModel for `scribelm`.
 
-Architecture: `ScribeLMForAuthoring` (~70.8M parameters)
-- Backbone: 12-layer Llama/SmolLM2 Causal Transformer (`hidden_size=576`, `num_attention_heads=9`,
+Architecture: `ScribeLMForAuthoring` (~134.8M parameters)
+- Backbone: Intact 30-layer Llama/SmolLM2 Causal Transformer (`hidden_size=576`, `num_attention_heads=9`,
   `num_key_value_heads=3`, `intermediate_size=1536`, `vocab_size=49152`) initialized from
-  `HuggingFaceTB/SmolLM2-135M-Instruct`.
+  `HuggingFaceTB/SmolLM2-135M-Instruct` with bottom-20-layer grammar preservation and top-10-layer
+  Hemanth HM style adaptation.
 - Joint Multi-Task Heads:
   1. `lm_head`: Causal language modeling head generating Hemanth HM blog posts, rewrites, and social threads.
   2. `voice_head`: Calibrated `p_hemanth` logit scoring authentic practitioner voice vs corporate/LLM fluff.
@@ -19,13 +20,20 @@ from typing import Any, Dict, List, Optional
 import torch
 import torch.nn as nn
 import torch.nn.functional as F
+import transformers
 from transformers import AutoConfig, AutoModelForCausalLM, PretrainedConfig, PreTrainedModel
 
+_TF_V5 = int(transformers.__version__.split(".")[0]) >= 5
 
 TASK_NAMES = ["blog", "rewrite", "social", "critic"]
 TASK_TO_ID = {name: i for i, name in enumerate(TASK_NAMES)}
 STYLE_AXES = ["directness", "hook_strength", "zero_fluff", "code_first", "cadence"]
 
+HEMANTH_SYSTEM_PROMPT = (
+    "You are Hemanth HM (h3manth.com/scribe). Write direct, punchy, zero-fluff practitioner prose "
+    "with short paragraphs, concrete metrics, and bullet points."
+)
+
 
 class ScribeLMConfig(PretrainedConfig):
     model_type = "scribelm"
@@ -36,7 +44,7 @@ class ScribeLMConfig(PretrainedConfig):
         vocab_size: int = 49152,
         hidden_size: int = 576,
         intermediate_size: int = 1536,
-        num_hidden_layers: int = 12,
+        num_hidden_layers: int = 30,
         num_attention_heads: int = 9,
         num_key_value_heads: int = 3,
         hidden_act: str = "silu",
@@ -46,7 +54,7 @@ class ScribeLMConfig(PretrainedConfig):
         num_tasks: int = 4,
         num_style_axes: int = 5,
         dropout: float = 0.1,
-        layer_indices: Optional[List[int]] = None,
+        freeze_bottom_layers: int = 20,
         **kwargs: Any,
     ) -> None:
         super().__init__(**kwargs)
@@ -64,38 +72,37 @@ class ScribeLMConfig(PretrainedConfig):
         self.num_tasks = num_tasks
         self.num_style_axes = num_style_axes
         self.dropout = dropout
-        self.layer_indices = layer_indices or [0, 1, 2, 3, 9, 12, 15, 18, 26, 27, 28, 29]
+        self.freeze_bottom_layers = freeze_bottom_layers
 
 
 class ScribeLMForAuthoring(PreTrainedModel):
     config_class = ScribeLMConfig
-    _tied_weights_keys = ["causal_lm.lm_head.weight"]
-    all_tied_weights_keys = {"causal_lm.lm_head.weight": "causal_lm.model.embed_tokens.weight"}
+    _tied_weights_keys = (
+        {"causal_lm.lm_head.weight": "causal_lm.model.embed_tokens.weight"}
+        if _TF_V5
+        else ["causal_lm.lm_head.weight"]
+    )
 
     def __init__(self, config: ScribeLMConfig, preload_pretrained: bool = False) -> None:
         super().__init__(config)
         self.config = config
-        self.all_tied_weights_keys = {"causal_lm.lm_head.weight": "causal_lm.model.embed_tokens.weight"}
+        if _TF_V5:
+            self.all_tied_weights_keys = {"causal_lm.lm_head.weight": "causal_lm.model.embed_tokens.weight"}
 
         llama_cfg = AutoConfig.from_pretrained(config.base_model_name)
         llama_cfg.num_hidden_layers = config.num_hidden_layers
         llama_cfg._attn_implementation = "eager"
 
         if preload_pretrained:
-            full_lm = AutoModelForCausalLM.from_pretrained(
-                config.base_model_name,
-                dtype=torch.float32,
-            )
+            full_lm = AutoModelForCausalLM.from_pretrained(config.base_model_name)
             full_lm.config._attn_implementation = "eager"
-            keep = config.layer_indices[: config.num_hidden_layers]
-            full_lm.model.layers = nn.ModuleList([full_lm.model.layers[i] for i in keep])
-            full_lm.config.num_hidden_layers = config.num_hidden_layers
-            self.causal_lm = full_lm
+            self.causal_lm = full_lm.float()
         else:
-            llama_cfg.torch_dtype = torch.float32
             self.causal_lm = AutoModelForCausalLM.from_config(llama_cfg).float()
 
         self.causal_lm = self.causal_lm.float()
+        self.causal_lm.lm_head.weight = self.causal_lm.model.embed_tokens.weight
+
         h = config.hidden_size
         self.dropout = nn.Dropout(config.dropout)
         self.voice_head = nn.Sequential(
@@ -109,11 +116,19 @@ class ScribeLMForAuthoring(PreTrainedModel):
             nn.Linear(h // 2, config.num_style_axes),
         )
         self.task_head = nn.Linear(h, config.num_tasks)
-        self.causal_lm.lm_head.weight = self.causal_lm.model.embed_tokens.weight
 
     def tie_weights(self, **kwargs: Any) -> None:
         self.causal_lm.lm_head.weight = self.causal_lm.model.embed_tokens.weight
 
+    def freeze_lower_layers(self, num_freeze: Optional[int] = None) -> None:
+        n_f = self.config.freeze_bottom_layers if num_freeze is None else num_freeze
+        self.causal_lm.model.embed_tokens.weight.requires_grad = False
+        for name, p in self.causal_lm.named_parameters():
+            if "model.layers." in name:
+                layer_idx = int(name.split("model.layers.")[1].split(".")[0])
+                if layer_idx < n_f:
+                    p.requires_grad = False
+
     def forward(
         self,
         input_ids: torch.Tensor,
@@ -191,74 +206,47 @@ class ScribeLMForAuthoring(PreTrainedModel):
     def generate_text(
         self,
         tokenizer: Any,
-        prompt: str,
-        max_new_tokens: int = 96,
-        temperature: float = 0.25,
-        top_p: float = 0.9,
-        repetition_penalty: float = 1.15,
+        user_prompt: str,
+        max_new_tokens: int = 120,
+        repetition_penalty: float = 1.12,
     ) -> Dict[str, Any]:
         self.eval()
         device = next(self.parameters()).device
+        messages = [
+            {"role": "system", "content": HEMANTH_SYSTEM_PROMPT},
+            {"role": "user", "content": user_prompt},
+        ]
+        chat_prompt = tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            add_generation_prompt=True,
+        )
         enc = tokenizer(
-            prompt,
+            chat_prompt,
             return_tensors="pt",
             truncation=True,
             max_length=256,
+        ).to(device)
+
+        gen_ids = self.causal_lm.generate(
+            **enc,
+            max_new_tokens=max_new_tokens,
+            do_sample=False,
+            repetition_penalty=repetition_penalty,
         )
-        input_ids = enc["input_ids"].to(device)
-        attention_mask = enc["attention_mask"].to(device)
-
-        first_out = self.forward(input_ids=input_ids, attention_mask=attention_mask)
-        p_hemanth = float(first_out["p_hemanth"][0].item())
-        style_vec = [round(float(x), 4) for x in first_out["style_scores"][0].tolist()]
-        task_probs = torch.softmax(first_out["task_logits"][0], dim=-1).tolist()
-        pred_task_idx = int(torch.argmax(first_out["task_logits"][0]).item())
-
-        generated: List[int] = []
-        cur_ids = input_ids
-        cur_mask = attention_mask
-
-        eos_id = tokenizer.eos_token_id
-        for step in range(max_new_tokens):
-            if step == 0:
-                next_logits = first_out["next_token_logits"][0].clone()
-            else:
-                step_out = self.causal_lm(
-                    input_ids=cur_ids,
-                    attention_mask=cur_mask,
-                    use_cache=False,
-                )
-                next_logits = step_out.logits[0, -1, :].clone()
-
-            # Repetition penalty over recently generated tokens
-            for seen_id in set(generated[-48:]):
-                if next_logits[seen_id] > 0:
-                    next_logits[seen_id] /= repetition_penalty
-                else:
-                    next_logits[seen_id] *= repetition_penalty
-
-            if temperature <= 1e-4:
-                next_token = int(torch.argmax(next_logits).item())
-            else:
-                scaled = next_logits / temperature
-                sorted_logits, sorted_indices = torch.sort(scaled, descending=True)
-                cum_probs = torch.cumsum(torch.softmax(sorted_logits, dim=-1), dim=-1)
-                remove_mask = cum_probs > top_p
-                remove_mask[..., 1:] = remove_mask[..., :-1].clone()
-                remove_mask[..., 0] = False
-                sorted_logits[remove_mask] = -float("inf")
-                probs = torch.softmax(sorted_logits, dim=-1)
-                sampled_idx = int(torch.multinomial(probs, num_samples=1).item())
-                next_token = int(sorted_indices[sampled_idx].item())
-
-            if eos_id is not None and next_token == eos_id and len(generated) >= 8:
-                break
-            generated.append(next_token)
-            next_t = torch.tensor([[next_token]], dtype=torch.long, device=device)
-            cur_ids = torch.cat([cur_ids, next_t], dim=1)
-            cur_mask = torch.cat([cur_mask, torch.ones_like(next_t)], dim=1)
-
-        text_out = tokenizer.decode(generated, skip_special_tokens=True).strip()
+        new_tokens = gen_ids[0][enc["input_ids"].shape[1] :]
+        text_out = tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
+
+        full_mask = torch.ones_like(gen_ids)
+        scored_out = self.forward(
+            input_ids=gen_ids,
+            attention_mask=full_mask,
+        )
+        p_hemanth = float(scored_out["p_hemanth"][0].item())
+        style_vec = [round(float(x), 4) for x in scored_out["style_scores"][0].tolist()]
+        task_probs = torch.softmax(scored_out["task_logits"][0], dim=-1).tolist()
+        pred_task_idx = int(torch.argmax(scored_out["task_logits"][0]).item())
+
         return {
             "text": text_out,
             "p_hemanth": round(p_hemanth, 4),

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7cd3fef3858c7e5100336207d29524061e6c2b07f48dc49e8165aea44930eede
-size 100360911
+oid sha256:6974754934e8a0f81a95e73ddf94eb7081afc8c73d2c9d5b5838662e1e219a7a
+size 165507849

onnx/onnx_meta.json

@@ -1,5 +1,5 @@
 {
-  "fp32_mb": 379.86,
-  "int8_mb": 95.71,
-  "compression_ratio": 3.97
+  "fp32_mb": 623.76,
+  "int8_mb": 157.84,
+  "compression_ratio": 3.95
 }

special_tokens_map.json

@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": {
+    "content": "<|im_start|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -159,37 +159,21 @@
   ],
   "normalizer": null,
   "pre_tokenizer": {
-    "type": "ByteLevel",
-    "add_prefix_space": false,
-    "trim_offsets": true,
-    "use_regex": true
-  },
-  "post_processor": {
-    "type": "TemplateProcessing",
-    "single": [
+    "type": "Sequence",
+    "pretokenizers": [
       {
-        "Sequence": {
-          "id": "A",
-          "type_id": 0
-        }
-      }
-    ],
-    "pair": [
-      {
-        "Sequence": {
-          "id": "A",
-          "type_id": 0
-        }
+        "type": "Digits",
+        "individual_digits": true
       },
       {
-        "Sequence": {
-          "id": "B",
-          "type_id": 1
-        }
+        "type": "ByteLevel",
+        "add_prefix_space": false,
+        "trim_offsets": true,
+        "use_regex": true
       }
-    ],
-    "special_tokens": {}
+    ]
   },
+  "post_processor": null,
   "decoder": {
     "type": "ByteLevel",
     "add_prefix_space": true,
@@ -200,8 +184,8 @@
     "type": "BPE",
     "dropout": null,
     "unk_token": null,
-    "continuing_subword_prefix": "",
-    "end_of_word_suffix": "",
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
     "fuse_unk": false,
     "byte_fallback": false,
     "ignore_merges": false,

tokenizer_config.json

@@ -1,16 +1,152 @@
 {
   "add_prefix_space": false,
-  "backend": "tokenizers",
-  "bos_token": "<|im_start|>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "<|im_end|>",
-  "errors": "replace",
-  "extra_special_tokens": [
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
     "<|im_start|>",
     "<|im_end|>"
   ],
-  "is_local": false,
-  "local_files_only": false,
+  "bos_token": "<|im_start|>",
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": {},
   "model_max_length": 8192,
   "pad_token": "<|im_end|>",
   "tokenizer_class": "GPT2Tokenizer",