Matilda Platypus matilda-platypus-1.5b-base: upper block, Representation Bridge, remote code, model card

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+technical_report.pdf filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+upper_block/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,101 @@
+---
+license: apache-2.0
+base_model:
+- Qwen/Qwen3-1.7B-Base
+- Qwen/Qwen3-0.6B-Base
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- matilda
+- platypus
+- representation-bridge
+- qwen3
+- base-model
+---
+# Matilda Platypus 1.5B (base)
+
+**Any size you want.** Matilda Platypus is a family of intermediate-size language models built by connecting the lower
+layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear **Representation
+Bridge** aligns the hidden representations at the connection. This checkpoint is the **1.5B** member
+(cut `18+10`): 1.53B parameters, 2.7 GFLOP per token, 73% of the
+larger parent's compute. It is a base model (no instruction tuning); see the technical report (`technical_report.pdf`)
+for the method and the full evaluation.
+
+## How this model is built
+
+| Component | Source | Layers | Notes |
+|---|---|---|---|
+| Lower block | `Qwen/Qwen3-1.7B-Base` (28 layers) | 1–18 | frozen at the released weights, loaded from the hub |
+| Representation Bridge | learned | – | one linear map, 2048 → 1024 hidden units, 2.1M parameters (`bridge.safetensors`) |
+| Upper block | `Qwen/Qwen3-0.6B-Base` (28 layers) | 19–28 + final norm + output head | lightly adapted with low-rank updates (merged), shipped as `upper_block/` |
+
+The notation `18+10` means 18 layers of the larger parent followed by 10 layers of the smaller parent. The
+forward pass is: embedding and layers 1–18 of `Qwen3-1.7B-Base` → bridge → layers 19–28 of
+`Qwen3-0.6B-Base` → its final normalisation and output head. The tokenizer is the Qwen3 tokenizer.
+
+## Results
+
+Base-model evaluation with lm-eval-harness 0.4.13, identical protocol for the model and both parents (0-shot
+multiple choice, MMLU 5-shot, GSM8K 8-shot chain-of-thought, BBH 3-shot chain-of-thought, HumanEval 0-shot and MBPP
+3-shot with the generated code executed; HumanEval scored with markdown fences stripped). Scores are proportions.
+
+| Benchmark | Qwen3-0.6B-Base | **Platypus 1.5B** | Qwen3-1.7B-Base |
+|---|---|---|---|
+| HumanEval pass@1 | 0.287 | **0.409** | 0.494 |
+| GSM8K (8-shot CoT) | 0.595 | **0.691** | 0.729 |
+| MMLU (5-shot) | 0.527 | **0.606** | 0.625 |
+| BBH (3-shot CoT) | 0.406 | **0.473** | 0.529 |
+| HellaSwag | 0.538 | **0.595** | 0.665 |
+| ARC-Easy | 0.579 | **0.660** | 0.686 |
+| ARC-Challenge | 0.385 | **0.407** | 0.446 |
+| OpenBookQA | 0.344 | **0.374** | 0.388 |
+| WinoGrande | 0.587 | **0.635** | 0.644 |
+| PIQA | 0.701 | **0.726** | 0.758 |
+| LAMBADA | 0.537 | **0.574** | 0.633 |
+| MBPP pass@1 | 0.390 | **0.456** | 0.566 |
+| BoolQ | 0.697 | **0.777** | 0.790 |
+
+Above the smaller parent on 13 of 13 benchmarks and above the larger parent on 0 of 13, at
+73% of the larger parent's compute (2.7 vs 3.7 GFLOP per token; the
+smaller parent is 1.4). Compute is the report's relative measure, normalised to the larger
+parent; it is not a verified latency or cost measurement. Full tables and figures: `technical_report.pdf`.
+
+## Training recipe (summary)
+
+1. **Initialise the bridge by least squares.** Both released parents are run on a small calibration set and a linear
+   map is fitted between their representations at the connection; initial language-modelling loss ranks candidate cuts.
+2. **Supervised adaptation.** The bridge and light low-rank updates to the retained upper layers are trained on a few
+   billion tokens of mixed web text, code, mathematics and worked solutions, with the lower block frozen at its released weights.
+3. **Reinforcement learning with verifiable rewards.** A short policy-optimisation stage on code problems with unit
+   tests and mathematics problems with checkable answers.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "Maincode/matilda-platypus-1.5b-base"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda()
+inputs = tok("The capital of Australia is", return_tensors="pt").to("cuda")
+out = model.generate(**inputs, max_new_tokens=32)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+The remote code loads the lower block from `Qwen/Qwen3-1.7B-Base`, the upper
+block from `upper_block/` and the bridge from `bridge.safetensors`, truncates both parents to the retained layers and
+runs them through their standard `transformers` forward passes with one KV cache per block. `generate` supports greedy
+decoding and nucleus sampling (`do_sample`, `temperature`, `top_p`). Requires `transformers >= 4.51` (Qwen3 support).
+
+## Limitations
+
+A base model: it continues text and is not instruction-tuned or safety-tuned. Its knowledge and behaviour derive from
+the two Qwen3 parents and the adaptation data. The evaluation above is a base-model protocol shared by the model and its
+parents; differences of a few tenths of a point are within run-to-run variance.
+
+## Citation
+
+Maincode Team (Yue Yang, Arsh Chawla, David Lemphers). *Matilda Platypus: Any Size You Want. Intelligence You Need.*
+Technical report, Maincode / Matilda Research, October 2026. Included in this repository as `technical_report.pdf`.
+Built on Qwen3 (Qwen Team, *Qwen3 Technical Report*, arXiv:2505.09388). Released under the Apache 2.0 license.

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

bridge.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54c079a65656081ae329ab85a7ad33d1b7a7ed5a624d4c1f36591c4580e33342
+size 8392848

config.json

@@ -0,0 +1,21 @@
+{
+ "model_type": "matilda-platypus",
+ "architectures": [
+  "MatildaPlatypusForCausalLM"
+ ],
+ "auto_map": {
+  "AutoConfig": "configuration_platypus.MatildaPlatypusConfig",
+  "AutoModelForCausalLM": "modeling_platypus.MatildaPlatypusForCausalLM"
+ },
+ "lower_model": "Qwen/Qwen3-1.7B-Base",
+ "lower_layers": 18,
+ "upper_model": "upper_block",
+ "upper_start_layer": 18,
+ "upper_layers": 10,
+ "bridge_file": "bridge.safetensors",
+ "lower_hidden_size": 2048,
+ "upper_hidden_size": 1024,
+ "vocab_size": 151936,
+ "torch_dtype": "bfloat16",
+ "transformers_version": "4.56.0"
+}

configuration_platypus.py

@@ -0,0 +1,22 @@
+from transformers import PretrainedConfig
+
+
+class MatildaPlatypusConfig(PretrainedConfig):
+    """A Matilda Platypus model: the lower block of a larger Qwen3 parent, a learned linear Representation Bridge,
+    and the upper block (plus final norm and output head) of a smaller Qwen3 parent."""
+
+    model_type = "matilda-platypus"
+
+    def __init__(self, lower_model="Qwen/Qwen3-8B-Base", lower_layers=25, upper_model="upper_block",
+                 upper_start_layer=25, upper_layers=11, bridge_file="bridge.safetensors",
+                 lower_hidden_size=4096, upper_hidden_size=2560, vocab_size=151936, **kwargs):
+        self.lower_model = lower_model          # hub id, or a sub-folder of this repo
+        self.lower_layers = lower_layers        # k: decoder layers kept from the larger parent (layers 1..k)
+        self.upper_model = upper_model          # sub-folder holding the adapted smaller parent (HF checkpoint)
+        self.upper_start_layer = upper_start_layer  # 0-based index of the first retained upper layer (= j - 1)
+        self.upper_layers = upper_layers        # m: decoder layers kept from the smaller parent
+        self.bridge_file = bridge_file
+        self.lower_hidden_size = lower_hidden_size
+        self.upper_hidden_size = upper_hidden_size
+        self.vocab_size = vocab_size
+        super().__init__(**kwargs)

eval_results.json

@@ -0,0 +1,54 @@
+{
+ "protocol": "lm-eval-harness 0.4.13, base-model protocol, HumanEval fence-stripped",
+ "cut": "18+10",
+ "params_b": 1.53,
+ "gflop_per_token": 2.68,
+ "compute_pct_of_larger_parent": 73,
+ "smaller_parent": {
+  "name": "Qwen3-0.6B-Base",
+  "humaneval": 0.287,
+  "gsm8k_cot": 0.5951,
+  "mmlu": 0.5271,
+  "bbh_cot_fewshot": 0.4061,
+  "hellaswag": 0.5384,
+  "arc_easy": 0.5791,
+  "arc_challenge": 0.3848,
+  "openbookqa": 0.344,
+  "winogrande": 0.5872,
+  "piqa": 0.7013,
+  "lambada_openai": 0.5372,
+  "mbpp": 0.39,
+  "boolq": 0.6972
+ },
+ "model": {
+  "humaneval": 0.409,
+  "gsm8k_cot": 0.6914,
+  "mmlu": 0.6056,
+  "bbh_cot_fewshot": 0.4732,
+  "hellaswag": 0.5951,
+  "arc_easy": 0.6604,
+  "arc_challenge": 0.407,
+  "openbookqa": 0.374,
+  "winogrande": 0.6354,
+  "piqa": 0.7258,
+  "lambada_openai": 0.5742,
+  "mbpp": 0.456,
+  "boolq": 0.7771
+ },
+ "larger_parent": {
+  "name": "Qwen3-1.7B-Base",
+  "humaneval": 0.494,
+  "gsm8k_cot": 0.7293,
+  "mmlu": 0.6249,
+  "bbh_cot_fewshot": 0.529,
+  "hellaswag": 0.6648,
+  "arc_easy": 0.6864,
+  "arc_challenge": 0.4462,
+  "openbookqa": 0.388,
+  "winogrande": 0.644,
+  "piqa": 0.7579,
+  "lambada_openai": 0.633,
+  "mbpp": 0.566,
+  "boolq": 0.7902
+ }
+}

modeling_platypus.py

@@ -0,0 +1,153 @@
+"""Matilda Platypus: lower block of a larger Qwen3 parent -> Representation Bridge -> upper block of a smaller parent.
+
+Self-contained: the two blocks are ordinary Hugging Face Qwen3 checkpoints run through their public forward passes
+(the lower one truncated to its first k layers with its final norm removed, the upper one starting at layer j with
+`inputs_embeds`), so no custom kernels are involved. `generate` keeps one KV cache per block.
+
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    tok = AutoTokenizer.from_pretrained("maincode/matilda-platypus-7b-base")
+    model = AutoModelForCausalLM.from_pretrained("maincode/matilda-platypus-7b-base", trust_remote_code=True,
+                                                 dtype=torch.bfloat16).cuda()
+    out = model.generate(**tok("The capital of Australia is", return_tensors="pt").to("cuda"), max_new_tokens=32)
+"""
+from __future__ import annotations
+
+import os
+from typing import Optional, Tuple
+
+import torch
+import torch.nn as nn
+from safetensors.torch import load_file
+from transformers import AutoModelForCausalLM, PreTrainedModel
+from transformers.cache_utils import DynamicCache
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+from .configuration_platypus import MatildaPlatypusConfig
+
+
+class RepresentationBridge(nn.Module):
+    """y = W x + b on the residual stream, computed in float32 (W: [d_upper, d_lower])."""
+
+    def __init__(self, d_lower: int, d_upper: int):
+        super().__init__()
+        self.proj = nn.Linear(d_lower, d_upper, bias=True)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        with torch.autocast(device_type=x.device.type, enabled=False):
+            return self.proj(x.float())
+
+
+class MatildaPlatypusForCausalLM(PreTrainedModel):
+    config_class = MatildaPlatypusConfig
+    _supports_cache_class = True
+
+    def __init__(self, config: MatildaPlatypusConfig, lower: Optional[PreTrainedModel] = None,
+                 upper: Optional[PreTrainedModel] = None, bridge: Optional[RepresentationBridge] = None):
+        super().__init__(config)
+        self.lower, self.upper, self.bridge = lower, upper, bridge
+
+    # ------------------------------------------------------------------ loading
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *args, dtype=None, torch_dtype=None, token=None,
+                        revision=None, device_map=None, **kwargs):
+        from huggingface_hub import snapshot_download
+        path = str(pretrained_model_name_or_path)
+        local = path if os.path.isdir(path) else snapshot_download(path, token=token, revision=revision)
+        config = MatildaPlatypusConfig.from_pretrained(local)
+        dt = dtype or torch_dtype or torch.bfloat16
+        lower_src = os.path.join(local, config.lower_model) if os.path.isdir(os.path.join(local, config.lower_model)) else config.lower_model
+        lower = AutoModelForCausalLM.from_pretrained(lower_src, dtype=dt, token=token)
+        upper = AutoModelForCausalLM.from_pretrained(os.path.join(local, config.upper_model), dtype=dt)
+        k, j, m = config.lower_layers, config.upper_start_layer, config.upper_layers
+        # lower block: first k decoder layers, no final norm (the bridge takes the raw residual stream)
+        lower.model.layers = nn.ModuleList(list(lower.model.layers)[:k])
+        lower.model.norm = nn.Identity()
+        lower.config.num_hidden_layers = k
+        if getattr(lower.config, "layer_types", None):
+            lower.config.layer_types = list(lower.config.layer_types)[:k]
+        lower.lm_head = nn.Identity()
+        # upper block: layers j.. of the smaller parent, re-indexed so a fresh KV cache is contiguous
+        layers = list(upper.model.layers)[j:j + m]
+        for n, layer in enumerate(layers):
+            if hasattr(layer, "self_attn"):
+                layer.self_attn.layer_idx = n
+            if hasattr(layer, "layer_idx"):
+                layer.layer_idx = n
+        upper.model.layers = nn.ModuleList(layers)
+        upper.config.num_hidden_layers = m
+        if getattr(upper.config, "layer_types", None):
+            upper.config.layer_types = list(upper.config.layer_types)[j:j + m]
+        upper.model.embed_tokens = nn.Identity()  # the upper block never embeds tokens
+        bridge = RepresentationBridge(config.lower_hidden_size, config.upper_hidden_size)
+        t = load_file(os.path.join(local, config.bridge_file))
+        bridge.proj.weight.data.copy_(t["W"].float()); bridge.proj.bias.data.copy_(t["b"].float())
+        for p in lower.parameters():
+            p.requires_grad_(False)
+        model = cls(config, lower, upper, bridge).eval()
+        if device_map is not None and isinstance(device_map, (str, torch.device)) and device_map != "auto":
+            model.to(device_map)
+        return model
+
+    # ------------------------------------------------------------------ forward
+    @property
+    def dtype(self) -> torch.dtype:
+        return self.upper.lm_head.weight.dtype
+
+    def get_input_embeddings(self):
+        return self.lower.model.embed_tokens
+
+    def get_output_embeddings(self):
+        return self.upper.lm_head
+
+    def forward(self, input_ids: torch.LongTensor, attention_mask: Optional[torch.Tensor] = None,
+                past_key_values: Optional[Tuple[DynamicCache, DynamicCache]] = None, use_cache: bool = False,
+                **kwargs) -> CausalLMOutputWithPast:
+        cl, cu = past_key_values if past_key_values is not None else (None, None)
+        if use_cache and cl is None:
+            cl, cu = DynamicCache(), DynamicCache()
+        lo = self.lower.model(input_ids=input_ids, attention_mask=attention_mask, past_key_values=cl, use_cache=use_cache)
+        h = self.bridge(lo.last_hidden_state).to(self.dtype)
+        up = self.upper.model(inputs_embeds=h, attention_mask=attention_mask, past_key_values=cu, use_cache=use_cache)
+        logits = self.upper.lm_head(up.last_hidden_state)
+        return CausalLMOutputWithPast(logits=logits, past_key_values=(cl, cu) if use_cache else None)
+
+    # ------------------------------------------------------------------ generation
+    @torch.no_grad()
+    def generate(self, input_ids: torch.LongTensor, attention_mask: Optional[torch.Tensor] = None,
+                 max_new_tokens: int = 128, do_sample: bool = False, temperature: float = 1.0, top_p: float = 1.0,
+                 eos_token_id: Optional[int] = None, pad_token_id: Optional[int] = None, **kwargs) -> torch.LongTensor:
+        """Greedy or nucleus sampling with KV caches for both blocks; stops when every row has produced EOS."""
+        if eos_token_id is None:
+            eos_token_id = getattr(self.upper.generation_config, "eos_token_id", None) or self.upper.config.eos_token_id
+        if isinstance(eos_token_id, (list, tuple)):
+            eos_token_id = eos_token_id[0]
+        if pad_token_id is None:
+            pad_token_id = getattr(self.upper.config, "pad_token_id", None) or eos_token_id
+        device = input_ids.device
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+        out = self.forward(input_ids, attention_mask=attention_mask, use_cache=True)
+        caches = out.past_key_values
+        seq = input_ids
+        done = torch.zeros(input_ids.shape[0], dtype=torch.bool, device=device)
+        for _ in range(max_new_tokens):
+            logits = out.logits[:, -1, :].float()
+            if do_sample:
+                logits = logits / max(temperature, 1e-5)
+                if top_p < 1.0:
+                    sorted_logits, idx = torch.sort(logits, descending=True)
+                    probs = torch.softmax(sorted_logits, dim=-1)
+                    keep = probs.cumsum(-1) - probs < top_p
+                    sorted_logits[~keep] = float("-inf")
+                    logits = torch.full_like(logits, float("-inf")).scatter(1, idx, sorted_logits)
+                nxt = torch.multinomial(torch.softmax(logits, dim=-1), 1).squeeze(1)
+            else:
+                nxt = logits.argmax(-1)
+            nxt = torch.where(done, torch.full_like(nxt, pad_token_id), nxt)
+            seq = torch.cat([seq, nxt[:, None]], dim=1)
+            attention_mask = torch.cat([attention_mask, torch.ones_like(nxt[:, None])], dim=1)
+            done |= nxt == eos_token_id
+            if bool(done.all()):
+                break
+            out = self.forward(nxt[:, None], attention_mask=attention_mask, past_key_values=caches, use_cache=True)
+        return seq

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

technical_report.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c302983953d3557bf527187068de754fd433517970cb09d8e0c1a611993d03cb
+size 1279700

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

upper_block/added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

upper_block/chat_template.jinja

@@ -0,0 +1,85 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set content = message.content %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in message.content %}
+                {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
+                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

upper_block/config.json

@@ -0,0 +1,60 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.56.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

upper_block/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "max_new_tokens": 2048,
+  "transformers_version": "4.56.0"
+}

upper_block/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95edbe59ec0ae705facc2eee3c8ab7057272fac2d36b355e4dcce8fe4435df2f
+size 1192135096

upper_block/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

upper_block/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

upper_block/tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}