README.md

---
license: apache-2.0
base_model:
- Qwen/Qwen3-1.7B-Base
- Qwen/Qwen3-0.6B-Base
library_name: transformers
pipeline_tag: text-generation
tags:
- matilda
- platypus
- representation-bridge
- qwen3
- base-model
---
# Matilda Platypus 1.5B (base)

**Any size you want.** Matilda Platypus is a family of intermediate-size language models built by connecting the lower
layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear **Representation
Bridge** aligns the hidden representations at the connection. This checkpoint is the **1.5B** member
(cut `18+10`): 1.53B parameters, 2.7 GFLOP per token, 73% of the
larger parent's compute. It is a base model (no instruction tuning); see the technical report (`technical_report.pdf`)
for the method and the full evaluation.

## How this model is built

| Component | Source | Layers | Notes |
|---|---|---|---|
| Lower block | `Qwen/Qwen3-1.7B-Base` (28 layers) | 1–18 | frozen at the released weights, loaded from the hub |
| Representation Bridge | learned | – | one linear map, 2048 → 1024 hidden units, 2.1M parameters (`bridge.safetensors`) |
| Upper block | `Qwen/Qwen3-0.6B-Base` (28 layers) | 19–28 + final norm + output head | lightly adapted with low-rank updates (merged), shipped as `upper_block/` |

The notation `18+10` means 18 layers of the larger parent followed by 10 layers of the smaller parent. The
forward pass is: embedding and layers 1–18 of `Qwen3-1.7B-Base` → bridge → layers 19–28 of
`Qwen3-0.6B-Base` → its final normalisation and output head. The tokenizer is the Qwen3 tokenizer.

## Results

Base-model evaluation with lm-eval-harness 0.4.13, identical protocol for the model and both parents (0-shot
multiple choice, MMLU 5-shot, GSM8K 8-shot chain-of-thought, BBH 3-shot chain-of-thought, HumanEval 0-shot and MBPP
3-shot with the generated code executed; HumanEval scored with markdown fences stripped). Scores are proportions.

| Benchmark | Qwen3-0.6B-Base | **Platypus 1.5B** | Qwen3-1.7B-Base |
|---|---|---|---|
| HumanEval pass@1 | 0.287 | **0.409** | 0.494 |
| GSM8K (8-shot CoT) | 0.595 | **0.691** | 0.729 |
| MMLU (5-shot) | 0.527 | **0.606** | 0.625 |
| BBH (3-shot CoT) | 0.406 | **0.473** | 0.529 |
| HellaSwag | 0.538 | **0.595** | 0.665 |
| ARC-Easy | 0.579 | **0.660** | 0.686 |
| ARC-Challenge | 0.385 | **0.407** | 0.446 |
| OpenBookQA | 0.344 | **0.374** | 0.388 |
| WinoGrande | 0.587 | **0.635** | 0.644 |
| PIQA | 0.701 | **0.726** | 0.758 |
| LAMBADA | 0.537 | **0.574** | 0.633 |
| MBPP pass@1 | 0.390 | **0.456** | 0.566 |
| BoolQ | 0.697 | **0.777** | 0.790 |

Above the smaller parent on 13 of 13 benchmarks and above the larger parent on 0 of 13, at
73% of the larger parent's compute (2.7 vs 3.7 GFLOP per token; the
smaller parent is 1.4). Compute is the report's relative measure, normalised to the larger
parent; it is not a verified latency or cost measurement. Full tables and figures: `technical_report.pdf`.

## Training recipe (summary)

1. **Initialise the bridge by least squares.** Both released parents are run on a small calibration set and a linear
   map is fitted between their representations at the connection; initial language-modelling loss ranks candidate cuts.
2. **Supervised adaptation.** The bridge and light low-rank updates to the retained upper layers are trained on a few
   billion tokens of mixed web text, code, mathematics and worked solutions, with the lower block frozen at its released weights.
3. **Reinforcement learning with verifiable rewards.** A short policy-optimisation stage on code problems with unit
   tests and mathematics problems with checkable answers.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Maincode/Matilda-Platypus-1.5B-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda()
inputs = tok("The capital of Australia is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))
```

The remote code loads the lower block from `Qwen/Qwen3-1.7B-Base`, the upper
block from `upper_block/` and the bridge from `bridge.safetensors`, truncates both parents to the retained layers and
runs them through their standard `transformers` forward passes with one KV cache per block. `generate` follows the
standard `transformers` keyword arguments (`max_new_tokens` / `max_length`, `min_new_tokens`, `stopping_criteria`,
`logits_processor`, `do_sample`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `eos_token_id`, `pad_token_id`,
`streamer`), so the model runs unchanged under lm-eval-harness (`--model hf --model_args pretrained=Maincode/Matilda-Platypus-1.5B-Base,trust_remote_code=True`).
Greedy and sampled decoding only; beam search raises `NotImplementedError`. Requires `transformers >= 4.56` (tested on 4.56 and 5.x).

Remote-code revision 2 (2026-10-31): revision 1 ignored `max_length` and `stopping_criteria` and always produced 128 new
tokens, which truncated chain-of-thought answers under lm-eval and under-reported generative benchmarks (GSM8K read ~0.58
instead of the table value); `tie_weights()` is now a no-op at the top level. Model weights are unchanged.

## Limitations

A base model: it continues text and is not instruction-tuned or safety-tuned. Its knowledge and behaviour derive from
the two Qwen3 parents and the adaptation data. The evaluation above is a base-model protocol shared by the model and its
parents; differences of a few tenths of a point are within run-to-run variance.

## Citation

Maincode Team (Yue Yang, Arsh Chawla, David Lemphers). *Matilda Platypus: Any Size You Want. Intelligence You Need.*
Technical report, Maincode / Matilda Research, October 2026. Included in this repository as `technical_report.pdf`.
Built on Qwen3 (Qwen Team, *Qwen3 Technical Report*, arXiv:2505.09388). Released under the Apache 2.0 license.