Remote code revision 2: generate() honours max_length / stopping_criteria / logits_processor and the other standard transformers arguments (revision 1 always produced 128 new tokens, truncating CoT under lm-eval); tie_weights() is a no-op at the top level; card documents the change. Weights unchanged.
059efd9 verified | license: apache-2.0 | |
| base_model: | |
| - Qwen/Qwen3-1.7B-Base | |
| - Qwen/Qwen3-0.6B-Base | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - matilda | |
| - platypus | |
| - representation-bridge | |
| - qwen3 | |
| - base-model | |
| # Matilda Platypus 1.6B (base) | |
| **Any size you want.** Matilda Platypus is a family of intermediate-size language models built by connecting the lower | |
| layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear **Representation | |
| Bridge** aligns the hidden representations at the connection. This checkpoint is the **1.6B** member | |
| (cut `20+8`): 1.60B parameters, 2.8 GFLOP per token, 77% of the | |
| larger parent's compute. It is a base model (no instruction tuning); see the technical report (`technical_report.pdf`) | |
| for the method and the full evaluation. | |
| ## How this model is built | |
| | Component | Source | Layers | Notes | | |
| |---|---|---|---| | |
| | Lower block | `Qwen/Qwen3-1.7B-Base` (28 layers) | 1–20 | lightly adapted alongside the upper block and shipped in this repo as `lower_block/` | | |
| | Representation Bridge | learned | – | one linear map, 2048 → 1024 hidden units, 2.1M parameters (`bridge.safetensors`) | | |
| | Upper block | `Qwen/Qwen3-0.6B-Base` (28 layers) | 21–28 + final norm + output head | lightly adapted with low-rank updates (merged), shipped as `upper_block/` | | |
| The notation `20+8` means 20 layers of the larger parent followed by 8 layers of the smaller parent. The | |
| forward pass is: embedding and layers 1–20 of `Qwen3-1.7B-Base` → bridge → layers 21–28 of | |
| `Qwen3-0.6B-Base` → its final normalisation and output head. The tokenizer is the Qwen3 tokenizer. | |
| ## Results | |
| Base-model evaluation with lm-eval-harness 0.4.13, identical protocol for the model and both parents (0-shot | |
| multiple choice, MMLU 5-shot, GSM8K 8-shot chain-of-thought, BBH 3-shot chain-of-thought, HumanEval 0-shot and MBPP | |
| 3-shot with the generated code executed; HumanEval scored with markdown fences stripped). Scores are proportions. | |
| | Benchmark | Qwen3-0.6B-Base | **Platypus 1.6B** | Qwen3-1.7B-Base | | |
| |---|---|---|---| | |
| | HumanEval pass@1 | 0.287 | **0.348** | 0.494 | | |
| | GSM8K (8-shot CoT) | 0.595 | **0.686** | 0.729 | | |
| | MMLU (5-shot) | 0.527 | **0.616** | 0.625 | | |
| | BBH (3-shot CoT) | 0.406 | **0.500** | 0.529 | | |
| | HellaSwag | 0.538 | **0.593** | 0.665 | | |
| | ARC-Easy | 0.579 | **0.638** | 0.686 | | |
| | ARC-Challenge | 0.385 | **0.399** | 0.446 | | |
| | OpenBookQA | 0.344 | **0.390** | 0.388 | | |
| | WinoGrande | 0.587 | **0.656** | 0.644 | | |
| | PIQA | 0.701 | **0.724** | 0.758 | | |
| | LAMBADA | 0.537 | **0.599** | 0.633 | | |
| | MBPP pass@1 | 0.390 | **0.444** | 0.566 | | |
| | BoolQ | 0.697 | **0.757** | 0.790 | | |
| Above the smaller parent on 13 of 13 benchmarks and above the larger parent on 2 of 13, at | |
| 77% of the larger parent's compute (2.8 vs 3.7 GFLOP per token; the | |
| smaller parent is 1.4). Compute is the report's relative measure, normalised to the larger | |
| parent; it is not a verified latency or cost measurement. Full tables and figures: `technical_report.pdf`. | |
| ## Training recipe (summary) | |
| 1. **Initialise the bridge by least squares.** Both released parents are run on a small calibration set and a linear | |
| map is fitted between their representations at the connection; initial language-modelling loss ranks candidate cuts. | |
| 2. **Supervised adaptation.** The bridge and light low-rank updates to the retained upper layers are trained on a few | |
| billion tokens of mixed web text, code, mathematics and worked solutions; the lower block receives light low-rank updates too. | |
| 3. **Reinforcement learning with verifiable rewards.** A short policy-optimisation stage on code problems with unit | |
| tests and mathematics problems with checkable answers. | |
| ## Usage | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| repo = "Maincode/Matilda-Platypus-1.6B-Base" | |
| tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) | |
| model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda() | |
| inputs = tok("The capital of Australia is", return_tensors="pt").to("cuda") | |
| out = model.generate(**inputs, max_new_tokens=32) | |
| print(tok.decode(out[0], skip_special_tokens=True)) | |
| ``` | |
| The remote code loads the lower block from `Qwen/Qwen3-1.7B-Base` (here: the adapted copy in `lower_block/`), the upper | |
| block from `upper_block/` and the bridge from `bridge.safetensors`, truncates both parents to the retained layers and | |
| runs them through their standard `transformers` forward passes with one KV cache per block. `generate` follows the | |
| standard `transformers` keyword arguments (`max_new_tokens` / `max_length`, `min_new_tokens`, `stopping_criteria`, | |
| `logits_processor`, `do_sample`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `eos_token_id`, `pad_token_id`, | |
| `streamer`), so the model runs unchanged under lm-eval-harness (`--model hf --model_args pretrained=Maincode/Matilda-Platypus-1.6B-Base,trust_remote_code=True`). | |
| Greedy and sampled decoding only; beam search raises `NotImplementedError`. Requires `transformers >= 4.56` (tested on 4.56 and 5.x). | |
| Remote-code revision 2 (2026-10-31): revision 1 ignored `max_length` and `stopping_criteria` and always produced 128 new | |
| tokens, which truncated chain-of-thought answers under lm-eval and under-reported generative benchmarks (GSM8K read ~0.58 | |
| instead of the table value); `tie_weights()` is now a no-op at the top level. Model weights are unchanged. | |
| ## Limitations | |
| A base model: it continues text and is not instruction-tuned or safety-tuned. Its knowledge and behaviour derive from | |
| the two Qwen3 parents and the adaptation data. The evaluation above is a base-model protocol shared by the model and its | |
| parents; differences of a few tenths of a point are within run-to-run variance. | |
| ## Citation | |
| Maincode Team (Yue Yang, Arsh Chawla, David Lemphers). *Matilda Platypus: Any Size You Want. Intelligence You Need.* | |
| Technical report, Maincode / Matilda Research, October 2026. Included in this repository as `technical_report.pdf`. | |
| Built on Qwen3 (Qwen Team, *Qwen3 Technical Report*, arXiv:2505.09388). Released under the Apache 2.0 license. | |