Maincode/Matilda-Platypus-1.6B-Base

Matilda Platypus 1.6B (base)

Any size you want. Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 1.6B member (cut 20+8): 1.60B parameters, 2.8 GFLOP per token, 77% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

How this model is built

Component	Source	Layers	Notes
Lower block	Qwen/Qwen3-1.7B-Base (28 layers)	1–20	lightly adapted alongside the upper block and shipped in this repo as lower_block/
Representation Bridge	learned	–	one linear map, 2048 → 1024 hidden units, 2.1M parameters (bridge.safetensors)
Upper block	Qwen/Qwen3-0.6B-Base (28 layers)	21–28 + final norm + output head	lightly adapted with low-rank updates (merged), shipped as upper_block/

The notation 20+8 means 20 layers of the larger parent followed by 8 layers of the smaller parent. The forward pass is: embedding and layers 1–20 of Qwen3-1.7B-Base → bridge → layers 21–28 of Qwen3-0.6B-Base → its final normalisation and output head. The tokenizer is the Qwen3 tokenizer.

Results

Base-model evaluation with lm-eval-harness 0.4.13, identical protocol for the model and both parents (0-shot multiple choice, MMLU 5-shot, GSM8K 8-shot chain-of-thought, BBH 3-shot chain-of-thought, HumanEval 0-shot and MBPP 3-shot with the generated code executed; HumanEval scored with markdown fences stripped). Scores are proportions.

Benchmark	Qwen3-0.6B-Base	Platypus 1.6B	Qwen3-1.7B-Base
HumanEval pass@1	0.287	0.348	0.494
GSM8K (8-shot CoT)	0.595	0.686	0.729
MMLU (5-shot)	0.527	0.616	0.625
BBH (3-shot CoT)	0.406	0.500	0.529
HellaSwag	0.538	0.593	0.665
ARC-Easy	0.579	0.638	0.686
ARC-Challenge	0.385	0.399	0.446
OpenBookQA	0.344	0.390	0.388
WinoGrande	0.587	0.656	0.644
PIQA	0.701	0.724	0.758
LAMBADA	0.537	0.599	0.633
MBPP pass@1	0.390	0.444	0.566
BoolQ	0.697	0.757	0.790

Above the smaller parent on 13 of 13 benchmarks and above the larger parent on 2 of 13, at 77% of the larger parent's compute (2.8 vs 3.7 GFLOP per token; the smaller parent is 1.4). Compute is the report's relative measure, normalised to the larger parent; it is not a verified latency or cost measurement. Full tables and figures: technical_report.pdf.

Training recipe (summary)

1. Initialise the bridge by least squares. Both released parents are run on a small calibration set and a linear map is fitted between their representations at the connection; initial language-modelling loss ranks candidate cuts.
2. Supervised adaptation. The bridge and light low-rank updates to the retained upper layers are trained on a few billion tokens of mixed web text, code, mathematics and worked solutions; the lower block receives light low-rank updates too.
3. Reinforcement learning with verifiable rewards. A short policy-optimisation stage on code problems with unit tests and mathematics problems with checkable answers.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Maincode/Matilda-Platypus-1.6B-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda()
inputs = tok("The capital of Australia is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))

The remote code loads the lower block from Qwen/Qwen3-1.7B-Base (here: the adapted copy in lower_block/), the upper block from upper_block/ and the bridge from bridge.safetensors, truncates both parents to the retained layers and runs them through their standard transformers forward passes with one KV cache per block. generate follows the standard transformers keyword arguments (max_new_tokens / max_length, min_new_tokens, stopping_criteria, logits_processor, do_sample, temperature, top_k, top_p, repetition_penalty, eos_token_id, pad_token_id, streamer), so the model runs unchanged under lm-eval-harness (--model hf --model_args pretrained=Maincode/Matilda-Platypus-1.6B-Base,trust_remote_code=True). Greedy and sampled decoding only; beam search raises NotImplementedError. Requires transformers >= 4.56 (tested on 4.56 and 5.x).

Remote-code revision 2 (2026-10-31): revision 1 ignored max_length and stopping_criteria and always produced 128 new tokens, which truncated chain-of-thought answers under lm-eval and under-reported generative benchmarks (GSM8K read ~0.58 instead of the table value); tie_weights() is now a no-op at the top level. Model weights are unchanged.

Limitations

A base model: it continues text and is not instruction-tuned or safety-tuned. Its knowledge and behaviour derive from the two Qwen3 parents and the adaptation data. The evaluation above is a base-model protocol shared by the model and its parents; differences of a few tenths of a point are within run-to-run variance.

Citation

Maincode Team (Yue Yang, Arsh Chawla, David Lemphers). Matilda Platypus: Any Size You Want. Intelligence You Need. Technical report, Maincode / Matilda Research, October 2026. Included in this repository as technical_report.pdf. Built on Qwen3 (Qwen Team, Qwen3 Technical Report, arXiv:2505.09388). Released under the Apache 2.0 license.