Skill Cartridges

Deletion Tuning masks used as skill cartridges: small files that switch off a fraction of a percent of a base model's MLP gate-projection weights to improve one skill. Apply one to the frozen base model at request time; eject it and the base weights come back bit for bit. A cartridge only deletes weights. It never writes new values.

📄 Paper and code: github.com/austindixson/skill-cartridges (PDF: paper/skill-cartridges.pdf)

Built on Deletion Tuning

These masks were trained with Deletion Tuning by Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou and Mengye Ren, The Surprising Effectiveness of Deleting Weights in LLM Post-Training (arXiv:2611.10866; code: agentic-learning-ai-lab/deleting-weights; their masks: Jacklu0831/deletion-tuning-masks). The training objective, Wanda initialization, sharpness schedule, regularizer, mask file format and the ToolAlpaca and Science settings are theirs, as is the idea of combining masks by union. ToolAlpaca and Science data and scorers come from Shenfeld et al., Self-Distillation Enables Continual Learning.

Cartridges in this repo

All scores are from the paper. "Base → cartridge" is the same model with and without the mask, scored on the same items; gained/lost are paired; p is an exact two-sided McNemar test.

Folder	Base model	Skill	Base → cartridge	Gained / lost	Deleted	Size
`qwen3.5-9b/toolalpaca-seed1`	Qwen/Qwen3.5-9B	ToolAlpaca (97)	42 → 69 (bf16); 38 → 66 (MLX 4-bit)	27 / 0; 28 / 0	10,702,823 (0.6645%)	21.1 MB
`qwen3.5-9b/toolalpaca-seed2`	Qwen/Qwen3.5-9B	ToolAlpaca (97)	42 → 71; 38 → 66 (4-bit)	30 / 1; 28 / 0	10,950,222 (0.6799%)	21.5 MB
`qwen3.5-9b/toolalpaca-seed3`	Qwen/Qwen3.5-9B	ToolAlpaca (97)	42 → 68; 38 → 61 (4-bit)	26 / 0; 24 / 1	10,819,819 (0.6718%)	21.3 MB
`qwen3.5-9b/science`	Qwen/Qwen3.5-9B	SciKnowEval Chem L-3 (507)	120 → 316 (MLX bf16)	211 / 15	8,696,798 (0.5400%)	17.9 MB
`qwen3.5-9b/recall-seed1`	Qwen/Qwen3.5-9B	Latest-value recall (synthetic, 200)	89 → 200; longer held-out 8 → 100	111 / 0; 92 / 0	5,078,363 (0.315%)	11.3 MB
`qwen3.5-9b/recall-seed2`	Qwen/Qwen3.5-9B	Latest-value recall (synthetic, 200)	90 → 200; longer held-out 8 → 99	not recorded per item	4,719,792 (0.293%)	10.6 MB
`qwen3.8-27b/toolalpaca-run1`	Qwen/Qwen3.8-27B	ToolAlpaca (97)	42 → 65 (bf16); 41 → 67 (MLX 4-bit)	29 / 6; 28 / 2	28,730,353 (0.5037%)	60.0 MB
`lfm3-1.7b/toolalpaca`	LiquidAI/LFM3-1.7B	ToolAlpaca (97)	11 → 59	48 / 0	2,690,252 (0.713%)	5.2 MB

Training seeds: toolalpaca-seed1 used seed 0, seed2 seed 2, seed3 seed 3; recall-seed1 seed 0, recall-seed2 seed 2. The Science cartridge stopped at 330 of 400 planned steps; the 27B run at 272 steps on two GPUs. Each folder holds mask/ (the cartridge), summary.json (the training job's own record: arguments, scores, timings) and metrics.jsonl (per-step loss and deletion rate).

Stacking. The union of toolalpaca-seed1 and science keeps 87.5% of the tool gain (45 → 69 alone, 66 union, MLX bf16) and 90.8% of the science gain (120 → 316 alone, 298 union). The 18-item science drop from solo to union is not significant (p = 0.105).

Forgetting (27B, 4-bit, 34% subset of HellaSwag, MMLU, ARC-Challenge and WinoGrande). Mean −0.76 points (paired bootstrap 95% CI −1.41 to −0.13); no task down by more than 1.2. WikiText-2 perplexity 7.5956 off, 7.5712 on.

Not included. A code cartridge trained on MBPP reference solutions made the 9B worse (MBPP 349 → 293 of 500, HumanEval 139 → 114 of 164). It is reported in the paper and not distributed here.

Caveats. ToolAlpaca has 97 items. Part of each exact-match gain may be formatting; for LFM3 it is a large part (median response length drops from 1,152 to 205 characters). The recall task is synthetic. There is no LoRA baseline yet. See the paper's limitations section.

Format

Upstream Deletion Tuning packbits_zstd format. mask/model.safetensors holds, per decoder layer, model.layers.{L}.mlp.gate_proj.weight.shape and .packed, where packed = zstd(np.packbits(keep.ravel())) and bit 1 = keep, 0 = delete (feed_forward.w1 for LFM3). mask/config.json gives total_params and zero_params.

Apply and eject

hf download austindixson/skill-cartridges --include "qwen3.5-9b/toolalpaca-seed1/*" --local-dir cartridges
git clone https://github.com/austindixson/skill-cartridges

MLX, 4-bit, Apple silicon (live toggle):

import sys; sys.path.insert(0, "skill-cartridges/mlx")
from mlx_lm import load
import cartridge, quant_cartridge as qc

model, tok = load("mlx-community/Qwen3.5-9B-MLX-4bit")
keep = cartridge.load_mask("cartridges/qwen3.5-9b/toolalpaca-seed1/mask")

saved = qc.apply_nearest_zero(model, keep)   # cartridge in (warm swap measured at 0.04 ms)
# ... generate ...
qc.eject(model, saved)                        # base weights restored exactly

Stack cartridges with cartridge.combine([keep_a, keep_b], "union"). For bf16 MLX models use cartridge.apply_dense / cartridge.eject_dense. The MLX apply functions target Qwen-style mlp.gate_proj layers; for the LFM3 cartridge use PyTorch. In PyTorch, upstream's src/apply_mask.py reads this format directly.

License

Each cartridge follows its base model's license:

qwen3.5-9b/* and qwen3.8-27b/*: Apache-2.0 (as are Qwen/Qwen3.5-9B and Qwen/Qwen3.8-27B). Text in LICENSE-APACHE-2.0.
lfm3-1.7b/*: LFM Open License v1.0 (LiquidAI/LFM3-1.7B's license, including its commercial-use threshold). Text in lfm3-1.7b/toolalpaca/LICENSE.

The code that trains and applies them is MIT, the same as upstream Deletion Tuning. See LICENSE.md.

Citation

@misc{dixson2026skillcartridges,
  title  = {Skill Cartridges: Composable, Hot-Swappable Deletion Masks for Small Language Models},
  author = {Dixson, Austin},
  year   = {2026},
  url    = {https://github.com/austindixson/skill-cartridges}
}

@article{lu2026deleting,
  title   = {The Surprising Effectiveness of Deleting Weights in {LLM} Post-Training},
  author  = {Lu, Jack and Yang, Zhenbang and Lasby, Mike and Pote, Tejas and Ioannou, Yani and Ren, Mengye},
  journal = {arXiv preprint arXiv:2611.10866},
  year    = {2026}
}