| --- |
| base_model: |
| - bottlecapai/ThinkingCap-Qwen3.6-27B |
| - ProCreations/grug-27b |
| base_model_relation: merge |
| library_name: transformers |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - merge |
| - layer-splice |
| - frankenmerge |
| - qwen3_5 |
| - token-efficient |
| - reasoning |
| --- |
| |
| # GrugCap-27B |
|
|
| A layer splice, not a finetune. Decoder blocks 10–44 of |
| [ProCreations/grug-27b](https://huggingface.co/ProCreations/grug-27b) were copied |
| byte-for-byte into |
| [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B), |
| replacing that range wholesale. Everything outside the range is ThinkingCap's. |
|
|
| Both parents are Apache-2.0 finetunes of |
| [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B), which is what makes a |
| raw weight-space splice coherent: their residual streams share a basis, so |
| transplanted layers land in a space the surrounding layers still understand. |
| Splicing across unrelated pretraining runs does not work this way and produces |
| noise. |
|
|
| The premise being tested is that the middle depth band of a transformer carries |
| most of the feature-composition work, so swapping it swaps "how the model |
| thinks" while the outer layers keep the host's tokenization behaviour, output |
| formatting and answer style. Blocks 10–44 are 15–70% of depth, a heuristic |
| choice, not a discovered boundary. |
|
|
| ## Composition |
|
|
| | Component | Source | |
| |---|---| |
| | `embed_tokens`, `lm_head`, final `norm` | ThinkingCap (identical in both parents) | |
| | Decoder blocks 0–9 | ThinkingCap | |
| | **Decoder blocks 10–44** | **grug-27b** | |
| | Decoder blocks 45–63 | ThinkingCap | |
| | Vision tower, MTP head | ThinkingCap | |
| | Tokenizer, chat template, config | ThinkingCap | |
|
|
| 463 of 1199 tensors come from grug: 24.8 GiB of 51.7 GiB, or 47.9% by weight. |
| The 35 spliced blocks are 26 linear-attention and 9 full-attention layers, the |
| hybrid pattern placing full attention every fourth layer. |
|
|
| Provenance was verified after the build by hashing tensors in the output against |
| the same tensors fetched from both parent repos. The boundary is exact: |
| `blk.44.mlp.down_proj` matches grug and not ThinkingCap, `blk.45.mlp.down_proj` |
| matches ThinkingCap and not grug, with no unexplained tensors. |
|
|
| A useful incidental finding: the two parents have **byte-identical** |
| `embed_tokens`, `lm_head`, every layernorm and every `conv1d` weight. They differ |
| only in projection matrices, in all 64 blocks. That is the expected signature of |
| grug's LoRA-on-linears method, and it means a provenance check that samples only |
| layernorms cannot distinguish the parents at all. |
|
|
| ## What was tested |
|
|
| On a Q8_0 GGUF under llama.cpp on 2×RTX 3090: |
| |
| - Loads and generates cleanly; thinking mode intact and correctly delimited. |
| - A small battery of reasoning prompts with deterministic answers: 4/4 on the |
| auto-graded items, with the free-form items (bracket-matching code, strict JSON |
| formatting, a 24-game expression, a deliberately unsatisfiable ordering puzzle) |
| all correct on inspection. |
| - 220.6 tok/s prompt processing, 25.2 tok/s generation. |
| |
| Thinking traces are terse and enumerative, closer to grug's register than to a |
| verbose reasoner. Mean trace length across the battery was 155 characters. On the |
| 24-game prompt it enumerated ten dead ends and verified the winner in 438 |
| characters. |
| |
| ## What was NOT tested |
| |
| Be skeptical of this model until someone runs real evals. In particular: |
| |
| - **No head-to-head comparison against either parent.** No claim is made that |
| this is better, or even as good. |
| - No standard benchmarks (GSM8K, MMLU-Pro, GPQA, HumanEval). The battery above is |
| a smoke test, not an evaluation. |
| - No perplexity measurement. |
| - No long-context, multi-turn, agentic, or tool-use testing. |
| - Vision inputs untested, though the tower and projector are carried through. |
| - No check for seam artifacts at the two splice boundaries, where a short |
| healing finetune would be the obvious remedy if quality problems appear. |
| |
| ## Files |
| |
| BF16 safetensors in 11 shards, plus `model-base-aux.safetensors` holding |
| ThinkingCap's MTP draft head. That aux file is not in the weight index; move it |
| out of the directory before running `convert_hf_to_gguf.py` or the converter will |
| trip over it. |
|
|
| ## Reproducing |
|
|
| ```bash |
| python splice_remote.py \ |
| --recipient bottlecapai/ThinkingCap-Qwen3.6-27B \ |
| --donor ProCreations/grug-27b \ |
| --start 10 --end 45 \ |
| --out ./GrugCap-27B |
| ``` |
|
|
| The splice is IO-bound, so re-cutting at different boundaries is cheap relative |
| to any training. If you want a different band, change `--start`/`--end`; the |
| tooling streams tensors by HTTP range request and never stores either parent |
| locally. |
|
|
| ## Credits |
|
|
| - [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B) — host model, Apache-2.0 |
| - [ProCreations/grug-27b](https://huggingface.co/ProCreations/grug-27b) — donor blocks, Apache-2.0 |
| - [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — shared base |
|
|
| Neither parent's authors were involved in or endorse this merge. |
|
|
