pragmaticcs/PentaCoder-9B

Most sub-10B coding models fail in realistic agentic environments due to a common trade-off: aggressive fine-tuning on synthetic coding instructions improves immediate benchmark pass rates but introduces brittle syntactic degradation and catastrophic repetition loops when shell commands or compiler checks fail.

PentaCoder addresses these limitations by uniting five specialized post-trained checkpoints of Qwen 3.5 9B via GeoDELLA (Geometric Drop-and-Rescale with Task-Covariance De-Biasing and Spectral Norm Anchoring). The model synthesizes the distinct mathematical distributions of each donor:

• Algorithmic Correctness & Architectural Decomposition from Qwopus3.5-Coder (Claude 3.5 Opus distillation trajectories).
• Multi-Turn SWE-bench Planning & Tool Protocol Integrity from MiMo-V2.6-Distill (Large-scale agentic execution traces).
• Terminal Execution Discipline & Error-Recovery Heuristics from Ornith-1.5 (Reinforcement learning for anti-looping).
• Low-Level Systems Implementation & Runtime Robustness from OxCoder (Deep API, CLI, and operational coding specialization).
• Abstract Structural Reasoning & Syntax Grounding from Qwen3.8-Distill (Distilled frontier chain-of-thought representations).

The result is a lean, blisteringly fast 9B pure-text causal engine with a native 256k context window that runs comfortably on consumer GPUs.

---

Contents

• Architectural Specifications
• Composition & Donor Weighting
• Merge Methodology & Mathematical Formulation
• Layer-Stratified Component Policies
• Agentic Chat Template & Operational Directives
• Recommended Generation Parameters
• How to Use
• Citation & References

---

Architectural Specifications

Parameter	Specification
Total Parameters	8.8B (Pure Text Backbone)
Architecture Type	Hybrid Recurrent-Attention Causal LM (qwen3_5_text)
Hidden Dimension (dmodel)	4096
Intermediate Dimension (dmlp)	12288 (SwiGLU)
Decoder Layers	32
Attention Layout	8 Blocks × (3 Gated DeltaNet Linear Layers : 1 Gated Softmax Layer)
Full Attention Layers	Layers 3, 7, 11, 15, 19, 23, 27, 31
Linear Attention Configuration	16 Key Heads / 32 Value Heads (dk = dv = 128)
Full Attention Configuration	16 Query Heads / 4 Key-Value Heads (GQA, dh = 256)
Rotary Position Embedding (RoPE)	1D Partial RoPE (θ = 10⁷, Factor = 0.25 → 64 dimensions)
Context Window Length	262,144 tokens (256k)
Native Precision	bfloat16
Vocabulary Size	248,320 (Padded for Fill-In-The-Middle and Tool Tokens)

---

Composition & Donor Weighting

The foundational weights of Qwen/Qwen3.5-9B serve as the shared topological base (W₀). Five specialized donor checkpoints provide non-overlapping task vectors mapped across normalized layer depth u ∈ [0, 1]:

Model	Primary Focus	Depth Target
Qwen/Qwen3.5-9B	Base pre-trained manifold and state-space anchors	Global (W₀)
empero-ai/Qwen3.8-9B-Distill	Abstract token synthesis, reasoning structure, syntax	Lower & Mid Decoders
Jackrong/Qwopus3.5-9B-Coder	Typing discipline, algorithm design, functional purity	Mid Decoders (Bell Curve)
ornith-ai/Ornith-1.5-9B	Circuit-breaker recovery, environment feedback integration	Upper-Mid Decoders
OrionLLM/OxCoder-9B	Systems engineering, runtime bug localization, CLI tooling	Deep Layers (Ascending Ramp)
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B	SWE-bench multi-step planning, long-context tool invocation	Broad Central Decoders

---

Merge Methodology & Mathematical Formulation

The merge was executed using the GeoDELLA-Coder pipeline, which solves weight interference through dynamic task-correlation de-biasing, row-wise sign consensus, sub-tensor GQA disentanglement, and power-iteration spectral norm stabilization.

1. Task Vector Formulation

For each donor checkpoint k ∈ {1, ..., 5}, the parameter update delta τ_k is computed relative to the base anchor W₀:

$${\tau }_k=D_k-W_0,k\in \{\text{Qwen3.8},\text{Qwopus},\text{Ornith},\text{OxCoder},\text{MiMo}\}\backslash tau\_k\; =\; D\_k\; -\; W\_0,\; \backslash quad\; k\; \backslash in\; \backslash \{\backslash text\{Qwen3.8\},\; \backslash text\{Qwopus\},\; \backslash text\{Ornith\},\; \backslash text\{OxCoder\},\; \backslash text\{MiMo\}\backslash \}$$

2. Hybrid-Aware Continuous Depth Modulation

Task vector mixing coefficients are continuously modulated over normalized depth u = l / (L - 1), where l ∈ [0, 31] and L = 32:

$$u_{\text{qwen38}}(u)=0.15+0.35{\cos }^2\left(\frac{\pi }{2}u\right)+0.15{\sin }^2(\pi u)u\_\{\backslash text\{qwen38\}\}(u)\; =\; 0.15\; +\; 0.35\; \backslash cos^2\backslash left(\backslash frac\{\backslash pi\}\{2\}\; u\backslash right)\; +\; 0.15\; \backslash sin^2(\backslash pi\; u)$$

$$u_{\text{qwopus}}(u)=0.05+0.35{\sin }^2(\pi u)u\_\{\backslash text\{qwopus\}\}(u)\; =\; 0.05\; +\; 0.35\; \backslash sin^2(\backslash pi\; u)$$

$$u_{\text{ornith}}(u)=0.05+0.35{\sin }^2\left(\frac{\pi }{2}u\right)u\_\{\backslash text\{ornith\}\}(u)\; =\; 0.05\; +\; 0.35\; \backslash sin^2\backslash left(\backslash frac\{\backslash pi\}\{2\}\; u\backslash right)$$

$$u_{\text{oxcoder}}(u)=0.05+0.25{\sin }^2\left(\frac{\pi }{2}u\right)u\_\{\backslash text\{oxcoder\}\}(u)\; =\; 0.05\; +\; 0.25\; \backslash sin^2\backslash left(\backslash frac\{\backslash pi\}\{2\}\; u\backslash right)$$

$$u_{\text{mimo}}(u)=0.05+0.15\sin (\pi u)u\_\{\backslash text\{mimo\}\}(u)\; =\; 0.05\; +\; 0.15\; \backslash sin(\backslash pi\; u)$$

The initial coefficients are normalized to form a partition of unity across all layers:

$$w_k(l)=\frac{u_k(u)}{\sum _{j=1}^5{u}_j(u)},\sum _{k=1}^5{w}_k(l)=1.0w\_k(l)\; =\; \backslash frac\{u\_k(u)\}\{\backslash sum\_\{j=1\}^5\; u\_j(u)\},\; \backslash quad\; \backslash sum\_\{k=1\}^5\; w\_k(l)\; =\; 1.0$$

3. Dynamic Gram-Matrix Task De-Biasing

Fine-tuned models frequently share underlying distillation datasets, leading to collinearity that can drown out specialized task vectors. To correct for this over-representation, the empirical Gram correlation matrix G ∈ ℝ^{K × K} is evaluated per tensor:

$$G_{ij}=\frac{\mathrm{\mid }⟨\text{vec}({\tau }_i),\text{vec}({\tau }_j)⟩\mathrm{\mid }}{\mathrm{\parallel }{\tau }_i{\mathrm{\parallel }}_2\mathrm{\parallel }{\tau }_j{\mathrm{\parallel }}_2}G\_\{ij\}\; =\; \backslash frac\{|\backslash langle\; \backslash text\{vec\}(\backslash tau\_i),\; \backslash text\{vec\}(\backslash tau\_j)\; \backslash rangle|\}\{\backslash |\backslash tau\_i\backslash |\_2\; \backslash |\backslash tau\_j\backslash |\_2\}$$

A uniqueness coefficient U_k is derived from the inverse column sum of task vector correlations:

$$U_k=\frac{1}{\sum _{j=1}^K{G}_{kj}}U\_k\; =\; \backslash frac\{1\}\{\backslash sum\_\{j=1\}^K\; G\_\{kj\}\}$$

The dynamic weights are then re-balanced and normalized:

$${\tilde{w}}_k=\frac{w_k{U}_k}{\sum _{j=1}^K{w}_j{U}_j}\backslash tilde\{w\}\_k\; =\; \backslash frac\{w\_k\; U\_k\}\{\backslash sum\_\{j=1\}^K\; w\_j\; U\_j\}$$

This prevents dataset overlap from suppressing distinct algorithmic representations.

4. Asymmetric GQA Disentangled QKV Slicing

Qwen 3.5 9B features asymmetric head ratios in both its linear attention and full attention layers. Standard fused tensor merging causes cross-head pollution by treating routing projections identically to memory projections.

Fused projection tensors are sliced into their functional sub-matrices prior to merging:

• Gated DeltaNet Layers (8192 × d_model): Sliced into Query (2048), Key (2048), and Value (4096).
• Gated Attention Layers (6144 × d_model): Sliced into Query (4096), Key (1024), and Value (1024).

Query and Key slices are processed with a conservative retention density (ρ = 0.95) to preserve sharp context routing. Value matrices are processed with adaptive MLP density (ρ = 0.70) to maximize conceptual synthesis. The components are then re-concatenated along the head dimension.

5. Neuron-Coherent Row Gating

To eliminate destructive interference in Feed-Forward Networks (MLPs), task vectors are gated at the single-neuron (row) level:

$$\bar{\tau }=\frac{1}{K}\sum _{k=1}^K{\tau }_k\backslash bar\{\backslash tau\}\; =\; \backslash frac\{1\}\{K\}\; \backslash sum\_\{k=1\}^K\; \backslash tau\_k$$

For each row r of donor delta τ_k, the directional alignment with the consensus mean is evaluated:

$$\cos {\theta }_{k,r}=\frac{⟨{\tau }_{k,r},{\bar{\tau }}_r⟩}{\mathrm{\parallel }{\tau }_{k,r}{\mathrm{\parallel }}_2\mathrm{\parallel }{\bar{\tau }}_r{\mathrm{\parallel }}_2+ϵ}\backslash cos\; \backslash theta\_\{k,\; r\}\; =\; \backslash frac\{\backslash langle\; \backslash tau\_\{k,\; r\},\; \backslash bar\{\backslash tau\}\_r\; \backslash rangle\}\{\backslash |\backslash tau\_\{k,\; r\}\backslash |\_2\; \backslash |\backslash bar\{\backslash tau\}\_r\backslash |\_2\; +\; \backslash epsilon\}$$

Rows exhibiting severe directional opposition (cos θ_{k, r} < -0.10) are masked out:

$${\widehat{\tau }}_{k,r}={\tau }_{k,r}\cdot \mathbb{I}(\cos {\theta }_{k,r}\ge -0.10)\backslash hat\{\backslash tau\}\_\{k,\; r\}\; =\; \backslash tau\_\{k,\; r\}\; \backslash cdot\; \backslash mathbb\{I\}\backslash left(\backslash cos\; \backslash theta\_\{k,\; r\}\; \backslash ge\; -0.10\backslash right)$$

This eliminates opposing gradient vectors that produce incoherent syntax generation.

6. Heavy-Tailed DELLA Adaptive Rescaling

Surviving parameters undergo non-linear magnitude-based sampling. Using parameter rank indices R_{k, ij} ∈ [0, 1] sorted by absolute magnitude, a Pareto-style retention probability p_{k, ij} is established:

$$p_{k,ij}=p_{\min }+(p_{\max }-p_{\min })\cdot \sqrt{R_{k,ij}}p\_\{k,\; ij\}\; =\; p\_\{\backslash min\}\; +\; (p\_\{\backslash max\}\; -\; p\_\{\backslash min\})\; \backslash cdot\; \backslash sqrt\{R\_\{k,\; ij\}\}$$

Parameters are sampled via a Bernoulli trial and rescaled by their inverse survival probability:

$$M_{k,ij}\sim \text{Bernoulli}(p_{k,ij})M\_\{k,\; ij\}\; \backslash sim\; \backslash text\{Bernoulli\}(p\_\{k,\; ij\})$$

$${\tilde{\tau }}_{k,ij}=\frac{{\widehat{\tau }}_{k,ij}\odot M_{k,ij}}{p_{k,ij}}\backslash tilde\{\backslash tau\}\_\{k,\; ij\}\; =\; \backslash frac\{\backslash hat\{\backslash tau\}\_\{k,\; ij\}\; \backslash odot\; M\_\{k,\; ij\}\}\{p\_\{k,\; ij\}\}$$

7. Coordinate-Wise Sign Consensus & Model Stock Scaling

Directional consensus is determined via weighted sign agreement:

$$\mathrm{\Gamma }=\mathrm{sgn}\left(\sum _{k=1}^K{\tilde{w}}_k{\tilde{\tau }}_k\right)\backslash Gamma\; =\; \backslash operatorname\{sgn\}\backslash left(\backslash sum\_\{k=1\}^K\; \backslash tilde\{w\}\_k\; \backslash tilde\{\backslash tau\}\_k\backslash right)$$

$$A_k=\mathbb{I}(\mathrm{sgn}({\tilde{\tau }}_k)=\mathrm{\Gamma })\odot \mathbb{I}({\tilde{\tau }}_k\mathrm{\ne }0)A\_k\; =\; \backslash mathbb\{I\}\backslash left(\backslash operatorname\{sgn\}(\backslash tilde\{\backslash tau\}\_k)\; =\; \backslash Gamma\backslash right)\; \backslash odot\; \backslash mathbb\{I\}\backslash left(\backslash tilde\{\backslash tau\}\_k\; \backslash neq\; 0\backslash right)$$

$${\mathrm{\Delta }}_{\text{consensus}}=\frac{\sum _{k=1}^K{\tilde{\tau }}_k\odot A_k}{\sum _{k=1}^K{A}_k+ϵ}\backslash Delta\_\{\backslash text\{consensus\}\}\; =\; \backslash frac\{\backslash sum\_\{k=1\}^K\; \backslash tilde\{\backslash tau\}\_k\; \backslash odot\; A\_k\}\{\backslash sum\_\{k=1\}^K\; A\_k\; +\; \backslash epsilon\}$$

The aggregated delta is projected onto the non-linear manifold using the Model Stock analytic scaling factor t^*:

$$t^{\ast }=\frac{K\bar{\rho }}{1+(K-1)\bar{\rho }}t^*\; =\; \backslash frac\{K\; \backslash bar\{\backslash rho\}\}\{1\; +\; (K\; -\; 1)\backslash bar\{\backslash rho\}\}$$

$${\mathrm{\Delta }}_{\text{final}}=t^{\ast }\cdot {\mathrm{\Delta }}_{\text{consensus}}\backslash Delta\_\{\backslash text\{final\}\}\; =\; t^*\; \backslash cdot\; \backslash Delta\_\{\backslash text\{consensus\}\}$$

8. Spectral Norm & Attention Entropy Anchoring

Attention projection matrices (W_attn) are vulnerable to spectral explosion during merges, which contracts attention entropy and leads to repetitive generation loops.

The dominant singular value σ(W) is calculated via a three-iteration deterministic power iteration:

$$v^{(t+1)}=\frac{W^T{u}^{(t)}}{\mathrm{\parallel }{W}^T{u}^{(t)}{\mathrm{\parallel }}_2},u^{(t+1)}=\frac{W{v}^{(t+1)}}{\mathrm{\parallel }W{v}^{(t+1)}{\mathrm{\parallel }}_2}v^\{(t+1)\}\; =\; \backslash frac\{W^T\; u^\{(t)\}\}\{\backslash |W^T\; u^\{(t)\}\backslash |\_2\},\; \backslash quad\; u^\{(t+1)\}\; =\; \backslash frac\{W\; v^\{(t+1)\}\}\{\backslash |W\; v^\{(t+1)\}\backslash |\_2\}$$

$$\sigma (W)\approx {u^{(3)}}^{T}W{v}^{(3)}\backslash sigma(W)\; \backslash approx\; \{u^\{(3)\}\}^T\; W\; v^\{(3)\}$$

If the merged spectral radius grows more than 5% relative to the base model, it is scaled down:

---

Layer-Stratified Component Policies

Parameter Class	Target Identifiers	Applied Policy	Density (ρ)	Mathematical Constraints
Embeddings & LM Head	embed_tokens, lm_head	Low-Memory Streaming Blend	1.0	Convex iterative accumulation; vocab dimension aligned to 248,320.
Linear State-Space Projections	linear_attn.in_proj_qkv	Asymmetric GQA DELLA	0.95 (QK) / 0.70 (V)	Sub-tensor slicing; separate routing and associative memory passes.
Self-Attention Projections	self_attn.qkv_proj, o_proj	Asymmetric GQA + Spectral Anchor	0.95 (QK) / 0.70 (V)	Power-iteration clipping prevents σ > 1.05 σ₀.
Feed-Forward Blocks	mlp.gate_proj, up_proj, down_proj	Neuron-Gated HG-DELLA	0.50 – 0.70	Row-wise cosine filtering (cos θ ≥ -0.10); sign consensus.
Recurrent Gates & Normalization	A_log, norm, conv1d	Convex Parameter Blend	1.0	Preservation of Alog ≤ 0 to guarantee BIBO stability.

---

Agentic Chat Template & Operational Directives

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

---

Recommended Generation Parameters

For deterministic software engineering and complex reasoning benchmarks:

Parameter	Recommended Value	Description
Temperature	0.6	Balances strict syntactic validity with algorithmic path exploration.
Top-P	0.95	Eliminates low-probability token tails while preserving alternative logic paths.
Top-K	20	Restricts token candidate pools to prevent architectural syntax drift.
Min-P	0.0 (Off)	Disabled in favor of explicit Top-K / Top-P governance.
Repetition Penalty	1.0 (Off)	Disabled to prevent syntax degradation in repetitive code patterns (indentation, braces).
Presence Penalty	0.0	Prevents naming mutations across long-context symbol resolution.

---

How to Use

Serving via vLLM

vllm serve pragmaticcs/PentaCoder-9B \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-reasoning \
  --reasoning-parser qwen3

Inference via Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/PentaCoder-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are an expert systems engineer. Reason step by step and output clean, robust implementations.",
    },
    {
        "role": "user",
        "content": "Write an asynchronous connection pool in Python for TCP sockets with active health-checking, backpressure control, and graceful shutdown handling.",
    },
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=4096,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
    do_sample=True,
)

response = tokenizer.decode(outputs[0][inputs.shape[-1] :], skip_special_tokens=True)
print(response)

---

Citation & References

• Qwen/Qwen3.5-9B
• XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
• ornith-ai/Ornith-1.5-9B
• Jackrong/Qwopus3.5-9B-Coder
• OrionLLM/OxCoder-9B
• empero-ai/Qwen3.8-9B-Distill
• Improved Chat Template for Qwen 3.x

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={36},
  pages={7093--7115},
  year={2023}
}

@article{deep2024della,
  title={DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling},
  author={Deep, Pala Tej and Bhardwaj, Rishabh and Poria, Soujanya},
  journal={arXiv preprint arXiv:2406.11617},
  year={2024}
}

@article{jang2024modelstock,
  title={Model Stock: All We Need Is just a Few Fine-Tuned Models},
  author={Jang, Dong-Hwan and Yoon, Sang-Doo and Song, Gyeong-Moon},
  journal={arXiv preprint arXiv:2403.19522},
  year={2024}
}