Qwimi-3.6-27B-Coder-MTP-BF16

A coding-focused SFT of Qwen 3.6 27B, merged to full-precision BF16. Trained on a mixed corpus of coding, tool-calling, and agentic (SWE-agent) data in a single supervised fine-tuning run, with the coding domain deliberately kept dominant (~82.5% of training tokens) to avoid the smaller tool/agentic buckets degrading coding capability.

Status: public repository. This is the merged, ready-to-use model. For a smaller adapter-only download see Qwimi-3.6-27B-LoRA; for quantized GGUF files (llama.cpp / Ollama / LM Studio) see Qwimi-3.6-27B-GGUF.

---

Contents

1. Model summary
2. Intended use
3. Training configuration
4. Dataset composition
5. Training results
6. Chat template & tool-calling format
7. How to use
8. Verification & reproducibility
9. Known limitations
10. Benchmarks

---

1. Model summary

Base model	unsloth/Qwen3.6-27B (finetune of Qwen/Qwen3.6-27B)
Architecture	Dense, 27.8B params. Hybrid: standard attention layers interleaved with GatedDeltaNet linear-attention layers. model_type: qwen3_5, class Qwen3_5ForConditionalGeneration
Modality	Multimodal (image-text-to-text) checkpoint. This SFT is text-only — the vision tower was frozen and untouched (see §9)
Native context	262,144 tokens (base). This SFT trained and validates up to 16,384 tokens
Fine-tuning method	4-bit QLoRA (r=64), merged to BF16
Trainable parameters	466,911,232 / 27,823,639,792 (1.68%) — language layers only (attention + MLP), vision layers frozen
License	Apache 2.0 (inherited from base)

2. Intended use

Coding assistance (generation, debugging, refactoring, explanation) with visible chain-of-thought reasoning, plus native function/tool calling and basic agentic (SWE-agent style: multi-turn tool use over a repository) behavior.

Not validated for: image/vision input (untouched by this SFT — see §9), non-English use, safety-critical or high-stakes deployment, or any task outside code/tool/agentic reasoning.

3. Training configuration

Setting	Value
Precision (training)	4-bit QLoRA (NF4)
Max sequence length	16,384 tokens
LoRA rank / alpha	64 / 64
LoRA dropout	0.0
rsLoRA	off
LoRA scope	language layers only (attention + MLP); vision tower frozen
Optimizer	paged_adamw_8bit
Learning rate	2e-5, cosine schedule, warmup ratio 0.03
Weight decay	0.01
Max grad norm	1.0
Per-device batch size	2
Gradient accumulation	8
Effective batch size	16 (2 × 8 × 1 GPU)
Epochs	1
Optimizer steps	1,398
Training tokens	72,529,798
Masking	assistant-turns-only (train_on_responses_only); instruction marker <|im_start|>user\n, response marker <|im_start|>assistant\n
Gradient checkpointing	Unsloth offloaded mode, verified active (measured activation delta 3.1→5.5→9.2 GB across 2k→8k→16.4k token samples)
Hardware	1× A100-SXM4-80GB (Google Colab Pro)
Software	Unsloth 2026.6.9, Transformers 5.5.0, Torch 2.10.0+cu128, CUDA 12.8, Triton 3.6.0, flash-linear-attention + causal-conv1d (required for the GatedDeltaNet fast path)
Checkpoint cadence	every 150 steps, rolling window of 4, persisted to Google Drive

4. Dataset composition

Single mixed SFT run — coding, tool-calling, and agentic examples trained together (not sequential stages), so the model learns all three without later stages overwriting earlier ones. Coding was deliberately kept dominant by token share, not just row count.

Coding (5 sources, deduplicated, content- or domain-filtered for code where the source had no reliable label; one low-quality synthetic source capped hard):

Source	Raw rows contributed	Filter
trjxter/Kimi-K2.7-CodingTraces-9000x	9,014	none — used in full
trjxter/Kimi-K2.6-Technical-Reasoning-AddOn-3300x	1,645	domain ∈ {coding, debugging}
trjxter/Kimi-K2.6-Reasoning-3300x-WandB	798	content-filtered for code
Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned (General-Distillation config)	6,500	content-filtered for code, capped at target
Jackrong/Claude-opus-4.7-TraceInversion-5000x	577	content-filtered; hard cap 600 — synthetically reverse-engineered reasoning traces, lowest-quality source, deliberately minimized

Tool-calling: minpeter/xlam-function-calling-60k-parsed — 6,000 rows (APIGen-generated, verifiable function calls, native tools schema). No chain-of-thought; direct tool calls.

Agentic (SWE-agent trajectories; the dominant length constraint in this run — most raw trajectories exceed 16,384 tokens and were dropped whole rather than truncated):

Source	Contribution	Note
SWE-Factory/DeepSWE-Agent-Kimi-K2-Trajectories-2.8K	~200 rows survive the 16k length filter	p50 trajectory length ≈ 23,000 tokens — most don't fit
nebius/SWE-agent-trajectories (top-up)	788 unique successful issues	target == True only; 1 trajectory per GitHub issue (deduplicated at the issue level so the later train/eval split can't leak the same issue into both)

Final composition after dedup, 90/10 per-domain split, and whole-fit-or-drop length filtering at 16,384 tokens:

Domain	Train rows	Eval rows	Train token share	Dropped for length
Coding	16,083	1,778	82.5%	598 (3.6% — accepted as pathological-outlier policy)
Tool	5,360	595	4.8%	0
Agentic	917	102	12.7%	2,321 (multi-turn trajectories that don't fit whole at 16k)
Total	22,360†	2,475	72.5M tokens	—

† One row was dropped by the trainer itself (all labels masked after truncation-safety filtering) — 22,359 rows actually trained on.

Reasoning-token fraction (coding + agentic, which carry <think> traces) was 95.2% of all training tokens, comfortably above the 75% floor generally recommended to preserve reasoning ability when mixing in non-reasoning (tool-calling) data.

5. Training results

Eval loss reported per domain via three held-out splits, evaluated every 150 steps (subsampled during training for speed: 400/1,778 coding rows, 200/595 tool rows, all 102/102 agentic rows — full-split numbers are pending a re-run and will be added if produced).

Step	Train loss	Eval — Coding	Eval — Tool	Eval — Agentic
150	—	0.5070	0.0246	0.3574
300	—	0.4911	0.0226	0.3257
450	—	0.4853	0.0217	0.3118
600	—	0.4823	0.0214	0.3056
750	0.5004	0.4794	0.0211	0.2981
900	0.4769	0.4781	0.0209	0.2946
1050	0.4931	0.4767	0.0208	0.2915
1200	—	0.4762	0.0208	0.2899
1350	—	0.4761	0.0208	0.2893
1398 (final)	PENDING	0.4761	0.0208	0.2892

All three eval splits decreased monotonically across the entire run with no split turning up — no overfitting signal at 1 epoch. Coding, the dominant and highest-value domain, continued improving through the final checkpoint rather than being degraded by the smaller tool/agentic data — the outcome the mixed-run design targeted. Note that eval-loss magnitudes are not directly comparable across domains (different sequence structure and supervised-token counts per split); track each split against itself over time.

Single-batch training loss (0.500, 0.477, 0.493 at steps 750/900/1050) is noisy at effective batch 16 and is shown only as a secondary signal — eval loss is the trend to read.

6. Chat template & tool-calling format

Standard ChatML markers, with <think> reasoning canonically formatted by the tokenizer's chat template (verified — the template re-renders think content even if you pass it inline):

<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
{reasoning}
</think>

{answer}<|im_end|>

Tool calls use Qwen 3.6's native XML-style format (confirmed empirically against the installed chat template — this is not the JSON-object tool_call format used by earlier Qwen versions):

<|im_start|>assistant
<think>

</think>

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call><|im_end|>

An empty <think></think> block is the canonical "direct response" form — tool calls in this model are trained to skip reasoning and act directly. Parallel tool calls render as sibling <tool_call> blocks. Pass your tool definitions via the standard tools= argument to apply_chat_template; the template injects a # Tools system block describing the exact call format automatically.

7. How to use

Recommended (tested path) — Unsloth:

from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained(
    "trjxter/Qwimi-3.6-27B-BF16",
    load_in_4bit=False,        # this repo is already full-precision BF16
    max_seq_length=16384,
)
tokenizer = processor.tokenizer
FastVisionModel.for_inference(model)

messages = [{"role": "user", "content": "Write a Python function to reverse a linked list."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, temperature=0.7, top_p=0.95)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Alternative — plain transformers (requires transformers>=5 for qwen3_5 support; confirm the exact model class name against your installed version's docs before relying on AutoModelForImageTextToText resolving correctly):

from transformers import AutoModelForImageTextToText, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("trjxter/Qwimi-3.6-27B-BF16")
model = AutoModelForImageTextToText.from_pretrained("trjxter/Qwimi-3.6-27B-BF16", dtype="bfloat16", device_map="auto")

This model was fine-tuned and validated on text-only input; no image inputs were used during training (see §9).

8. Verification & reproducibility

• Adapter integrity: final adapter adapter_model.safetensors = 1,867.8 MB, matching 466,911,232 trainable params × 4 bytes (fp32 save) exactly — confirms the saved file contains the complete, correctly-scoped trained adapter.
• Merge integrity: max |lora_B| (the check that the adapter actually trained before merging — an all-zero adapter would merge silently into an unchanged copy of the base) — PENDING, not yet recorded from the merge run. Treat the merge as provisionally verified pending this number.
• Tokenizer/config repair: the merged repo's tokenizer_config.json originally recorded a runtime-internal tokenizer_class value (TokenizersBackend, from transformers v5's tokenizer backend) that AutoTokenizer.from_pretrained cannot resolve. This was corrected in the repo to match the base model's tokenizer class — confirm your load succeeds before reporting issues; this repair should make plain AutoTokenizer loading work correctly.
• Checkpoint recovery: this run survived two Colab VM interruptions via checkpoints persisted to Drive every 150 steps; the final adapter reflects the complete 1,398-step run, not a recovered partial checkpoint.

9. Known limitations

• Vision tower untouched. The base checkpoint is multimodal; this SFT explicitly froze the vision layers (finetune_vision_layers=False) and trained on text-only data. No claims are made about image understanding — it is whatever the base model shipped with, unvalidated here.
• Agentic data is thin and short. After length-filtering, only 917 agentic training rows survived (12.7% of tokens), and DeepSWE trajectories longer than ~16k tokens were entirely excluded rather than truncated — this model has seen comparatively little of the longest, most complex agentic trajectories.
• One low-quality source, deliberately capped. Claude-opus-4.7-TraceInversion-5000x contains reasoning traces reverse-engineered by a small model from compressed summaries, not genuine teacher traces. Capped at 577 rows (3.5% of the coding split) specifically to limit this risk.
• 3.6% of coding examples were dropped for length (598 rows exceeding 16,384 tokens) — accepted as pathological outliers rather than truncated, since truncating a reasoning trace teaches the model to stop mid-thought.
• Context validated to 16,384 tokens despite the base model's much longer native context; behavior beyond that length is unvalidated by this training run.

10. Benchmarks

Coming soon.