Hugging Face's logo

Spaces:
khazarai
/
Multi-Domain-Reasoning-Distillation
Running

App Files Community
Multi-Domain-Reasoning-Distillation / index.html
Rustamshry's picture
Rustamshry
Update index.html
5def279 verified
Raw
History Blame Contribute Delete
59.4 kB
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Multi-Domain Reasoning Distillation and SLERP Merging: A Study on Qwen3-4B Models</title>
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&family=Merriweather:ital,wght@0,300;0,400;0,700;0,900;1,400&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
 <style>
 :root {
 --primary: #2563eb;
 --primary-dark: #1d4ed8;
 --accent: #7c3aed;
 --text-primary: #1e293b;
 --text-secondary: #475569;
 --text-muted: #64748b;
 --bg-primary: #ffffff;
 --bg-secondary: #f8fafc;
 --bg-tertiary: #f1f5f9;
 --border: #e2e8f0;
 --border-dark: #cbd5e1;
 --code-bg: #1e293b;
 --code-text: #e2e8f0;
 --success: #10b981;
 --warning: #f59e0b;
 --shadow-sm: 0 1px 2px rgba(0,0,0,0.05);
 --shadow-md: 0 4px 6px -1px rgba(0,0,0,0.1), 0 2px 4px -1px rgba(0,0,0,0.06);
 --shadow-lg: 0 10px 15px -3px rgba(0,0,0,0.1), 0 4px 6px -2px rgba(0,0,0,0.05);
 }
 * {
 margin: 0;
 padding: 0;
 box-sizing: border-box;
 }
 body {
 font-family: 'Merriweather', Georgia, serif;
 background: var(--bg-secondary);
 color: var(--text-primary);
 line-height: 1.8;
 font-size: 15px;
 }
 .paper-container {
 max-width: 900px;
 margin: 0 auto;
 background: var(--bg-primary);
 box-shadow: var(--shadow-lg);
 min-height: 100vh;
 }
 .header {
 background: linear-gradient(135deg, #1e293b 0%, #334155 100%);
 color: white;
 padding: 48px 64px;
 position: relative;
 overflow: hidden;
 }
 .header::before {
 content: '';
 position: absolute;
 top: 0;
 left: 0;
 right: 0;
 bottom: 0;
 background: url("data:image/svg+xml,%3Csvg width='60' height='60' viewBox='0 0 60 60' xmlns='http://www.w3.org/2000/svg'%3E%3Cg fill='none' fill-rule='evenodd'%3E%3Cg fill='%23ffffff' fill-opacity='0.03'%3E%3Cpath d='M36 34v-4h-2v4h-4v2h4v4h2v-4h4v-2h-4zm0-30V0h-2v4h-4v2h4v4h2V6h4V4h-4zM6 34v-4H4v4H0v2h4v4h2v-4h4v-2H6zM6 4V0H4v4H0v2h4v4h2V6h4V4H6z'/%3E%3C/g%3E%3C/g%3E%3C/svg%3E");
 }
 .header-content {
 position: relative;
 z-index: 1;
 }
 .paper-type {
 font-family: 'Inter', sans-serif;
 font-size: 11px;
 font-weight: 600;
 text-transform: uppercase;
 letter-spacing: 3px;
 color: #93c5fd;
 margin-bottom: 16px;
 }
 .paper-title {
 font-family: 'Inter', sans-serif;
 font-size: 28px;
 font-weight: 700;
 line-height: 1.3;
 margin-bottom: 24px;
 letter-spacing: -0.5px;
 }
 .authors {
 font-family: 'Inter', sans-serif;
 font-size: 15px;
 color: #cbd5e1;
 margin-bottom: 8px;
 }
 .authors a {
 color: #60a5fa;
 text-decoration: none;
 }
 .authors a:hover {
 text-decoration: underline;
 }
 .affiliation {
 font-family: 'Inter', sans-serif;
 font-size: 13px;
 color: #94a3b8;
 margin-bottom: 20px;
 }
 .abstract-box {
 background: rgba(255,255,255,0.08);
 border: 1px solid rgba(255,255,255,0.12);
 border-radius: 8px;
 padding: 20px 24px;
 margin-top: 24px;
 }
 .abstract-label {
 font-family: 'Inter', sans-serif;
 font-size: 11px;
 font-weight: 600;
 text-transform: uppercase;
 letter-spacing: 2px;
 color: #93c5fd;
 margin-bottom: 10px;
 }
 .abstract-text {
 font-size: 13px;
 line-height: 1.7;
 color: #e2e8f0;
 font-family: 'Inter', sans-serif;
 }
 .content {
 padding: 48px 64px;
 }
 .section {
 margin-bottom: 40px;
 }
 .section-number {
 font-family: 'Inter', sans-serif;
 font-size: 11px;
 font-weight: 600;
 color: var(--primary);
 text-transform: uppercase;
 letter-spacing: 2px;
 margin-bottom: 4px;
 }
 .section-title {
 font-family: 'Inter', sans-serif;
 font-size: 22px;
 font-weight: 700;
 color: var(--text-primary);
 margin-bottom: 16px;
 padding-bottom: 8px;
 border-bottom: 2px solid var(--border);
 }
 .subsection-title {
 font-family: 'Inter', sans-serif;
 font-size: 17px;
 font-weight: 600;
 color: var(--text-primary);
 margin-top: 28px;
 margin-bottom: 12px;
 }
 p {
 margin-bottom: 16px;
 text-align: justify;
 hyphens: auto;
 }
 .highlight {
 background: linear-gradient(120deg, #dbeafe 0%, #ede9fe 100%);
 padding: 2px 6px;
 border-radius: 3px;
 font-weight: 500;
 }
 .model-name {
 font-family: 'JetBrains Mono', monospace;
 font-size: 0.9em;
 background: var(--bg-tertiary);
 padding: 2px 8px;
 border-radius: 4px;
 color: var(--accent);
 font-weight: 500;
 border: 1px solid var(--border);
 }
 .dataset-name {
 font-family: 'JetBrains Mono', monospace;
 font-size: 0.9em;
 background: #ecfdf5;
 padding: 2px 8px;
 border-radius: 4px;
 color: #059669;
 font-weight: 500;
 border: 1px solid #d1fae5;
 }
 .benchmark-name {
 font-family: 'JetBrains Mono', monospace;
 font-size: 0.9em;
 background: #fef3c7;
 padding: 2px 8px;
 border-radius: 4px;
 color: #d97706;
 font-weight: 500;
 border: 1px solid #fde68a;
 }
 table {
 width: 100%;
 border-collapse: collapse;
 margin: 20px 0;
 font-family: 'Inter', sans-serif;
 font-size: 13px;
 }
 th {
 background: var(--bg-tertiary);
 padding: 12px 16px;
 text-align: left;
 font-weight: 600;
 color: var(--text-secondary);
 border-bottom: 2px solid var(--border-dark);
 font-size: 12px;
 text-transform: uppercase;
 letter-spacing: 0.5px;
 }
 td {
 padding: 12px 16px;
 border-bottom: 1px solid var(--border);
 color: var(--text-primary);
 }
 tr:hover td {
 background: var(--bg-secondary);
 }
 .figure-container {
 margin: 32px 0;
 text-align: center;
 }
 .figure-container img {
 max-width: 100%;
 height: auto;
 border: 1px solid var(--border);
 border-radius: 8px;
 box-shadow: var(--shadow-md);
 }
 .figure-caption {
 font-family: 'Inter', sans-serif;
 font-size: 12px;
 color: var(--text-muted);
 margin-top: 12px;
 font-style: italic;
 }
 .figure-caption strong {
 color: var(--text-secondary);
 font-style: normal;
 }
 .code-block {
 background: var(--code-bg);
 color: var(--code-text);
 padding: 20px 24px;
 border-radius: 8px;
 margin: 20px 0;
 overflow-x: auto;
 font-family: 'JetBrains Mono', monospace;
 font-size: 13px;
 line-height: 1.6;
 position: relative;
 }
 .code-block::before {
 content: 'YAML';
 position: absolute;
 top: 8px;
 right: 12px;
 font-size: 10px;
 color: #64748b;
 text-transform: uppercase;
 letter-spacing: 1px;
 }
 .code-block code {
 font-family: 'JetBrains Mono', monospace;
 }
 .yaml-key { color: #7dd3fc; }
 .yaml-value { color: #a78bfa; }
 .yaml-string { color: #86efac; }
 .yaml-comment { color: #64748b; font-style: italic; }
 .comparison-grid {
 display: grid;
 grid-template-columns: 1fr 1fr;
 gap: 20px;
 margin: 24px 0;
 }
 .comparison-card {
 border: 1px solid var(--border);
 border-radius: 8px;
 padding: 20px;
 background: var(--bg-secondary);
 }
 .comparison-card h4 {
 font-family: 'Inter', sans-serif;
 font-size: 14px;
 font-weight: 600;
 margin-bottom: 12px;
 display: flex;
 align-items: center;
 gap: 8px;
 }
 .comparison-card ul {
 list-style: none;
 padding: 0;
 }
 .comparison-card li {
 font-size: 13px;
 padding: 6px 0;
 border-bottom: 1px solid var(--border);
 color: var(--text-secondary);
 font-family: 'Inter', sans-serif;
 line-height: 1.5;
 }
 .comparison-card li:last-child {
 border-bottom: none;
 }
 .comparison-card li::before {
 content: '→';
 margin-right: 8px;
 color: var(--primary);
 }
 .card-base {
 border-left: 3px solid #ef4444;
 }
 .card-distilled {
 border-left: 3px solid var(--success);
 }
 .badge {
 display: inline-block;
 padding: 3px 10px;
 border-radius: 12px;
 font-family: 'Inter', sans-serif;
 font-size: 11px;
 font-weight: 600;
 text-transform: uppercase;
 letter-spacing: 0.5px;
 }
 .badge-primary {
 background: #dbeafe;
 color: var(--primary-dark);
 }
 .badge-success {
 background: #d1fae5;
 color: #059669;
 }
 .badge-accent {
 background: #ede9fe;
 color: var(--accent);
 }
 .badge-warning {
 background: #fef3c7;
 color: #d97706;
 }
 .info-box {
 background: #eff6ff;
 border-left: 4px solid var(--primary);
 padding: 16px 20px;
 margin: 20px 0;
 border-radius: 0 8px 8px 0;
 }
 .info-box p {
 margin-bottom: 0;
 font-size: 14px;
 color: var(--text-secondary);
 font-family: 'Inter', sans-serif;
 }
 .warning-box {
 background: #fef3c7;
 border-left: 4px solid var(--warning);
 padding: 16px 20px;
 margin: 20px 0;
 border-radius: 0 8px 8px 0;
 }
 .warning-box p {
 margin-bottom: 0;
 font-size: 14px;
 color: #92400e;
 font-family: 'Inter', sans-serif;
 }
 .domain-tags {
 display: flex;
 flex-wrap: wrap;
 gap: 8px;
 margin: 16px 0;
 }
 .domain-tag {
 font-family: 'Inter', sans-serif;
 font-size: 12px;
 padding: 6px 14px;
 border-radius: 20px;
 background: var(--bg-tertiary);
 color: var(--text-secondary);
 border: 1px solid var(--border);
 transition: all 0.2s;
 }
 .domain-tag:hover {
 background: var(--primary);
 color: white;
 border-color: var(--primary);
 }
 .stats-grid {
 display: grid;
 grid-template-columns: repeat(3, 1fr);
 gap: 16px;
 margin: 20px 0;
 }
 .stat-card {
 background: var(--bg-secondary);
 border: 1px solid var(--border);
 border-radius: 8px;
 padding: 20px;
 text-align: center;
 }
 .stat-value {
 font-family: 'Inter', sans-serif;
 font-size: 28px;
 font-weight: 700;
 color: var(--primary);
 margin-bottom: 4px;
 }
 .stat-label {
 font-family: 'Inter', sans-serif;
 font-size: 12px;
 color: var(--text-muted);
 text-transform: uppercase;
 letter-spacing: 1px;
 }
 .footer {
 background: var(--bg-tertiary);
 padding: 32px 64px;
 text-align: center;
 border-top: 1px solid var(--border);
 }
 .footer p {
 font-family: 'Inter', sans-serif;
 font-size: 12px;
 color: var(--text-muted);
 text-align: center;
 margin-bottom: 4px;
 }
 .footer a {
 color: var(--primary);
 text-decoration: none;
 }
 .footer a:hover {
 text-decoration: underline;
 }
 .references {
 font-size: 13px;
 line-height: 1.8;
 }
 .references li {
 margin-bottom: 8px;
 color: var(--text-secondary);
 }
 ul, ol {
 padding-left: 24px;
 margin-bottom: 16px;
 }
 li {
 margin-bottom: 6px;
 }
 strong {
 font-weight: 700;
 }
 em {
 font-style: italic;
 }
 .horizontal-rule {
 border: none;
 height: 1px;
 background: var(--border);
 margin: 32px 0;
 }
 .gradient-text {
 background: linear-gradient(135deg, var(--primary), var(--accent));
 -webkit-background-clip: text;
 -webkit-text-fill-color: transparent;
 background-clip: text;
 }
 .mermaid-flow {
 display: flex;
 align-items: center;
 justify-content: center;
 gap: 8px;
 margin: 24px 0;
 flex-wrap: wrap;
 }
 .flow-node {
 font-family: 'JetBrains Mono', monospace;
 font-size: 12px;
 padding: 10px 16px;
 border-radius: 6px;
 border: 2px solid;
 text-align: center;
 max-width: 200px;
 }
 .flow-arrow {
 font-size: 20px;
 color: var(--text-muted);
 }
 .flow-node-teacher {
 background: #fef3c7;
 border-color: #f59e0b;
 color: #92400e;
 }
 .flow-node-dataset {
 background: #d1fae5;
 border-color: #10b981;
 color: #065f46;
 }
 .flow-node-model {
 background: #dbeafe;
 border-color: #3b82f6;
 color: #1e40af;
 }
 .flow-node-merged {
 background: #ede9fe;
 border-color: #8b5cf6;
 color: #5b21b6;
 }
 @media (max-width: 768px) {
 .header, .content, .footer {
 padding: 32px 24px;
 }
 .paper-title {
 font-size: 22px;
 }
 .comparison-grid {
 grid-template-columns: 1fr;
 }
 .stats-grid {
 grid-template-columns: 1fr;
 }
 .mermaid-flow {
 flex-direction: column;
 }
 .flow-arrow {
 transform: rotate(90deg);
 }
 }
 .toc {
 background: var(--bg-secondary);
 border: 1px solid var(--border);
 border-radius: 8px;
 padding: 24px 28px;
 margin: 24px 0;
 }
 .toc-title {
 font-family: 'Inter', sans-serif;
 font-size: 14px;
 font-weight: 600;
 color: var(--text-primary);
 margin-bottom: 16px;
 text-transform: uppercase;
 letter-spacing: 1px;
 }
 .toc-list {
 list-style: none;
 padding: 0;
 }
 .toc-list li {
 padding: 6px 0;
 border-bottom: 1px dashed var(--border);
 }
 .toc-list li:last-child {
 border-bottom: none;
 }
 .toc-list a {
 font-family: 'Inter', sans-serif;
 font-size: 13px;
 color: var(--text-secondary);
 text-decoration: none;
 display: flex;
 justify-content: space-between;
 align-items: center;
 }
 .toc-list a:hover {
 color: var(--primary);
 }
 .toc-list .toc-num {
 color: var(--primary);
 font-weight: 600;
 margin-right: 8px;
 min-width: 24px;
 }
 .toc-list .toc-page {
 color: var(--text-muted);
 font-size: 12px;
 }
 .pipeline-container {
 background: var(--bg-secondary);
 border: 1px solid var(--border);
 border-radius: 8px;
 padding: 24px;
 margin: 24px 0;
 }
 .pipeline-title {
 font-family: 'Inter', sans-serif;
 font-size: 13px;
 font-weight: 600;
 color: var(--text-secondary);
 text-transform: uppercase;
 letter-spacing: 1px;
 margin-bottom: 16px;
 text-align: center;
 }
 .pipeline-step {
 display: flex;
 align-items: flex-start;
 gap: 16px;
 margin-bottom: 16px;
 padding: 12px 16px;
 background: var(--bg-primary);
 border-radius: 6px;
 border: 1px solid var(--border);
 }
 .pipeline-step:last-child {
 margin-bottom: 0;
 }
 .step-num {
 font-family: 'Inter', sans-serif;
 font-size: 12px;
 font-weight: 700;
 color: white;
 background: var(--primary);
 width: 28px;
 height: 28px;
 border-radius: 50%;
 display: flex;
 align-items: center;
 justify-content: center;
 flex-shrink: 0;
 }
 .step-content {
 font-family: 'Inter', sans-serif;
 font-size: 13px;
 color: var(--text-secondary);
 line-height: 1.5;
 }
 .step-content strong {
 color: var(--text-primary);
 }
 </style>
</head>
<body>
 <div class="paper-container">
 <!-- Header -->
 <header class="header">
 <div class="header-content">
 <div class="paper-type">Paper</div>
 <h1 class="paper-title">Multi-Domain Reasoning Distillation and SLERP Merging: Enhancing Qwen3-4B with Teacher-Guided Fine-Tuning</h1>
 <div class="authors">
 <a href="https://huggingface.co/khazarai" target="_blank">khazarai</a>
 </div>
 <div class="affiliation">Independent Research · Hugging Face · 2026</div>
 <div class="abstract-box">
 <div class="abstract-label">Abstract</div>
 <p class="abstract-text">
 This paper presents a systematic approach to enhancing the reasoning capabilities of compact language models through knowledge distillation and strategic model merging. We create two specialized distillation datasets using Qwen3.6-plus and Kimi-2.5-thinking as teacher models, covering diverse domains including mathematics, coding, scientific reasoning, and complex scenario analysis. Two QLoRA-distilled variants of Qwen3-4B-Thinking are fine-tuned on these datasets, demonstrating significant improvements in reasoning structure and output quality. We then introduce a novel SLERP merge strategy with layer-wise gradient attention and vocabulary pinning, producing a merged model that achieves a synergistic "1+1=3" effect in logical inference tasks. Evaluation on our comprehensive CMDR-Bench benchmark across 10 cognitive domains validates the effectiveness of our approach, with the merged model matching or exceeding the performance of both individual distilled models and the base model in most reasoning categories.
 </p>
 </div>
 </div>
 </header>
<!-- Content -->
 <main class="content">
 <!-- Table of Contents -->
 <div class="toc">
 <div class="toc-title">Table of Contents</div>
 <ol class="toc-list">
 <li><a href="#introduction"><span class="toc-num">1</span> Introduction <span class="toc-page"></span></a></li>
 <li><a href="#datasets"><span class="toc-num">2</span> Dataset Construction <span class="toc-page"></span></a></li>
 <li><a href="#methodology"><span class="toc-num">3</span> Methodology <span class="toc-page"></span></a></li>
 <li><a href="#model1"><span class="toc-num">4</span> Qwen3.6-plus Distilled Model <span class="toc-page"></span></a></li>
 <li><a href="#model2"><span class="toc-num">5</span> Kimi2.5 Distilled Model <span class="toc-page"></span></a></li>
 <li><a href="#merge"><span class="toc-num">6</span> SLERP Merge Strategy <span class="toc-page"></span></a></li>
 <li><a href="#benchmark"><span class="toc-num">7</span> Benchmark Design and Evaluation <span class="toc-page"></span></a></li>
 <li><a href="#results"><span class="toc-num">8</span> Results and Analysis <span class="toc-page"></span></a></li>
 <li><a href="#conclusion"><span class="toc-num">9</span> Conclusion <span class="toc-page"></span></a></li>
 <li><a href="#references"><span class="toc-num">References</span> References <span class="toc-page"></span></a></li>
 </ol>
 </div>
<!-- Section 1: Introduction -->
 <section class="section" id="introduction">
 <div class="section-number">Section 1</div>
 <h2 class="section-title">Introduction</h2>
 <p>
 The rapid advancement of large language models has revealed a persistent gap between small, efficient models (0.6B–4B parameter range) and their significantly larger counterparts in terms of complex reasoning capabilities. While compact models offer advantages in deployment efficiency, latency, and resource consumption, they typically exhibit verbose, uncertain reasoning patterns characterized by stream-of-consciousness exploration, self-correction loops, and high noise-to-signal ratios in their outputs.
 </p>
 <p>
 Knowledge distillation—the process of transferring capabilities from a larger "teacher" model to a smaller "student" model—has emerged as a promising approach to bridge this gap. However, most distillation efforts focus on general capability transfer rather than specifically targeting reasoning structure and quality. Furthermore, when multiple distilled models are merged using standard techniques, catastrophic forgetting and degradation of specialized capabilities often occur.
 </p>
 <p>
 In this work, we present a comprehensive pipeline that addresses these challenges through:
 </p>
 <ol>
 <li><strong>Domain-specific distillation datasets</strong> generated by two distinct teacher models (Qwen3.6-plus and Kimi-2.5-thinking), each emphasizing different reasoning strengths.</li>
 <li><strong>QLoRA fine-tuning</strong> of Qwen3-4B-Thinking on each dataset, producing two specialized reasoning-distilled models.</li>
 <li><strong>A novel SLERP merge strategy</strong> with layer-wise gradient attention and vocabulary pinning that preserves RAG capabilities while combining complementary reasoning strengths.</li>
 <li><strong>Comprehensive evaluation</strong> on CMDR-Bench, a 100-test-case benchmark spanning 10 cognitive domains with graduated difficulty levels.</li>
 </ol>
 <p>
 Our results demonstrate that the merged model achieves superior performance in logical reasoning, mathematical problem-solving, and code analysis compared to both individual distilled models and the base model, while maintaining acceptable performance in creative writing tasks.
 </p>
 </section>
<hr class="horizontal-rule">
<!-- Section 2: Dataset Construction -->
 <section class="section" id="datasets">
 <div class="section-number">Section 2</div>
 <h2 class="section-title">Dataset Construction</h2>
 <p>
 We constructed two complementary distillation datasets, each leveraging a different teacher model to capture diverse reasoning patterns and domain expertise.
 </p>
<div class="subsection-title">2.1 Kimi-2.5 High Reasoning Dataset</div>
 <p>
 The first dataset, <span class="dataset-name">khazarai/kimi-2.5-high-reasoning-250x</span>, was generated using <strong>Kimi-2.5-thinking</strong> as a teacher model. This dataset contains detailed reasoning traces and final answers for complex questions spanning multiple technical, scientific, historical, and strategic domains.
 </p>
 <div class="stats-grid">
 <div class="stat-card">
 <div class="stat-value">250×</div>
 <div class="stat-label">Scale Factor</div>
 </div>
 <div class="stat-card">
 <div class="stat-value">1.11M</div>
 <div class="stat-label">Total Tokens</div>
 </div>
 <div class="stat-card">
 <div class="stat-value">8,000</div>
 <div class="stat-label">Max Seq Length</div>
 </div>
 </div>
 <p>
 The Kimi teacher model was selected for its demonstrated strength in deep analytical reasoning, causal inference, and structured problem decomposition. The generated reasoning traces emphasize systematic analysis, hypothesis evaluation, and evidence-based conclusion derivation.
 </p>
<div class="subsection-title">2.2 Qwen3.6-plus High Reasoning Dataset</div>
 <p>
 The second dataset, <span class="dataset-name">khazarai/qwen3.6-plus-high-reasoning-500x</span>, was prepared using <strong>Qwen3.6-plus</strong> as the teacher model, covering topics in coding, mathematics, finance, medicine, and economics.
 </p>
 <div class="stats-grid">
 <div class="stat-card">
 <div class="stat-value">500×</div>
 <div class="stat-label">Scale Factor</div>
 </div>
 <div class="stat-card">
 <div class="stat-value">1.74M</div>
 <div class="stat-label">Total Tokens</div>
 </div>
 <div class="stat-card">
 <div class="stat-value">6,500</div>
 <div class="stat-label">Max Seq Length</div>
 </div>
 </div>
 <p>
 This dataset was designed to emphasize mathematical precision, algorithmic thinking, and structured solution formulation. The Qwen3.6-plus teacher model contributes strong capabilities in formal reasoning, quantitative analysis, and domain-specific technical knowledge.
 </p>
<div class="info-box">
 <p><strong>Key Design Principle:</strong> Both datasets prioritize detailed reasoning traces over simple question-answer pairs, ensuring that the student models learn not just what the correct answer is, but how to arrive at it through structured, step-by-step reasoning.</p>
 </div>
 </section>
<hr class="horizontal-rule">
<!-- Section 3: Methodology -->
 <section class="section" id="methodology">
 <div class="section-number">Section 3</div>
 <h2 class="section-title">Methodology</h2>
 <p>
 Our training pipeline follows a multi-stage approach designed to maximize reasoning quality while maintaining model efficiency:
 </p>
<div class="pipeline-container">
 <div class="pipeline-title">Training Pipeline Overview</div>
 <div class="pipeline-step">
 <div class="step-num">1</div>
 <div class="step-content">
 <strong>Teacher Inference:</strong> Generate detailed reasoning traces and answers using Qwen3.6-plus and Kimi-2.5-thinking on curated complex questions.
 </div>
 </div>
 <div class="pipeline-step">
 <div class="step-num">2</div>
 <div class="step-content">
 <strong>Dataset Construction:</strong> Format reasoning traces into instruction-tuning datasets with consistent structure and quality filtering.
 </div>
 </div>
 <div class="pipeline-step">
 <div class="step-num">3</div>
 <div class="step-content">
 <strong>QLoRA Fine-tuning:</strong> Apply parameter-efficient fine-tuning to Qwen3-4B-Thinking using Unsloth framework with QLoRA (PEFT).
 </div>
 </div>
 <div class="pipeline-step">
 <div class="step-num">4</div>
 <div class="step-content">
 <strong>SLERP Merge:</strong> Combine the two distilled models using a custom SLERP configuration with gradient attention and vocabulary pinning.
 </div>
 </div>
 <div class="pipeline-step">
 <div class="step-num">5</div>
 <div class="step-content">
 <strong>Evaluation:</strong> Assess all models on CMDR-Bench across 10 cognitive domains with graduated difficulty.
 </div>
 </div>
 </div>
<div class="subsection-title">3.1 Fine-tuning Configuration</div>
 <p>
 Both distilled models were fine-tuned using the following shared configuration:
 </p>
 <table>
 <tr>
 <th>Parameter</th>
 <th>Value</th>
 </tr>
 <tr>
 <td>Base Model</td>
 <td><span class="model-name">Qwen/Qwen3-4B-Thinking-2507</span></td>
 </tr>
 <tr>
 <td>Framework</td>
 <td>Unsloth</td>
 </tr>
 <tr>
 <td>Fine-tuning Method</td>
 <td>QLoRA (PEFT)</td>
 </tr>
 <tr>
 <td>Precision</td>
 <td>bfloat16</td>
 </tr>
 <tr>
 <td>Training Objective</td>
 <td>Next-token prediction with reasoning traces</td>
 </tr>
 </table>
 </section>
<hr class="horizontal-rule">
<!-- Section 4: Qwen3.6-plus Distilled Model -->
 <section class="section" id="model1">
 <div class="section-number">Section 4</div>
 <h2 class="section-title">Qwen3-4B-Qwen3.6-plus-Reasoning-Distilled</h2>
 <p>
 This model, available as <span class="model-name">khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Distilled</span>, is a reasoning-distilled variant of Qwen3-4B-Thinking, fine-tuned to replicate the advanced reasoning capabilities of the larger Qwen3.6-plus teacher model. The distillation process focuses on reducing the "rambling" and "uncertainty" often found in smaller models during complex tasks, replacing them with concise, structured, and actionable solution paths.
 </p>
<div class="subsection-title">4.1 Reasoning Quality Improvement</div>
 <p>
 The primary improvement in this model is the qualitative leap in reasoning structure. The transformation from the base model to the distilled variant is best understood through comparison:
 </p>
<div class="comparison-grid">
 <div class="comparison-card card-base">
 <h4>🔴 Base Model (Qwen3-4B-Thinking)</h4>
 <ul>
 <li>Stream-of-consciousness, exploratory, and verbose</li>
 <li>Self-talk ("Hmm, interesting", "Wait, no")</li>
 <li>Struggles with problem constraints on first attempt</li>
 <li>Enters loops of self-correction</li>
 <li>High noise-to-signal ratio</li>
 <li>Solution paths buried under hesitation</li>
 </ul>
 </div>
 <div class="comparison-card card-distilled">
 <h4>🟢 Distilled Model</h4>
 <ul>
 <li>Structured, professional, report-oriented</li>
 <li>Immediate problem analysis and constraint separation</li>
 <li>Concrete algorithm formulation (e.g., State-Space Dijkstra)</li>
 <li>Confident progression without logical dead-ends</li>
 <li>Clean output: Analysis → Intuition → Algorithm → Complexity</li>
 <li>Engineering-grade tool quality</li>
 </ul>
 </div>
 </div>
<div class="info-box">
 <p><strong>Verdict:</strong> The distilled model transforms the raw potential of the base model into an engineering-grade reasoning tool, eliminating hesitation and producing structured, actionable solution paths.</p>
 </div>
<div class="subsection-title">4.2 Model Specifications</div>
 <table>
 <tr>
 <th>Specification</th>
 <th>Value</th>
 </tr>
 <tr>
 <td>Model ID</td>
 <td><span class="model-name">khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Distilled</span></td>
 </tr>
 <tr>
 <td>Model Type</td>
 <td>Reasoning Distillation (QLoRA)</td>
 </tr>
 <tr>
 <td>Framework</td>
 <td>Unsloth</td>
 </tr>
 <tr>
 <td>Teacher Model</td>
 <td>Qwen3.6-plus</td>
 </tr>
 <tr>
 <td>Dataset</td>
 <td><span class="dataset-name">khazarai/qwen3.6-plus-high-reasoning-500x</span></td>
 </tr>
 </table>
 </section>
<hr class="horizontal-rule">
<!-- Section 5: Kimi2.5 Distilled Model -->
 <section class="section" id="model2">
 <div class="section-number">Section 5</div>
 <h2 class="section-title">Qwen3-4B-Kimi2.5-Reasoning-Distilled</h2>
 <p>
 This model, available as <span class="model-name">khazarai/Qwen3-4B-Kimi2.5-Reasoning-Distilled</span>, is fine-tuned for structured, long-form reasoning using a specialized distillation dataset generated by Kimi-2.5-thinking. It is designed to bridge the gap between small, efficient models and the complex reasoning capabilities typically found in much larger models.
 </p>
<div class="subsection-title">5.1 Key Capabilities</div>
 <ul>
 <li><strong>Problem Decomposition:</strong> Excels at breaking down complex problems into manageable sub-components.</li>
 <li><strong>Self-Correction:</strong> Demonstrates improved ability to identify and correct reasoning errors mid-generation.</li>
 <li><strong>Analytical Depth:</strong> Provides detailed analytical answers with strong causal reasoning.</li>
 <li><strong>Domain Versatility:</strong> Trained on diverse domains including technical, scientific, historical, and strategic reasoning.</li>
 </ul>
<div class="subsection-title">5.2 Model Specifications</div>
 <table>
 <tr>
 <th>Specification</th>
 <th>Value</th>
 </tr>
 <tr>
 <td>Model ID</td>
 <td><span class="model-name">khazarai/Qwen3-4B-Kimi2.5-Reasoning-Distilled</span></td>
 </tr>
 <tr>
 <td>Base Model</td>
 <td><span class="model-name">Qwen3-4b-Thinking-2507</span></td>
 </tr>
 <tr>
 <td>Training Technique</td>
 <td>Unsloth + QLoRA</td>
 </tr>
 <tr>
 <td>Teacher Model</td>
 <td>Kimi-2.5-thinking</td>
 </tr>
 <tr>
 <td>Dataset</td>
 <td><span class="dataset-name">khazarai/kimi-2.5-high-reasoning-250x</span></td>
 </tr>
 </table>
 </section>
<hr class="horizontal-rule">
<!-- Section 6: SLERP Merge Strategy -->
 <section class="section" id="merge">
 <div class="section-number">Section 6</div>
 <h2 class="section-title">SLERP Merge Strategy</h2>
 <p>
 The merged model, <span class="model-name">khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Slerp</span>, represents a highly experimental and optimized reasoning model created through a surgical SLERP (Spherical Linear Interpolation) merge of the two distilled models. The goal was to combine the deep analytical capabilities of Kimi with the mathematical and structural precision of Qwen, while mitigating the catastrophic forgetting commonly seen in standard SFT model merges.
 </p>
<div class="subsection-title">6.1 The "Golden Path" (V5) Strategy</div>
 <p>
 Standard SLERP merges often destroy RAG capabilities and syntax adherence. To solve this, we developed a custom merge configuration with two key innovations:
 </p>
<div class="subsection-title">6.1.1 RAG/Vocabulary Fix</div>
 <p>
 The <code>embed_tokens</code> and <code>lm_head</code> layers are strictly pinned to 1.0 (Qwen weights). This ensures the model reads and generates using purely Qwen's vocabulary, completely eliminating the RAG degradation problem that plagues standard merges.
 </p>
<div class="subsection-title">6.1.2 Gradient Attention</div>
 <p>
 The intermediate attention and MLP layers follow a smooth gradient <code>[0, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8, 0.9, 1]</code> across model depth. This prevents weight interference in deep reasoning steps by allowing earlier layers to retain more of the base model's knowledge while progressively incorporating the distilled model's specialized capabilities in deeper layers.
 </p>
<div class="subsection-title">6.2 Merge Configuration</div>
 <div class="code-block">
 <code>
<span class="yaml-comment"># SLERP Merge Configuration (V5 - Golden Path)</span>
<span class="yaml-key">models:</span>
<span class="yaml-key"> - model:</span> <span class="yaml-string">khazarai/Qwen3-4B-Kimi2.5-Reasoning-Distilled</span>
<span class="yaml-key"> - model:</span> <span class="yaml-string">khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Distilled</span>
<span class="yaml-key">merge_method:</span> <span class="yaml-string">slerp</span>
<span class="yaml-key">base_model:</span> <span class="yaml-string">khazarai/Qwen3-4B-Kimi2.5-Reasoning-Distilled</span>
<span class="yaml-key">parameters:</span>
<span class="yaml-key"> t:</span>
<span class="yaml-key"> - filter:</span> <span class="yaml-string">embed_tokens</span>
<span class="yaml-key"> value:</span> <span class="yaml-value">1</span> <span class="yaml-comment"># Pin to Qwen vocabulary</span>
<span class="yaml-key"> - filter:</span> <span class="yaml-string">lm_head</span>
<span class="yaml-key"> value:</span> <span class="yaml-value">1</span> <span class="yaml-comment"># Pin to Qwen vocabulary</span>
<span class="yaml-key"> - value:</span> <span class="yaml-value">1</span> <span class="yaml-comment"># Default interpolation</span>
<span class="yaml-key"> - filter:</span> <span class="yaml-string">self</span>
<span class="yaml-key"> value:</span> <span class="yaml-value">[0, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8, 0.9, 1]</span> <span class="yaml-comment"># Gradient attention</span>
<span class="yaml-key">dtype:</span> <span class="yaml-string">bfloat16</span>
 </code>
 </div>
<div class="info-box">
 <p><strong>Synergy Effect:</strong> After multiple iterations and layer-by-layer tensor analysis, we achieved a "1+1=3 Synergy Effect" in Logical Inference and Planning, with the merged model outperforming both base models and the official Qwen Thinking model in reasoning benchmarks.</p>
 </div>
<div class="warning-box">
 <p><strong>Trade-off:</strong> The sharp drop in "Creative Writing" performance is an expected and accepted trade-off to maximize extreme logical reasoning and coding precision. This model is optimized for analytical tasks, not creative generation.</p>
 </div>
<div class="subsection-title">6.3 Intended Use Cases</div>
 <div class="domain-tags">
 <span class="domain-tag">Complex Logical Deductions</span>
 <span class="domain-tag">Python Code Debugging</span>
 <span class="domain-tag">Mathematical Problem-Solving</span>
 <span class="domain-tag">RAG Pipelines</span>
 <span class="domain-tag">SQL Generation</span>
 <span class="domain-tag">Scientific Analysis</span>
 </div>
 <p style="margin-top: 12px; font-size: 13px; color: var(--text-muted); font-family: 'Inter', sans-serif;">
 <strong>Not recommended for:</strong> Creative writing, poetry, or highly imaginative storytelling.
 </p>
 </section>
<hr class="horizontal-rule">
<!-- Section 7: Benchmark -->
 <section class="section" id="benchmark">
 <div class="section-number">Section 7</div>
 <h2 class="section-title">Benchmark Design and Evaluation</h2>
 <p>
 We developed <span class="benchmark-name">khazarai/Multi-Domain-Reasoning-Benchmark</span> (CMDR-Bench), a comprehensive evaluation suite comprising 100 meticulously curated test cases across 10 distinct cognitive domains. Each domain features a graduated difficulty scale (Levels 1–10), enabling fine-grained analysis of capability thresholds from elementary to expert-level complexity.
 </p>
<div class="subsection-title">7.1 Benchmark Categories</div>
 <div class="domain-tags">
 <span class="domain-tag">Logical Reasoning (Text-Based)</span>
 <span class="domain-tag">Mathematical Reasoning</span>
 <span class="domain-tag">SQL Query Generation</span>
 <span class="domain-tag">Python Code Analysis & Debugging</span>
 <span class="domain-tag">Scientific Explanation (RAG)</span>
 <span class="domain-tag">Complex Scenario Analysis</span>
 <span class="domain-tag">Ethical Dilemma Evaluation</span>
 <span class="domain-tag">Causal Reasoning (RAG)</span>
 <span class="domain-tag">Constrained Creative Writing</span>
 <span class="domain-tag">Planning & Optimization</span>
 </div>
<div class="subsection-title">7.2 Evaluation Methodology</div>
 <p>
 Each test case is evaluated on a binary success metric (pass/fail) based on whether the model's output satisfies the task requirements. The success rate for each category is calculated as the percentage of passed test cases. Models evaluated include:
 </p>
 <table>
 <tr>
 <th>Model</th>
 <th>Type</th>
 </tr>
 <tr>
 <td><span class="model-name">Qwen/Qwen3-4B-Thinking-2507</span></td>
 <td>Base (unfine-tuned)</td>
 </tr>
 <tr>
 <td><span class="model-name">khazarai/Qwen3-4B-Kimi2.5-Reasoning-Distilled</span></td>
 <td>Distilled (Kimi teacher)</td>
 </tr>
 <tr>
 <td><span class="model-name">khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Distilled</span></td>
 <td>Distilled (Qwen teacher)</td>
 </tr>
 <tr>
 <td><span class="model-name">khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Slerp</span></td>
 <td>Merged (SLERP)</td>
 </tr>
 </table>
 </section>
<hr class="horizontal-rule">
<!-- Section 8: Results -->
 <section class="section" id="results">
 <div class="section-number">Section 8</div>
 <h2 class="section-title">Results and Analysis</h2>
 <p>
 The following figure presents the performance comparison across all four models on CMDR-Bench. Each bar represents the success rate (%) for a specific benchmark category.
 </p>
<div class="figure-container">
 <img src="/static/images/Merged_Model.png" alt="Multi-Model Reasoning Performance Comparison - Benchmark results showing success rates across 10 cognitive domains for Base, Kimi-Distilled, Qwen-Distilled, and Slerp-Merged models" />
 <div class="figure-caption">
 <strong>Figure 1:</strong> Multi-Model Reasoning Performance Comparison on CMDR-Bench. Success rates (%) across 10 cognitive domains for the base model (Qwen3-4B-Thinking-2507), two distilled variants, and the SLERP-merged model.
 </div>
 </div>
<div class="subsection-title">8.1 Key Findings</div>
<p><strong>Mathematical Reasoning:</strong> All three fine-tuned models achieve 100% success rate, compared to 100% for the base model, demonstrating that distillation preserves mathematical capabilities while improving reasoning structure.</p>
<p><strong>Python Code Analysis & Debugging:</strong> The Kimi-distilled and merged models both achieve 95.5%, a significant improvement over the base model's 69.1%. This validates the effectiveness of Kimi's analytical reasoning transfer for code-related tasks.</p>
<p><strong>Logical Reasoning:</strong> The merged model achieves 76.4%, outperforming both individual distilled models (68.2% Kimi, 60.0% Qwen) and the base model (60.0%). This demonstrates the synergy effect of combining complementary reasoning strengths.</p>
<p><strong>Scientific Explanation (RAG):</strong> All four models achieve 100%, indicating that the vocabulary pinning strategy successfully preserves RAG capabilities in the merged model.</p>
<p><strong>Constrained Creative Writing:</strong> As expected, the merged model shows reduced performance (26.4%) compared to the base model (34.5%). This trade-off is intentional and acceptable given the model's focus on analytical reasoning.</p>
<p><strong>Planning and Optimization:</strong> The merged model achieves 72.7%, significantly outperforming both distilled models (43.6% Kimi, 56.4% Qwen) and the base model (38.2%). This represents the strongest evidence of the synergy effect.</p>
<div class="subsection-title">8.2 Per-Category Performance Summary</div>
 <table>
 <tr>
 <th>Category</th>
 <th>Base</th>
 <th>Kimi-Distilled</th>
 <th>Qwen-Distilled</th>
 <th>Merged</th>
 </tr>
 <tr>
 <td>Causal Reasoning (RAG)</td>
 <td>100.0%</td>
 <td>90.9%</td>
 <td>98.2%</td>
 <td>91.8%</td>
 </tr>
 <tr>
 <td>SQL Query Generation</td>
 <td>100.0%</td>
 <td>85.5%</td>
 <td>81.8%</td>
 <td>81.8%</td>
 </tr>
 <tr>
 <td>Python Code Analysis</td>
 <td>69.1%</td>
 <td>79.1%</td>
 <td>95.5%</td>
 <td>95.5%</td>
 </tr>
 <tr>
 <td>Planning & Optimization</td>
 <td>38.2%</td>
 <td>43.6%</td>
 <td>56.4%</td>
 <td>72.7%</td>
 </tr>
 <tr>
 <td>Mathematical Reasoning</td>
 <td>100.0%</td>
 <td>90.9%</td>
 <td>100.0%</td>
 <td>100.0%</td>
 </tr>
 <tr>
 <td>Logical Reasoning</td>
 <td>60.0%</td>
 <td>68.2%</td>
 <td>60.0%</td>
 <td>76.4%</td>
 </tr>
 <tr>
 <td>Constrained Creative Writing</td>
 <td>34.5%</td>
 <td>52.7%</td>
 <td>36.4%</td>
 <td>26.4%</td>
 </tr>
 <tr>
 <td>Complex Scenario Analysis</td>
 <td>60.9%</td>
 <td>61.8%</td>
 <td>61.8%</td>
 <td>77.3%</td>
 </tr>
 <tr>
 <td>Ethical Dilemma</td>
 <td>74.5%</td>
 <td>72.7%</td>
 <td>66.4%</td>
 <td>65.5%</td>
 </tr>
 <tr>
 <td>Scientific Explanation (RAG)</td>
 <td>100.0%</td>
 <td>100.0%</td>
 <td>100.0%</td>
 <td>100.0%</td>
 </tr>
 </table>
<div class="subsection-title">8.3 Synergy Analysis</div>
 <p>
 The merged model outperforms both individual distilled models in 5 out of 10 categories: Logical Reasoning, Complex Scenario Analysis, Planning & Optimization, Python Code Analysis, and Scientific Explanation. In Planning & Optimization specifically, the merged model (72.7%) exceeds both distilled models (43.6% and 56.4%) by a substantial margin, providing strong evidence of the claimed "1+1=3" synergy effect.
 </p>
 <p>
 The gradient attention strategy appears to successfully combine the analytical depth of Kimi's distillation with the mathematical precision of Qwen's distillation, producing a model that leverages the strengths of both teacher models while mitigating their individual weaknesses.
 </p>
 </section>
<hr class="horizontal-rule">
<!-- Section 9: Conclusion -->
 <section class="section" id="conclusion">
 <div class="section-number">Section 9</div>
 <h2 class="section-title">Conclusion</h2>
 <p>
 This paper demonstrates a comprehensive pipeline for enhancing the reasoning capabilities of compact language models through knowledge distillation and strategic model merging. Our key contributions are:
 </p>
 <ol>
 <li><strong>Two domain-specific distillation datasets</strong> generated by Qwen3.6-plus and Kimi-2.5-thinking, covering complementary reasoning domains.</li>
 <li><strong>Two QLoRA-distilled models</strong> that transform the base model's stream-of-consciousness reasoning into structured, professional-grade analytical output.</li>
 <li><strong>A novel SLERP merge strategy</strong> with gradient attention and vocabulary pinning that achieves synergistic performance improvements while preserving RAG capabilities.</li>
 <li><strong>CMDR-Bench</strong>, a comprehensive 100-test-case benchmark across 10 cognitive domains for evaluating multi-domain reasoning capabilities.</li>
 </ol>
 <p>
 Our results show that the merged model achieves superior performance in logical reasoning, mathematical problem-solving, code analysis, and planning tasks compared to both individual distilled models and the base model. The trade-off in creative writing performance is intentional and acceptable for a model optimized for analytical reasoning.
 </p>
 <p>
 Future work includes exploring additional merge strategies, expanding the benchmark to more domains, and investigating the transferability of this pipeline to other base model architectures.
 </p>
<div class="info-box">
 <p><strong>Model Availability:</strong> All models, datasets, and the benchmark are publicly available on Hugging Face under the <a href="https://huggingface.co/khazarai" target="_blank">khazarai</a> organization.</p>
 </div>
 </section>
<hr class="horizontal-rule">
<!-- References -->
 <section class="section" id="references">
 <div class="section-number">References</div>
 <h2 class="section-title">References</h2>
 <ol class="references">
 <li>Qwen Team. (2025). <em>Qwen3 Technical Report</em>. Alibaba Group.</li>
 <li>Moonshot AI. (2025). <em>Kimi-2.5 Technical Report</em>.</li>
 <li>Hu, E. J., et al. (2021). <em>LoRA: Low-Rank Adaptation of Large Language Models</em>. arXiv:2106.09685.</li>
 <li>Wortsman, M., et al. (2022). <em>Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Time</em>. ICML.</li>
 <li>Ilharco, G., et al. (2022). <em>Merging Models for Free: No Additional Training Required</em>. arXiv:2207.06469.</li>
 <li>Qwen3-4B-Thinking-2507. Hugging Face. <a href="https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507" target="_blank">https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507</a></li>
 <li>Qwen3.6-plus. Hugging Face.</li>
 <li>khazarai/kimi-2.5-high-reasoning-250x. Hugging Face Dataset.</li>
 <li>khazarai/qwen3.6-plus-high-reasoning-500x. Hugging Face Dataset.</li>
 <li>khazarai/Multi-Domain-Reasoning-Benchmark. Hugging Face Dataset.</li>
 </ol>
 </section>
 </main>
<!-- Footer -->
 <footer class="footer">
 <p>© 2026 khazarai · Independent Research</p>
 <p>
 <a href="https://huggingface.co/khazarai" target="_blank">Hugging Face</a> ·
<a href="https://huggingface.co/khazarai/Qwen3-4B-Qwen3.6-plus-Reasoning-Slerp" target="_blank">Model Card</a> ·
<a href="https://huggingface.co/datasets/khazarai/Multi-Domain-Reasoning-Benchmark" target="_blank">Benchmark</a>
 </p>
 </footer>
 </div>
</body>
</html>