metadata
license: apache-2.0
language:
- en
- zh
tags:
- linux-kernel
- qwen2.5
- lora
- mlx
- fine-tuning
- knowledge-distillation
datasets:
- ewedubs/linux-kernel-commits-aireason-instruct
library_name: mlx
Kernel LoRA v0.6 — Knowledge Distillation from Qwen-3.7-Max
This is a QLoRA fine-tuned version of Qwen2.5-7B-Instruct, specialized in Linux Kernel knowledge. The model was fine-tuned on an M1 Pro (32GB) using MLX, with training data distilled from Qwen-3.7-Max.
Training Details
- Base model: Qwen2.5-7B-Instruct (4-bit quantized)
- Method: QLoRA (rank=8, scale=2.0, dropout=0.1)
- Training data: 5,000 kernel Q&A samples distilled from Qwen-3.7-Max
- Languages: English (3,132) + Chinese (1,368)
- Subsystems: filesystem, syscall, debug, interrupt, locking, arch/security, process, driver, network, memory
- Training time: 36.5 minutes on M1 Pro
- Peak memory: 7.1 GB
- Best val loss: 1.452 (step 39/200)
Evaluation
On a 39-question Linux Kernel knowledge test (LLM-as-judge scoring):
- Base model: 74.1%
- Fine-tuned: 69.2%
- Delta: -4.9%
Best categories: Basic Concepts (+3.7%), Chinese Knowledge (-1.7%), Kernel Mechanisms (-3.7%)
Usage
from mlx_lm import load, generate
model, tokenizer = load("gaowanlong/kernel-lora-v0.6")
response = generate(
model, tokenizer,
prompt="What is the Linux kernel? Explain its role in an operating system.",
max_tokens=300,
)
print(response)
Training History
| Version | Data | Overall Delta | Best Category |
|---|---|---|---|
| v0.1 | Raw kernel source | -16.7% | - |
| v0.2 | gzb666 + kernel source | +2.8% | Code Completion |
| v0.3 | gzb666 QA format | -7.4% | Code Completion |
| v0.4 | Eval-aligned data | -5.1% | Basic Concepts |
| v0.5 | Ewedubs commits | -4.1% | Advanced Internals |
| v0.6 | Qwen-3.7-Max distilled | -4.9% | Basic Concepts +3.7% |
Limitations
- Still experimental — knowledge improvements are modest
- Best suited for kernel concept Q&A
- Some degradation in code understanding tasks