| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| tags: |
| - linux-kernel |
| - qwen2.5 |
| - lora |
| - mlx |
| - fine-tuning |
| - knowledge-distillation |
| datasets: |
| - ewedubs/linux-kernel-commits-aireason-instruct |
| library_name: mlx |
| --- |
| |
| # Kernel LoRA v0.6 — Knowledge Distillation from Qwen-3.7-Max |
|
|
| This is a QLoRA fine-tuned version of Qwen2.5-7B-Instruct, specialized in Linux Kernel knowledge. The model was fine-tuned on an M1 Pro (32GB) using MLX, with training data distilled from Qwen-3.7-Max. |
|
|
| ## Training Details |
|
|
| - **Base model**: Qwen2.5-7B-Instruct (4-bit quantized) |
| - **Method**: QLoRA (rank=8, scale=2.0, dropout=0.1) |
| - **Training data**: 5,000 kernel Q&A samples distilled from Qwen-3.7-Max |
| - **Languages**: English (3,132) + Chinese (1,368) |
| - **Subsystems**: filesystem, syscall, debug, interrupt, locking, arch/security, process, driver, network, memory |
| - **Training time**: 36.5 minutes on M1 Pro |
| - **Peak memory**: 7.1 GB |
| - **Best val loss**: 1.452 (step 39/200) |
|
|
| ## Evaluation |
|
|
| On a 39-question Linux Kernel knowledge test (LLM-as-judge scoring): |
| - Base model: 74.1% |
| - Fine-tuned: 69.2% |
| - Delta: -4.9% |
|
|
| Best categories: Basic Concepts (+3.7%), Chinese Knowledge (-1.7%), Kernel Mechanisms (-3.7%) |
|
|
| ## Usage |
|
|
| ```python |
| from mlx_lm import load, generate |
| |
| model, tokenizer = load("gaowanlong/kernel-lora-v0.6") |
| |
| response = generate( |
| model, tokenizer, |
| prompt="What is the Linux kernel? Explain its role in an operating system.", |
| max_tokens=300, |
| ) |
| print(response) |
| ``` |
|
|
| ## Training History |
|
|
| | Version | Data | Overall Delta | Best Category | |
| |---------|------|--------------|---------------| |
| | v0.1 | Raw kernel source | -16.7% | - | |
| | v0.2 | gzb666 + kernel source | +2.8% | Code Completion | |
| | v0.3 | gzb666 QA format | -7.4% | Code Completion | |
| | v0.4 | Eval-aligned data | -5.1% | Basic Concepts | |
| | v0.5 | Ewedubs commits | -4.1% | Advanced Internals | |
| | **v0.6** | **Qwen-3.7-Max distilled** | **-4.9%** | **Basic Concepts +3.7%** | |
|
|
| ## Limitations |
|
|
| - Still experimental — knowledge improvements are modest |
| - Best suited for kernel concept Q&A |
| - Some degradation in code understanding tasks |
|
|
