| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - mlx |
| - linear-algebra |
| - math |
| - qwen3 |
| - lora |
| base_model: Qwen/Qwen3-4B |
| --- |
| |
| # la-math-v9 — Qwen3-4B fine-tuned for linear algebra |
|
|
| A LoRA fine-tune of Qwen3-4B specialized in linear algebra: determinants, inverses, |
| eigenvalues, systems of equations, rank/spaces, factorizations, matrix operations, |
| and linear algebra theory (definitions, theorems, examples). |
|
|
| **Best LA specialist so far in the la-math series (v5 → fix2 → v9).** |
|
|
| ## Evaluation |
|
|
| 416-pair held-out linear algebra set, SymPy-graded with an identical harness across all models |
| (284 items computationally gradable; the rest are theory/definitional). |
|
|
| | Model | Accuracy (regraded) | |
| |---|---| |
| | **la-math-v9** | **75.0%** | |
| | la-math-v5-fix2 | 73.5% | |
| | la-math-v5 | 72.8% | |
| | la-math-v5-fix1 | 71.4% | |
| | Qwen3-4B base | 67.8% | |
|
|
| Category breakdown (v9): matrix_ops 100%, orthogonality 100%, lin_trans 88%, inverses 85%, |
| determinants 84%, rank_spaces 67%, systems 65%, least_squares 60%, eigen 58%. |
|
|
| ## Training |
|
|
| - Base: Qwen3-4B (MLX, 4-bit quantized) |
| - Data: `la_data_v8` — 9,703 training pairs (curated from v5 + gap-fill items + PDF-extracted |
| theorems/definitions/examples from 8 standard linear algebra textbooks), audited self-consistent, |
| answers capped at 1,024 tokens |
| - Method: LoRA, 16 layers, rank from mlx_lm defaults (0.182% trainable params) |
| - Fresh run: 9,703 iters (1 epoch), batch size 1, LR 1e-5, max seq 1,024, grad-checkpoint, mask-prompt |
| - Fused from the **iter-7000 checkpoint** (val-loss floor 0.047) — late-epoch overfit avoided |
| - Fused + requantized to 4-bit (group size 64) |
| |
| ## Usage |
| |
| MLX (mlx-lm): |
| |
| ```python |
| from mlx_lm import load, generate |
| model, tokenizer = load("j01001100/la-math-v9") |
| response = generate(model, tokenizer, prompt="Find the inverse of [[2,1],[5,3]]", max_tokens=768) |
| ``` |
| |
| LM Studio: local model, 2.28 GB, runs on Apple Silicon. For a tutor that verifies its own |
| answers, wrap with inference-time SymPy verification (generate → verify → retry). |
| |
| ## License & attribution |
| |
| Apache-2.0. Base model: Qwen/Qwen3-4B (Apache-2.0). Fine-tune by jason (j01001100). |
| |
