--- base_model: Qwen/Qwen3.6-27B base_model_relation: finetune license: apache-2.0 tags: - gguf - llama.cpp - lean4 - mathlib - lora pipeline_tag: text-generation --- # lean27b [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) LoRA-finetuned on Lean 4 / Mathlib4 source, merged and released as a single Q8_0 GGUF. ![Training curves](training_curves.png) *Weight-decay sweep at rank 32, all layers. The released model is **wd = 0.4** (darkest line). Final full-validation loss: 0.652. The tool-call sanity check passed 3/3 in every epoch.* ## Data - Every `.lean` file under `Mathlib/` in [mathlib4](https://github.com/leanprover-community/mathlib4) at tag `v4.33.0`. - 80/20 train/val split by file (seed 42). - Files shorter than 1024 tokens are kept whole. Longer files are split only at blank lines (`\n\n`) and packed greedily into chunks of at most 1024 tokens. - Plain causal-LM loss on all tokens. No chat template, no instruction data. ## Training | Parameter | Value | |---|---| | Method | LoRA (peft), bf16 base, gradient checkpointing | | Rank / alpha / dropout | 32 / 32 / 0.05 | | Target modules | `q_proj k_proj v_proj o_proj in_proj_qkv out_proj gate_proj up_proj down_proj` | | Layers | all 64 | | Optimizer | AdamW, lr 1e-5 (constant, no warmup), weight decay 0.4 | | Grad clip | 1.0 | | Batch size | 8 (no accumulation) | | Max sequence length | 1024 | | Epochs | 5 | | Seed | 42 (data order reshuffled with seed + epoch) | | Hardware | 1× 80 GB GPU | | Libraries | torch 2.13.0, transformers 5.15.1, peft 0.20.0, accelerate 1.14.0 | After each epoch, a check confirms the model still emits valid tool calls (grep / web_search / ls). Training stops early if 2 of the 3 calls fail in two consecutive epochs. This never triggered. **Export.** The LoRA was merged into the bf16 base, and the vision tower was dropped. The result was converted with llama.cpp `convert_hf_to_gguf.py --outtype q8_0`. The GGUF metadata was then patched from `block_count=65, nextn_predict_layers=1` to `64 / 0`, because the MTP head is not included. No tensor data was changed. ## Usage The file is `lean27b-r32_wd40_full-Q8_0.gguf` (28.6 GB). It needs a llama.cpp build that supports `qwen35`. ```bash llama-server -m lean27b-r32_wd40_full-Q8_0.gguf -ngl 99 -c 262144 --jinja ``` - Text only: no mmproj, no MTP head. - VRAM: about 26.6 GiB for the weights, about 41.6 GiB at the full 262k context. - Sampling defaults embedded in the GGUF: temperature 1.0, top_p 0.95, top_k 20.