lean27b / README.md
noahsta's picture
Concise README with full training params
d99fa63 verified
|
Raw History Blame Contribute Delete
2.51 kB
metadata
base_model: Qwen/Qwen3.6-27B
base_model_relation: finetune
license: apache-2.0
tags:
  - gguf
  - llama.cpp
  - lean4
  - mathlib
  - lora
pipeline_tag: text-generation

lean27b

Qwen/Qwen3.6-27B LoRA-finetuned on Lean 4 / Mathlib4 source, merged and released as a single Q8_0 GGUF.

Training curves

Weight-decay sweep at rank 32, all layers. The released model is wd = 0.4 (darkest line). Final full-validation loss: 0.652. The tool-call sanity check passed 3/3 in every epoch.

Data

  • Every .lean file under Mathlib/ in mathlib4 at tag v4.33.0.
  • 80/20 train/val split by file (seed 42).
  • Files shorter than 1024 tokens are kept whole. Longer files are split only at blank lines (\n\n) and packed greedily into chunks of at most 1024 tokens.
  • Plain causal-LM loss on all tokens. No chat template, no instruction data.

Training

Parameter Value
Method LoRA (peft), bf16 base, gradient checkpointing
Rank / alpha / dropout 32 / 32 / 0.05
Target modules q_proj k_proj v_proj o_proj in_proj_qkv out_proj gate_proj up_proj down_proj
Layers all 64
Optimizer AdamW, lr 1e-5 (constant, no warmup), weight decay 0.4
Grad clip 1.0
Batch size 8 (no accumulation)
Max sequence length 1024
Epochs 5
Seed 42 (data order reshuffled with seed + epoch)
Hardware 1× 80 GB GPU
Libraries torch 2.13.0, transformers 5.15.1, peft 0.20.0, accelerate 1.14.0

After each epoch, a check confirms the model still emits valid tool calls (grep / web_search / ls). Training stops early if 2 of the 3 calls fail in two consecutive epochs. This never triggered.

Export. The LoRA was merged into the bf16 base, and the vision tower was dropped. The result was converted with llama.cpp convert_hf_to_gguf.py --outtype q8_0. The GGUF metadata was then patched from block_count=65, nextn_predict_layers=1 to 64 / 0, because the MTP head is not included. No tensor data was changed.

Usage

The file is lean27b-r32_wd40_full-Q8_0.gguf (28.6 GB). It needs a llama.cpp build that supports qwen35.

llama-server -m lean27b-r32_wd40_full-Q8_0.gguf -ngl 99 -c 262144 --jinja
  • Text only: no mmproj, no MTP head.
  • VRAM: about 26.6 GiB for the weights, about 41.6 GiB at the full 262k context.
  • Sampling defaults embedded in the GGUF: temperature 1.0, top_p 0.95, top_k 20.