File size: 2,510 Bytes
54a3805
 
 
 
 
 
 
d99fa63
 
 
54a3805
 
 
 
 
d99fa63
54a3805
d99fa63
54a3805
d99fa63
54a3805
d99fa63
54a3805
d99fa63
 
 
 
 
 
54a3805
d99fa63
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54a3805
 
 
d99fa63
54a3805
 
 
 
 
d99fa63
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
---
base_model: Qwen/Qwen3.6-27B
base_model_relation: finetune
license: apache-2.0
tags:
  - gguf
  - llama.cpp
  - lean4
  - mathlib
  - lora
pipeline_tag: text-generation
---

# lean27b

[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) LoRA-finetuned on Lean 4 / Mathlib4 source, merged and released as a single Q8_0 GGUF.

![Training curves](training_curves.png)

*Weight-decay sweep at rank 32, all layers. The released model is **wd = 0.4** (darkest line). Final full-validation loss: 0.652. The tool-call sanity check passed 3/3 in every epoch.*

## Data

- Every `.lean` file under `Mathlib/` in [mathlib4](https://github.com/leanprover-community/mathlib4) at tag `v4.33.0`.
- 80/20 train/val split by file (seed 42).
- Files shorter than 1024 tokens are kept whole. Longer files are split only at blank lines (`\n\n`) and packed greedily into chunks of at most 1024 tokens.
- Plain causal-LM loss on all tokens. No chat template, no instruction data.

## Training

| Parameter | Value |
|---|---|
| Method | LoRA (peft), bf16 base, gradient checkpointing |
| Rank / alpha / dropout | 32 / 32 / 0.05 |
| Target modules | `q_proj k_proj v_proj o_proj in_proj_qkv out_proj gate_proj up_proj down_proj` |
| Layers | all 64 |
| Optimizer | AdamW, lr 1e-5 (constant, no warmup), weight decay 0.4 |
| Grad clip | 1.0 |
| Batch size | 8 (no accumulation) |
| Max sequence length | 1024 |
| Epochs | 5 |
| Seed | 42 (data order reshuffled with seed + epoch) |
| Hardware | 1× 80 GB GPU |
| Libraries | torch 2.13.0, transformers 5.15.1, peft 0.20.0, accelerate 1.14.0 |

After each epoch, a check confirms the model still emits valid tool calls (grep / web_search / ls). Training stops early if 2 of the 3 calls fail in two consecutive epochs. This never triggered.

**Export.** The LoRA was merged into the bf16 base, and the vision tower was dropped. The result was converted with llama.cpp `convert_hf_to_gguf.py --outtype q8_0`. The GGUF metadata was then patched from `block_count=65, nextn_predict_layers=1` to `64 / 0`, because the MTP head is not included. No tensor data was changed.

## Usage

The file is `lean27b-r32_wd40_full-Q8_0.gguf` (28.6 GB). It needs a llama.cpp build that supports `qwen35`.

```bash
llama-server -m lean27b-r32_wd40_full-Q8_0.gguf -ngl 99 -c 262144 --jinja
```

- Text only: no mmproj, no MTP head.
- VRAM: about 26.6 GiB for the weights, about 41.6 GiB at the full 262k context.
- Sampling defaults embedded in the GGUF: temperature 1.0, top_p 0.95, top_k 20.