noahsta commited on
Commit
d99fa63
·
verified ·
1 Parent(s): 258abe6

Concise README with full training params

Browse files
Files changed (1) hide show
  1. README.md +35 -24
README.md CHANGED
@@ -5,45 +5,56 @@ license: apache-2.0
5
  tags:
6
  - gguf
7
  - llama.cpp
8
- - qwen3.6
9
- - qwen35
 
10
  pipeline_tag: text-generation
11
  ---
12
 
13
  # lean27b
14
 
15
- A finetune of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B), released as a single Q8_0 GGUF file for llama.cpp.
16
 
17
- ## Training
18
 
19
- | Hyperparameter | Value |
20
- |---|---|
21
- | r | 32 |
22
- | weight decay (wd) | 0.04 |
23
 
24
- ## File
25
 
26
- | File | Quant | Size |
27
- |---|---|---|
28
- | `lean27b-r32_wd40_full-Q8_0.gguf` | Q8_0 | 28.6 GB (26.6 GiB) |
 
 
 
29
 
30
- - Architecture: `qwen35` (hybrid: 64 layers, of which every 4th uses full attention and the rest are SSM layers), so the KV cache is small (about 64 KB per token at f16).
31
- - Context length: 262144 (the model's trained maximum).
32
- - Text only: no vision projector (mmproj) file is included.
33
- - No MTP / speculative-decoding head (`nextn_predict_layers = 0`).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
  ## Usage
36
 
37
- You need a llama.cpp build that supports the `qwen35` architecture.
38
 
39
  ```bash
40
  llama-server -m lean27b-r32_wd40_full-Q8_0.gguf -ngl 99 -c 262144 --jinja
41
  ```
42
 
43
- Recommended sampling settings (embedded in the GGUF): temperature 1.0, top_p 0.95, top_k 20.
44
-
45
- Memory: the weights alone take about 26.6 GiB of VRAM. At the full 262144 context the total measured about 41.6 GiB.
46
-
47
- ## Note on metadata
48
-
49
- The original merged export carried over `block_count = 65` and `nextn_predict_layers = 1` from the MTP base checkpoint, although the tensors contain only 64 layers and no MTP head. In this file both values are corrected to `64` / `0`. No tensor data was changed.
 
5
  tags:
6
  - gguf
7
  - llama.cpp
8
+ - lean4
9
+ - mathlib
10
+ - lora
11
  pipeline_tag: text-generation
12
  ---
13
 
14
  # lean27b
15
 
16
+ [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) LoRA-finetuned on Lean 4 / Mathlib4 source, merged and released as a single Q8_0 GGUF.
17
 
18
+ ![Training curves](training_curves.png)
19
 
20
+ *Weight-decay sweep at rank 32, all layers. The released model is **wd = 0.4** (darkest line). Final full-validation loss: 0.652. The tool-call sanity check passed 3/3 in every epoch.*
 
 
 
21
 
22
+ ## Data
23
 
24
+ - Every `.lean` file under `Mathlib/` in [mathlib4](https://github.com/leanprover-community/mathlib4) at tag `v4.33.0`.
25
+ - 80/20 train/val split by file (seed 42).
26
+ - Files shorter than 1024 tokens are kept whole. Longer files are split only at blank lines (`\n\n`) and packed greedily into chunks of at most 1024 tokens.
27
+ - Plain causal-LM loss on all tokens. No chat template, no instruction data.
28
+
29
+ ## Training
30
 
31
+ | Parameter | Value |
32
+ |---|---|
33
+ | Method | LoRA (peft), bf16 base, gradient checkpointing |
34
+ | Rank / alpha / dropout | 32 / 32 / 0.05 |
35
+ | Target modules | `q_proj k_proj v_proj o_proj in_proj_qkv out_proj gate_proj up_proj down_proj` |
36
+ | Layers | all 64 |
37
+ | Optimizer | AdamW, lr 1e-5 (constant, no warmup), weight decay 0.4 |
38
+ | Grad clip | 1.0 |
39
+ | Batch size | 8 (no accumulation) |
40
+ | Max sequence length | 1024 |
41
+ | Epochs | 5 |
42
+ | Seed | 42 (data order reshuffled with seed + epoch) |
43
+ | Hardware | 1× 80 GB GPU |
44
+ | Libraries | torch 2.13.0, transformers 5.15.1, peft 0.20.0, accelerate 1.14.0 |
45
+
46
+ After each epoch, a check confirms the model still emits valid tool calls (grep / web_search / ls). Training stops early if 2 of the 3 calls fail in two consecutive epochs. This never triggered.
47
+
48
+ **Export.** The LoRA was merged into the bf16 base, and the vision tower was dropped. The result was converted with llama.cpp `convert_hf_to_gguf.py --outtype q8_0`. The GGUF metadata was then patched from `block_count=65, nextn_predict_layers=1` to `64 / 0`, because the MTP head is not included. No tensor data was changed.
49
 
50
  ## Usage
51
 
52
+ The file is `lean27b-r32_wd40_full-Q8_0.gguf` (28.6 GB). It needs a llama.cpp build that supports `qwen35`.
53
 
54
  ```bash
55
  llama-server -m lean27b-r32_wd40_full-Q8_0.gguf -ngl 99 -c 262144 --jinja
56
  ```
57
 
58
+ - Text only: no mmproj, no MTP head.
59
+ - VRAM: about 26.6 GiB for the weights, about 41.6 GiB at the full 262k context.
60
+ - Sampling defaults embedded in the GGUF: temperature 1.0, top_p 0.95, top_k 20.