Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -21,9 +21,34 @@ datasets:
|
|
| 21 |
# CodeK LoRA v1 -- Qwen2.5-Coder-7B-Instruct
|
| 22 |
|
| 23 |
A LoRA adapter fine-tuned on the **CodeK v2** dataset: a reasoning-first, pedagogical
|
| 24 |
-
coding dataset with ~2x the seeds of
|
| 25 |
reasoning, and hypothesis-driven thinking about code.
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
## Training
|
| 28 |
|
| 29 |
| Setting | Value |
|
|
@@ -53,27 +78,15 @@ reasoning, and hypothesis-driven thinking about code.
|
|
| 53 |
| 600 | 0.0747 |
|
| 54 |
| 700 | 0.0747 |
|
| 55 |
| 800 | 0.0689 |
|
| 56 |
-
| **900** | **0.0664 ← best** |
|
| 57 |
| 1000 | 0.0755 |
|
| 58 |
| 1100 | 0.0765 |
|
| 59 |
| 1200 | 0.0761 |
|
| 60 |
-
| 1300 | 0.0767 |
|
| 61 |
-
|
| 62 |
-
Best checkpoint (step 900) was rotated out by save_total_limit=3.
|
| 63 |
-
Checkpoint-1300 used for eval (eval loss 0.077).
|
| 64 |
-
|
| 65 |
-
## v0 Baseline Comparison
|
| 66 |
-
|
| 67 |
-
| Model | Train pairs | Best eval loss | Pass@1 (bug diagnosis) |
|
| 68 |
-
|-------|------------|---------------|----------------------|
|
| 69 |
-
| CodeK LoRA v0 (checkpoint-800) | 2,351 | 0.0583 | 58% |
|
| 70 |
-
| **CodeK LoRA v1 (checkpoint-1300)** | **4,567** | **0.0664** | **TBD** |
|
| 71 |
-
|
| 72 |
-
Pass@1 eval pending. See [CodeK LoRA v0](https://huggingface.co/mechramc/codek-qwen2.5-coder-7b-lora) for baseline analysis.
|
| 73 |
|
| 74 |
## Dataset
|
| 75 |
|
| 76 |
-
mechramc/codek-v2 (coming soon) --
|
| 77 |
398 seeds, 4 augmentation passes, 5,075 ShareGPT pairs.
|
| 78 |
Categories: data structures, algorithms, ML fundamentals, NN components,
|
| 79 |
training infra, utilities, numerical, parsing.
|
|
@@ -88,3 +101,7 @@ base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
|
|
| 88 |
model = PeftModel.from_pretrained(base, "mechramc/codek-qwen2.5-coder-7b-lora-v2")
|
| 89 |
tokenizer = AutoTokenizer.from_pretrained("mechramc/codek-qwen2.5-coder-7b-lora-v2")
|
| 90 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
# CodeK LoRA v1 -- Qwen2.5-Coder-7B-Instruct
|
| 22 |
|
| 23 |
A LoRA adapter fine-tuned on the **CodeK v2** dataset: a reasoning-first, pedagogical
|
| 24 |
+
coding dataset with ~2x the seeds of v0. Teaches decomposition, bug diagnosis, contrast
|
| 25 |
reasoning, and hypothesis-driven thinking about code.
|
| 26 |
|
| 27 |
+
## Eval Results (Pass 2 ground-truth, 50 seeds)
|
| 28 |
+
|
| 29 |
+
| Model | Pass@1 | vs v0 |
|
| 30 |
+
|-------|--------|-------|
|
| 31 |
+
| Base (Qwen2.5-Coder-7B-Instruct) | 62% | -2% |
|
| 32 |
+
| **LoRA v1 (checkpoint-1300)** | **60%** | **+2%** |
|
| 33 |
+
|
| 34 |
+
The regression gap vs base model closed from **-6% (v0)** to **-2% (v1)**.
|
| 35 |
+
Evaluated on the same 50 seeds as v0 for direct comparison.
|
| 36 |
+
|
| 37 |
+
Note: best checkpoint (step 900, eval loss 0.0664) was rotated out during training
|
| 38 |
+
(save_total_limit=3). checkpoint-1300 (eval loss 0.077) used instead. True best
|
| 39 |
+
checkpoint would likely score 62–64%.
|
| 40 |
+
|
| 41 |
+
## v0 → v1 Comparison
|
| 42 |
+
|
| 43 |
+
| | v0 | v1 |
|
| 44 |
+
|--|----|----|
|
| 45 |
+
| Dataset | codek-v1 (201 seeds) | codek-v2 (398 seeds) |
|
| 46 |
+
| Train pairs | 2,351 | 4,567 |
|
| 47 |
+
| Best eval loss | 0.0583 | 0.0664 (best surviving: 0.077) |
|
| 48 |
+
| LoRA Pass@1 | 58% | **60%** |
|
| 49 |
+
| Base Pass@1 | 64% | 62% |
|
| 50 |
+
| Gap (LoRA vs base) | -6% | **-2%** |
|
| 51 |
+
|
| 52 |
## Training
|
| 53 |
|
| 54 |
| Setting | Value |
|
|
|
|
| 78 |
| 600 | 0.0747 |
|
| 79 |
| 700 | 0.0747 |
|
| 80 |
| 800 | 0.0689 |
|
| 81 |
+
| **900** | **0.0664 ← best (rotated out)** |
|
| 82 |
| 1000 | 0.0755 |
|
| 83 |
| 1100 | 0.0765 |
|
| 84 |
| 1200 | 0.0761 |
|
| 85 |
+
| 1300 | 0.0767 ← used for eval |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
## Dataset
|
| 88 |
|
| 89 |
+
[mechramc/codek-v2](https://huggingface.co/datasets/mechramc/codek-v2) (coming soon) --
|
| 90 |
398 seeds, 4 augmentation passes, 5,075 ShareGPT pairs.
|
| 91 |
Categories: data structures, algorithms, ML fundamentals, NN components,
|
| 92 |
training infra, utilities, numerical, parsing.
|
|
|
|
| 101 |
model = PeftModel.from_pretrained(base, "mechramc/codek-qwen2.5-coder-7b-lora-v2")
|
| 102 |
tokenizer = AutoTokenizer.from_pretrained("mechramc/codek-qwen2.5-coder-7b-lora-v2")
|
| 103 |
```
|
| 104 |
+
|
| 105 |
+
## Links
|
| 106 |
+
|
| 107 |
+
- [v0 adapter (baseline)](https://huggingface.co/mechramc/codek-qwen2.5-coder-7b-lora)
|