build-small-hackathon/codeflow-qwen-3-finetuning

CodeFlow → Mermaid: a fine-tuning dataset for Qwen3-Coder

Teach a model to read line-numbered source code and emit a valid Mermaid.js flowchart of its control flow, preceded by a <thinking> parse and followed by a <linemap> that ties every node back to a source line.

Built to fine-tune Qwen3-Coder-30B-A3B-Instruct under the exact system prompt in dataset/system_prompt.py.

Input  (user turn):      line-numbered source, e.g.  "1| def f(x):\n2|     ..."
Output (assistant turn): <thinking>…</thinking> + graph TD … + <linemap>…</linemap>

What's in here

Path	Purpose
data/train.jsonl, data/val.jsonl	The dataset, chat messages JSONL
data/preview.md	Human-readable sample of generated pairs
dataset/	The synthetic-data engine that produces the dataset
finetune.py	LoRA SFT script (prompt-masked, loads the JSONL)

Dataset at a glance (default --n 2400 --seed 7)

• 2,400 examples → 2,208 train / 192 val (8% holdout).
• Languages: Python 864 · JavaScript 791 · C++ 417 · C 328 — mainly Python/JS, with C/C++ as a substantial minority.
• 22 control-flow templates: guard clauses, if/elif/else, nested conditionals, for/while accumulation & search, while with continue, nested loops, try/except|catch[/finally], switch/match, recursion, clamp, dict/map lookup, ternary, short-circuit &&, state machines, do/while, for/else, plus a small fraction of unparseable → error-node examples.
• ~5.5 nodes/graph, ~155 tokens per assistant target.

Each record:

{"messages": [
  {"role": "system",    "content": "## Role/Persona … (the full task spec)"},
  {"role": "user",      "content": "1| def check_status(value):\n2|     if value > 10:\n…"},
  {"role": "assistant", "content": "<thinking>\n…\n</thinking>\ngraph TD\n    A[Start: check_status]\n    …\n<linemap>\nA: 1\n…\n</linemap>"}
]}

Why the targets are trustworthy

The generator is correct by construction plus hard-validated, so the labels don't need a teacher model:

1. Shared state. Every template emits the source code and its flowchart from one builder. A node's <linemap> line is the live line number returned when that statement was written — so injected comments, docstrings, blank lines and C++ #includes shift line numbers and the map stays correct automatically.
2. Paraphrased labels. Conditions become plain-English questions (Index in bounds?, not i < len(nums)), honoring the system prompt's strict constraint #4 — no operators, quotes, parentheses or brackets in any label.
3. Mermaid validator. Every example is parsed: balanced node brackets, legal label charset, no dangling edges, no markdown fences, well-formed <thinking> and <linemap> (line numbers in range, no unknown nodes).
4. Real compilers. Every Python target is compile()-checked in-process; JavaScript via node --check; C via clang -fsyntax-only; C++ via clang++ -std=c++17 -fsyntax-only (libc++ auto-located). Generation aborts on any invalid sample.

Design note — strict constraint vs. few-shot. The provided system prompt's own few-shot example writes raw labels like B{val > 10} and C[Return 'Active'], which contradicts its strict constraint #4. The targets here follow the strict constraint (paraphrased, code-free labels) — the stronger, intended behavior. The system prompt is otherwise kept verbatim so training matches serving. If you'd rather the model mirror the looser few-shot style, regenerate after relaxing label paraphrasing in dataset/templates.py.

Regenerate / scale the dataset

Pure standard-library Python; validation shells out to node and clang/clang++.

# self-test: every template × every language, exhaustive syntax check
.venv/bin/python dataset/generate.py --selftest

# generate (tweak size/seed/split freely)
.venv/bin/python dataset/generate.py --n 2400 --val-frac 0.08 --seed 7
.venv/bin/python dataset/generate.py --n 6000            # scale up

Language mix is controlled by LANG_WEIGHT and per-template support by the TEMPLATES registry, both in dataset/.

Fine-tune

finetune.py reads the JSONL, applies Qwen's chat template, and masks the prompt so loss falls only on the assistant turn (the thinking + graph + linemap).

.venv/bin/python finetune.py --dry-run                       # token stats, no model download
.venv/bin/python finetune.py --4bit --epochs 3 --output-dir out/qwen-mermaid

Defaults: LoRA r=16, α=32 on attention + MLP projections (works with the A3B MoE blocks), cosine schedule, bf16. Use --4bit for QLoRA (CUDA + bitsandbytes).

About the UD-Q3_K_XL target

Qwen3-Coder-30B-A3B-Instruct-UD-Q3_K_XL is an Unsloth GGUF inference quant — you don't train it directly. Train a LoRA on the base weights (here, or via Unsloth for a turnkey path), merge the adapter, then convert + quantize to GGUF.

Important distinction: UD-Q3_K_XL is Unsloth Dynamic quantization, not the same as llama.cpp's standard Q3_K_XL. Unsloth Dynamic GGUFs are model-specific official Unsloth releases. The local llama.cpp exporter can create a standard Q3_K_XL GGUF from your fine-tuned checkpoint, but it cannot reproduce Unsloth's UD-* dynamic layer recipe for a custom fine-tune.

finetune.py can now do those post-training steps for you. When --export-gguf is used, it looks for llama.cpp under .venv/llama.cpp by default. If the checkout or quantizer is missing, it auto-clones and builds llama.cpp there. You still need system CUDA tooling plus a C++ compiler; requirements.txt installs the Python-side cmake command into the venv.

.venv/bin/python -m pip install -r requirements.txt

Then run training and export in one command. This will auto-clone/build llama.cpp into .venv/llama.cpp if the GGUF tools are not already there:

.venv/bin/python finetune.py \
  --model Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --4bit \
  --epochs 3 \
  --batch-size 1 \
  --grad-accum 16 \
  --max-seq-len 2048 \
  --output-dir out/qwen-mermaid-lora \
  --merge-dir out/qwen-mermaid-merged \
  --merge-device-map cpu \
  --export-gguf \
  --gguf-out out/qwen3-coder-codeflow-Q3_K_XL.gguf \
  --gguf-quant Q3_K_XL \
  --delete-hf-cache-before-gguf \
  --delete-merged-after-gguf

Colab Pro High-RAM A100 path

If you do not have enough local RAM/disk for the merge + GGUF export, use the Colab notebook:

notebooks/qwen3_coder_codeflow_gguf_to_hf_colab.ipynb

In Colab:

1. Open the notebook.
2. Select Runtime -> Change runtime type -> GPU, then choose an A100 with High-RAM.
3. Add a Colab Secret named HF_TOKEN with write access to the Hugging Face repo.
4. Run the cells top to bottom.

The notebook downloads this project from:

https://huggingface.co/build-small-hackathon/codeflow-qwen-3-finetuning

Then it fine-tunes, merges, auto-builds llama.cpp inside .venv/llama.cpp, deletes the local Hugging Face cache before GGUF conversion to reduce peak disk use, exports:

out/qwen3-coder-codeflow-Q3_K_XL.gguf

and uploads that GGUF back to:

build-small-hackathon/codeflow-qwen-3-finetuning

The final artifact will be out/qwen3-coder-codeflow-Q3_K_XL.gguf. That file is a fine-tuned standard Q3_K_XL GGUF, not UD-Q3_K_XL. The merge and conversion steps need substantial CPU RAM and disk space because they reload the base model in FP16 before quantizing. If GPU memory is tight during the merge, add --merge-device-map cpu; it is slower but avoids placing the merged base model on the RTX 5090.

If you need a true Unsloth Dynamic UD-Q3_K_XL file, the practical options are:

1. Use the official base GGUF from unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF.
2. Ask Unsloth/export through an Unsloth service that can apply their internal dynamic recipe to your merged fine-tune.
3. Keep the fine-tuned LoRA separate and apply it at inference time where your runtime supports GGUF + LoRA, accepting that the final artifact is not one merged UD-* GGUF.