--- license: mit language: - en tags: - code - mermaid - flowchart - control-flow - qwen - synthetic --- # CodeFlow → Mermaid: a fine-tuning dataset for Qwen3-Coder Teach a model to read **line-numbered source code** and emit a **valid Mermaid.js flowchart** of its control flow, preceded by a `` parse and followed by a `` that ties every node back to a source line. Built to fine-tune **`Qwen3-Coder-30B-A3B-Instruct`** under the exact system prompt in [dataset/system_prompt.py](dataset/system_prompt.py). ``` Input (user turn): line-numbered source, e.g. "1| def f(x):\n2| ..." Output (assistant turn): … + graph TD … + … ``` ## What's in here | Path | Purpose | |------|---------| | [data/train.jsonl](data/train.jsonl), [data/val.jsonl](data/val.jsonl) | The dataset, chat **messages** JSONL | | [data/preview.md](data/preview.md) | Human-readable sample of generated pairs | | [dataset/](dataset/) | The synthetic-data engine that produces the dataset | | [finetune.py](finetune.py) | LoRA SFT script (prompt-masked, loads the JSONL) | ### Dataset at a glance (default `--n 2400 --seed 7`) - **2,400 examples** → 2,208 train / 192 val (8% holdout). - **Languages:** Python 864 · JavaScript 791 · C++ 417 · C 328 — *mainly Python/JS*, with C/C++ as a substantial minority. - **22 control-flow templates:** guard clauses, `if/elif/else`, nested conditionals, `for`/`while` accumulation & search, `while` with `continue`, nested loops, `try/except|catch[/finally]`, `switch`/`match`, recursion, clamp, dict/map lookup, ternary, short-circuit `&&`, state machines, `do/while`, `for/else`, plus a small fraction of **unparseable → error-node** examples. - **~5.5 nodes/graph**, **~155 tokens** per assistant target. Each record: ```json {"messages": [ {"role": "system", "content": "## Role/Persona … (the full task spec)"}, {"role": "user", "content": "1| def check_status(value):\n2| if value > 10:\n…"}, {"role": "assistant", "content": "\n…\n\ngraph TD\n A[Start: check_status]\n …\n\nA: 1\n…\n"} ]} ``` ## Why the targets are trustworthy The generator is **correct by construction** plus **hard-validated**, so the labels don't need a teacher model: 1. **Shared state.** Every template emits the source code *and* its flowchart from one builder. A node's `` line is the live line number returned when that statement was written — so injected comments, docstrings, blank lines and C++ `#include`s shift line numbers and the map stays correct automatically. 2. **Paraphrased labels.** Conditions become plain-English questions (`Index in bounds?`, not `i < len(nums)`), honoring the system prompt's strict constraint #4 — no operators, quotes, parentheses or brackets in any label. 3. **Mermaid validator.** Every example is parsed: balanced node brackets, legal label charset, no dangling edges, no markdown fences, well-formed `` and `` (line numbers in range, no unknown nodes). 4. **Real compilers.** Every Python target is `compile()`-checked in-process; JavaScript via `node --check`; C via `clang -fsyntax-only`; C++ via `clang++ -std=c++17 -fsyntax-only` (libc++ auto-located). Generation aborts on any invalid sample. > **Design note — strict constraint vs. few-shot.** The provided system prompt's > own few-shot example writes raw labels like `B{val > 10}` and `C[Return 'Active']`, > which contradicts its strict constraint #4. The targets here follow the > **strict constraint** (paraphrased, code-free labels) — the stronger, intended > behavior. The system prompt is otherwise kept verbatim so training matches > serving. If you'd rather the model mirror the looser few-shot style, regenerate > after relaxing label paraphrasing in [dataset/templates.py](dataset/templates.py). ## Regenerate / scale the dataset Pure standard-library Python; validation shells out to `node` and `clang`/`clang++`. ```bash # self-test: every template × every language, exhaustive syntax check .venv/bin/python dataset/generate.py --selftest # generate (tweak size/seed/split freely) .venv/bin/python dataset/generate.py --n 2400 --val-frac 0.08 --seed 7 .venv/bin/python dataset/generate.py --n 6000 # scale up ``` Language mix is controlled by `LANG_WEIGHT` and per-template support by the `TEMPLATES` registry, both in [dataset/](dataset/). ## Fine-tune `finetune.py` reads the JSONL, applies Qwen's chat template, and **masks the prompt** so loss falls only on the assistant turn (the thinking + graph + linemap). ```bash .venv/bin/python finetune.py --dry-run # token stats, no model download .venv/bin/python finetune.py --4bit --epochs 3 --output-dir out/qwen-mermaid ``` Defaults: LoRA `r=16, α=32` on attention + MLP projections (works with the A3B MoE blocks), cosine schedule, bf16. Use `--4bit` for QLoRA (CUDA + `bitsandbytes`). ### About the `UD-Q3_K_XL` target `Qwen3-Coder-30B-A3B-Instruct-UD-Q3_K_XL` is an Unsloth **GGUF inference quant** — you don't train it directly. Train a LoRA on the base weights (here, or via Unsloth for a turnkey path), **merge** the adapter, then **convert + quantize** to GGUF. Important distinction: `UD-Q3_K_XL` is **Unsloth Dynamic** quantization, not the same as llama.cpp's standard `Q3_K_XL`. Unsloth Dynamic GGUFs are model-specific official Unsloth releases. The local `llama.cpp` exporter can create a standard `Q3_K_XL` GGUF from your fine-tuned checkpoint, but it cannot reproduce Unsloth's `UD-*` dynamic layer recipe for a custom fine-tune. `finetune.py` can now do those post-training steps for you. When `--export-gguf` is used, it looks for llama.cpp under `.venv/llama.cpp` by default. If the checkout or quantizer is missing, it auto-clones and builds llama.cpp there. You still need system CUDA tooling plus a C++ compiler; `requirements.txt` installs the Python-side `cmake` command into the venv. ```bash .venv/bin/python -m pip install -r requirements.txt ``` Then run training and export in one command. This will auto-clone/build llama.cpp into `.venv/llama.cpp` if the GGUF tools are not already there: ```bash .venv/bin/python finetune.py \ --model Qwen/Qwen3-Coder-30B-A3B-Instruct \ --4bit \ --epochs 3 \ --batch-size 1 \ --grad-accum 16 \ --max-seq-len 2048 \ --output-dir out/qwen-mermaid-lora \ --merge-dir out/qwen-mermaid-merged \ --merge-device-map cpu \ --export-gguf \ --gguf-out out/qwen3-coder-codeflow-Q3_K_XL.gguf \ --gguf-quant Q3_K_XL \ --delete-hf-cache-before-gguf \ --delete-merged-after-gguf ``` ### Colab Pro High-RAM A100 path If you do not have enough local RAM/disk for the merge + GGUF export, use the Colab notebook: [notebooks/qwen3_coder_codeflow_gguf_to_hf_colab.ipynb](notebooks/qwen3_coder_codeflow_gguf_to_hf_colab.ipynb) In Colab: 1. Open the notebook. 2. Select `Runtime -> Change runtime type -> GPU`, then choose an A100 with High-RAM. 3. Add a Colab Secret named `HF_TOKEN` with write access to the Hugging Face repo. 4. Run the cells top to bottom. The notebook downloads this project from: ```text https://huggingface.co/build-small-hackathon/codeflow-qwen-3-finetuning ``` Then it fine-tunes, merges, auto-builds llama.cpp inside `.venv/llama.cpp`, deletes the local Hugging Face cache before GGUF conversion to reduce peak disk use, exports: ```text out/qwen3-coder-codeflow-Q3_K_XL.gguf ``` and uploads that GGUF back to: ```text build-small-hackathon/codeflow-qwen-3-finetuning ``` The final artifact will be `out/qwen3-coder-codeflow-Q3_K_XL.gguf`. That file is a fine-tuned standard `Q3_K_XL` GGUF, not `UD-Q3_K_XL`. The merge and conversion steps need substantial CPU RAM and disk space because they reload the base model in FP16 before quantizing. If GPU memory is tight during the merge, add `--merge-device-map cpu`; it is slower but avoids placing the merged base model on the RTX 5090. If you need a true Unsloth Dynamic `UD-Q3_K_XL` file, the practical options are: 1. Use the official base GGUF from `unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF`. 2. Ask Unsloth/export through an Unsloth service that can apply their internal dynamic recipe to your merged fine-tune. 3. Keep the fine-tuned LoRA separate and apply it at inference time where your runtime supports GGUF + LoRA, accepting that the final artifact is not one merged `UD-*` GGUF.