README.md

---
license: mit
language:
  - en
tags:
  - code
  - mermaid
  - flowchart
  - control-flow
  - qwen
  - synthetic
---

# CodeFlow → Mermaid: a fine-tuning dataset for Qwen3-Coder

Teach a model to read **line-numbered source code** and emit a **valid Mermaid.js
flowchart** of its control flow, preceded by a `<thinking>` parse and followed by
a `<linemap>` that ties every node back to a source line.

Built to fine-tune **`Qwen3-Coder-30B-A3B-Instruct`** under the exact system
prompt in [dataset/system_prompt.py](dataset/system_prompt.py).

```
Input  (user turn):      line-numbered source, e.g.  "1| def f(x):\n2|     ..."
Output (assistant turn): <thinking>…</thinking> + graph TD … + <linemap>…</linemap>
```

## What's in here

| Path | Purpose |
|------|---------|
| [data/train.jsonl](data/train.jsonl), [data/val.jsonl](data/val.jsonl) | The dataset, chat **messages** JSONL |
| [data/preview.md](data/preview.md) | Human-readable sample of generated pairs |
| [dataset/](dataset/) | The synthetic-data engine that produces the dataset |
| [finetune.py](finetune.py) | LoRA SFT script (prompt-masked, loads the JSONL) |

### Dataset at a glance (default `--n 2400 --seed 7`)

- **2,400 examples** → 2,208 train / 192 val (8% holdout).
- **Languages:** Python 864 · JavaScript 791 · C++ 417 · C 328 — *mainly Python/JS*, with C/C++ as a substantial minority.
- **22 control-flow templates:** guard clauses, `if/elif/else`, nested conditionals, `for`/`while` accumulation & search, `while` with `continue`, nested loops, `try/except|catch[/finally]`, `switch`/`match`, recursion, clamp, dict/map lookup, ternary, short-circuit `&&`, state machines, `do/while`, `for/else`, plus a small fraction of **unparseable → error-node** examples.
- **~5.5 nodes/graph**, **~155 tokens** per assistant target.

Each record:

```json
{"messages": [
  {"role": "system",    "content": "## Role/Persona … (the full task spec)"},
  {"role": "user",      "content": "1| def check_status(value):\n2|     if value > 10:\n…"},
  {"role": "assistant", "content": "<thinking>\n…\n</thinking>\ngraph TD\n    A[Start: check_status]\n    …\n<linemap>\nA: 1\n…\n</linemap>"}
]}
```

## Why the targets are trustworthy

The generator is **correct by construction** plus **hard-validated**, so the
labels don't need a teacher model:

1. **Shared state.** Every template emits the source code *and* its flowchart
   from one builder. A node's `<linemap>` line is the live line number returned
   when that statement was written — so injected comments, docstrings, blank
   lines and C++ `#include`s shift line numbers and the map stays correct
   automatically.
2. **Paraphrased labels.** Conditions become plain-English questions
   (`Index in bounds?`, not `i < len(nums)`), honoring the system prompt's strict
   constraint #4 — no operators, quotes, parentheses or brackets in any label.
3. **Mermaid validator.** Every example is parsed: balanced node brackets, legal
   label charset, no dangling edges, no markdown fences, well-formed `<thinking>`
   and `<linemap>` (line numbers in range, no unknown nodes).
4. **Real compilers.** Every Python target is `compile()`-checked in-process;
   JavaScript via `node --check`; C via `clang -fsyntax-only`; C++ via
   `clang++ -std=c++17 -fsyntax-only` (libc++ auto-located). Generation aborts on
   any invalid sample.

> **Design note — strict constraint vs. few-shot.** The provided system prompt's
> own few-shot example writes raw labels like `B{val > 10}` and `C[Return 'Active']`,
> which contradicts its strict constraint #4. The targets here follow the
> **strict constraint** (paraphrased, code-free labels) — the stronger, intended
> behavior. The system prompt is otherwise kept verbatim so training matches
> serving. If you'd rather the model mirror the looser few-shot style, regenerate
> after relaxing label paraphrasing in [dataset/templates.py](dataset/templates.py).

## Regenerate / scale the dataset

Pure standard-library Python; validation shells out to `node` and `clang`/`clang++`.

```bash
# self-test: every template × every language, exhaustive syntax check
.venv/bin/python dataset/generate.py --selftest

# generate (tweak size/seed/split freely)
.venv/bin/python dataset/generate.py --n 2400 --val-frac 0.08 --seed 7
.venv/bin/python dataset/generate.py --n 6000            # scale up
```

Language mix is controlled by `LANG_WEIGHT` and per-template support by the
`TEMPLATES` registry, both in [dataset/](dataset/).

## Fine-tune

`finetune.py` reads the JSONL, applies Qwen's chat template, and **masks the
prompt** so loss falls only on the assistant turn (the thinking + graph + linemap).

```bash
.venv/bin/python finetune.py --dry-run                       # token stats, no model download
.venv/bin/python finetune.py --4bit --epochs 3 --output-dir out/qwen-mermaid
```

Defaults: LoRA `r=16, α=32` on attention + MLP projections (works with the A3B
MoE blocks), cosine schedule, bf16. Use `--4bit` for QLoRA (CUDA + `bitsandbytes`).

### About the `UD-Q3_K_XL` target

`Qwen3-Coder-30B-A3B-Instruct-UD-Q3_K_XL` is an Unsloth **GGUF inference quant** —
you don't train it directly. Train a LoRA on the base weights (here, or via
Unsloth for a turnkey path), **merge** the adapter, then **convert + quantize**
to GGUF.

Important distinction: `UD-Q3_K_XL` is **Unsloth Dynamic** quantization, not the
same as llama.cpp's standard `Q3_K_XL`. Unsloth Dynamic GGUFs are model-specific
official Unsloth releases. The local `llama.cpp` exporter can create a standard
`Q3_K_XL` GGUF from your fine-tuned checkpoint, but it cannot reproduce Unsloth's
`UD-*` dynamic layer recipe for a custom fine-tune.

`finetune.py` can now do those post-training steps for you. When `--export-gguf`
is used, it looks for llama.cpp under `.venv/llama.cpp` by default. If the
checkout or quantizer is missing, it auto-clones and builds llama.cpp there.
You still need system CUDA tooling plus a C++ compiler; `requirements.txt`
installs the Python-side `cmake` command into the venv.

```bash
.venv/bin/python -m pip install -r requirements.txt
```

Then run training and export in one command. This will auto-clone/build
llama.cpp into `.venv/llama.cpp` if the GGUF tools are not already there:

```bash
.venv/bin/python finetune.py \
  --model Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --4bit \
  --epochs 3 \
  --batch-size 1 \
  --grad-accum 16 \
  --max-seq-len 2048 \
  --output-dir out/qwen-mermaid-lora \
  --merge-dir out/qwen-mermaid-merged \
  --merge-device-map cpu \
  --export-gguf \
  --gguf-out out/qwen3-coder-codeflow-Q3_K_XL.gguf \
  --gguf-quant Q3_K_XL \
  --delete-hf-cache-before-gguf \
  --delete-merged-after-gguf
```

### Colab Pro High-RAM A100 path

If you do not have enough local RAM/disk for the merge + GGUF export, use the
Colab notebook:

[notebooks/qwen3_coder_codeflow_gguf_to_hf_colab.ipynb](notebooks/qwen3_coder_codeflow_gguf_to_hf_colab.ipynb)

In Colab:

1. Open the notebook.
2. Select `Runtime -> Change runtime type -> GPU`, then choose an A100 with
   High-RAM.
3. Add a Colab Secret named `HF_TOKEN` with write access to the Hugging Face
   repo.
4. Run the cells top to bottom.

The notebook downloads this project from:

```text
https://huggingface.co/build-small-hackathon/codeflow-qwen-3-finetuning
```

Then it fine-tunes, merges, auto-builds llama.cpp inside `.venv/llama.cpp`,
deletes the local Hugging Face cache before GGUF conversion to reduce peak disk
use, exports:

```text
out/qwen3-coder-codeflow-Q3_K_XL.gguf
```

and uploads that GGUF back to:

```text
build-small-hackathon/codeflow-qwen-3-finetuning
```

The final artifact will be `out/qwen3-coder-codeflow-Q3_K_XL.gguf`. That file is
a fine-tuned standard `Q3_K_XL` GGUF, not `UD-Q3_K_XL`. The merge and conversion
steps need substantial CPU RAM and disk space because they reload the base model
in FP16 before quantizing. If GPU memory is tight during the merge, add
`--merge-device-map cpu`; it is slower but avoids placing the merged base model
on the RTX 5090.

If you need a true Unsloth Dynamic `UD-Q3_K_XL` file, the practical options are:

1. Use the official base GGUF from `unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF`.
2. Ask Unsloth/export through an Unsloth service that can apply their internal
   dynamic recipe to your merged fine-tune.
3. Keep the fine-tuned LoRA separate and apply it at inference time where your
   runtime supports GGUF + LoRA, accepting that the final artifact is not one
   merged `UD-*` GGUF.