| --- |
| base_model: Qwen/Qwen3.5-35B-A3B |
| library_name: peft |
| pipeline_tag: text-generation |
| tags: |
| - harbour |
| - fivewin |
| - fwh |
| - lora |
| - sft |
| - transformers |
| - trl |
| - unsloth |
| - code-generation |
| - xbase |
| - clipper |
| language: |
| - en |
| - es |
| license: apache-2.0 |
| --- |
| |
| # Harbour/FWH Coder — Qwen3.5-35B-A3B LoRA v2 |
|
|
| > **Training code:** https://github.com/FiveTechSoft/finetune |
|
|
| LoRA adapter fine-tuned on **5,004 compilable Harbour and FiveWin (FWH) examples** for code generation. Built on top of [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), a 35B Mixture-of-Experts model with 256 experts (8 active per token). |
|
|
| ## What's New in v2 |
|
|
| | | v1 | **v2** | |
| |---|---|---| |
| | **Dataset** | 996 entries (7 categories) | **5,004 entries** (8 categories) | |
| | **Training time** | 3h 43min | **~8h** | |
| | **Eval loss** | 0.5957 | **0.4790** (↓20%) | |
| | **Train loss** | 0.6456 | **0.4211** (↓35%) | |
| | **Format** | messages (chat) | instruction/output | |
| | **Learning rate** | 1e-4 | 8e-5 (conservative) | |
| | **Epochs** | 3 | 2 | |
|
|
| ### Key improvements |
|
|
| - **5x more training data** — expanded from 996 to 5,004 unique, compilable examples |
| - **Better loss convergence** — 35% lower train loss, 20% lower eval loss |
| - **More conservative training** — lower learning rate preserves base model capabilities |
| - **FiveWin (FWH) coverage** — added FiveWin GUI framework examples |
| - **Verified code** — all examples verified with Harbour v3.2.0dev compiler |
|
|
| ## Dataset |
|
|
| Training data sourced from the [Harbour](https://harbour.github.io/) project — an open-source Clipper-compatible compiler — and [FiveWin](https://fivewin.com/) (FWH) GUI framework. |
|
|
| ### Categories |
|
|
| | Category | Count | Description | |
| |---|---|---| |
| | contrib | 583 | Contribution libraries (network, database, graphics, security...) | |
| | rtl | 80 | Harbour Runtime Library | |
| | include | 59 | Header files with constants/macros | |
| | tests | 225 | Test programs | |
| | extras | 25 | Extra libraries | |
| | utils | 13 | Utility programs | |
| | fwh | ~500+ | FiveWin GUI framework examples | |
| | low-level C | 500+ | HB_FUNC C extension wrappers | |
| |
| ### Format |
| |
| ```json |
| { |
| "instruction": "Write a Harbour function that creates a 2D array...", |
| "input": "", |
| "system": "You are an expert Harbour programmer...", |
| "output": "FUNCTION CreateTable()\n LOCAL aTable := {}\n ...", |
| "task_type": "code_generation" |
| } |
| ``` |
| |
| ## Training Details |
| |
| ### Hardware |
| |
| - **Device:** NVIDIA GB10 Grace Blackwell Superchip (DGX Spark) |
| - **Architecture:** ARM aarch64 (10 NVIDIA Grace CPU cores + Blackwell GPU) |
| - **RAM:** 121 GB unified memory (CPU + GPU shared) |
| - **OS:** Ubuntu 24.04.4 LTS (aarch64) |
| - **Training time:** 7h 49min (564 steps, 2 epochs over 5,004 samples) |
| |
| ### Hyperparameters |
| |
| | Parameter | Value | |
| |---|---| |
| | Base model | Qwen3.5-35B-A3B (MoE, 256 experts) | |
| | Method | QLoRA (4-bit) | |
| | LoRA rank | 8 | |
| | LoRA alpha | 16 | |
| | LoRA targets | q/k/v/o/gate/up/down_proj | |
| | Epochs | 2 | |
| | Learning rate | 8e-5 | |
| | LR scheduler | cosine | |
| | Warmup ratio | 0.05 | |
| | Batch size | 1 (effective: 16 via grad accum) | |
| | Max seq length | 1024 | |
| | Optimizer | adamw_8bit | |
| |
| ### Framework |
| |
| - Unsloth 2026.6.8 |
| - Transformers 5.5.0 |
| - PEFT 0.19.1 |
| - PyTorch 2.12.1 |
| |
| ## How to Use |
| |
| ### With PEFT + Transformers |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| base_model = AutoModelForCausalLM.from_pretrained( |
| "Qwen/Qwen3.5-35B-A3B", |
| load_in_4bit=True, |
| device_map="auto", |
| ) |
| model = PeftModel.from_pretrained(base_model, "fivetech/Harbour") |
| tokenizer = AutoTokenizer.from_pretrained("fivetech/Harbour") |
| |
| prompt = "Write a Harbour function that splits a CSV string into an array." |
| messages = [ |
| {"role": "system", "content": "You are an expert Harbour programmer. Write compilable code."}, |
| {"role": "user", "content": prompt}, |
| ] |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer([text], return_tensors="pt").to(model.device) |
|
|
| output = model.generate(**inputs, max_new_tokens=1500, temperature=0.2) |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
| |
| ### With Ollama (merge + quantize first) |
| |
| ```bash |
| # Export to GGUF |
| python -m unsloth.save_pretrained_gguf model_output/ ./tokenizer/ q4_k_m |
| |
| # Then use with Ollama |
| ollama create harbour-coder -f Modelfile |
| ``` |
| |
| ## Evaluation |
| |
| Evaluated on 100 Harbour programming tests (Arrays, OOP, Functions, Database, File I/O, Control flow): |
| |
| - **Compilation pass rate:** TBD (running test battery) |
| - **Categories tested:** Arrays (48), OOP (22), Other (9), Functions (8), Database (7), File I/O (4), Control (2) |
| |
| ## License |
| |
| Apache 2.0 |
| |
| ## Model Card Contact |
| |
| fivetech — https://github.com/fivetechsoft/finetune |
| |
