v2: Harbour/FWH Coder - Qwen3.5-35B-A3B LoRA (5004 examples, eval_loss=0.479)

README.md

@@ -1,64 +1,159 @@
-# Harbour Fine-tuning Dataset
+---
+base_model: Qwen/Qwen3.5-35B-A3B
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- harbour
+- fivewin
+- fwh
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+- code-generation
+- xbase
+- clipper
+language:
+- en
+- es
+license: apache-2.0
+---
 
-## Overview
-This dataset contains 996 training entries extracted from:
-- 1037 Harbour PRG (.prg) source files
-- 143 Harbour Header (.ch) files
+# Harbour/FWH Coder — Qwen3.5-35B-A3B LoRA v2
 
-177 PRG files and 7 CH files were skipped due to quality issues.
+LoRA adapter fine-tuned on **5,004 compilable Harbour and FiveWin (FWH) examples** for code generation. Built on top of [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), a 35B Mixture-of-Experts model with 256 experts (8 active per token).
 
-## Dataset Format
-The dataset is provided in JSONL format with the following structure:
+## What's New in v2
+
+| | v1 | **v2** |
+|---|---|---|
+| **Dataset** | 996 entries (7 categories) | **5,004 entries** (8 categories) |
+| **Training time** | 3h 43min | **~8h** |
+| **Eval loss** | 0.5957 | **0.4790** (↓20%) |
+| **Train loss** | 0.6456 | **0.4211** (↓35%) |
+| **Format** | messages (chat) | instruction/output |
+| **Learning rate** | 1e-4 | 8e-5 (conservative) |
+| **Epochs** | 3 | 2 |
+
+### Key improvements
+
+- **5x more training data** — expanded from 996 to 5,004 unique, compilable examples
+- **Better loss convergence** — 35% lower train loss, 20% lower eval loss
+- **More conservative training** — lower learning rate preserves base model capabilities
+- **FiveWin (FWH) coverage** — added FiveWin GUI framework examples
+- **Verified code** — all examples verified with Harbour v3.2.0dev compiler
+
+## Dataset
+
+Training data sourced from the [Harbour](https://harbour.github.io/) project — an open-source Clipper-compatible compiler — and [FiveWin](https://fivewin.com/) (FWH) GUI framework.
+
+### Categories
+
+| Category | Count | Description |
+|---|---|---|
+| contrib | 583 | Contribution libraries (network, database, graphics, security...) |
+| rtl | 80 | Harbour Runtime Library |
+| include | 59 | Header files with constants/macros |
+| tests | 225 | Test programs |
+| extras | 25 | Extra libraries |
+| utils | 13 | Utility programs |
+| fwh | ~500+ | FiveWin GUI framework examples |
+| low-level C | 500+ | HB_FUNC C extension wrappers |
+
+### Format
 
-### Instruction Format (harbour_train.jsonl / harbour_val.jsonl)
 ```json
-{"instruction": "...", "input": "", "output": "..."}
+{
+  "instruction": "Write a Harbour function that creates a 2D array...",
+  "input": "",
+  "system": "You are an expert Harbour programmer...",
+  "output": "FUNCTION CreateTable()\n   LOCAL aTable := {}\n   ...",
+  "task_type": "code_generation"
+}
 ```
 
-### Full Dataset (harbour_dataset_full.jsonl)
-```json
-{"instruction": "...", "input": "", "output": "...", "metadata": {"file_path": "...", "language": "harbour", "category": "...", "subcategory": "..."}}
+## Training Details
+
+### Hardware
+
+- **Device:** NVIDIA GB10 (Grace Blackwell Superchip)
+- **RAM:** 121 GB unified memory
+- **Training time:** ~8 hours
+
+### Hyperparameters
+
+| Parameter | Value |
+|---|---|
+| Base model | Qwen3.5-35B-A3B (MoE, 256 experts) |
+| Method | QLoRA (4-bit) |
+| LoRA rank | 8 |
+| LoRA alpha | 16 |
+| LoRA targets | q/k/v/o/gate/up/down_proj |
+| Epochs | 2 |
+| Learning rate | 8e-5 |
+| LR scheduler | cosine |
+| Warmup ratio | 0.05 |
+| Batch size | 1 (effective: 16 via grad accum) |
+| Max seq length | 1024 |
+| Optimizer | adamw_8bit |
+
+### Framework
+
+- Unsloth 2026.6.8
+- Transformers 5.5.0
+- PEFT 0.19.1
+- PyTorch 2.12.1
+
+## How to Use
+
+### With PEFT + Transformers
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+base_model = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen3.5-35B-A3B",
+    load_in_4bit=True,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(base_model, "fivetech/Harbour")
+tokenizer = AutoTokenizer.from_pretrained("fivetech/Harbour")
+
+prompt = "Write a Harbour function that splits a CSV string into an array."
+messages = [
+    {"role": "system", "content": "You are an expert Harbour programmer. Write compilable code."},
+    {"role": "user", "content": prompt},
+]
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+output = model.generate(**inputs, max_new_tokens=1500, temperature=0.2)
+print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
 
-## Categories
-- **include**: Header files with constants/macros (59 files)
-- **rtl**: Harbour Runtime Library (80 files)
-- **contrib**: Contribution libraries (583 files)
-- **tests**: Test programs (225 files)
-- **utils**: Utility programs (13 files)
-- **extras**: Extra libraries (25 files)
-
-## Cleaning Applied
-- Copyright/license headers removed
-- Disabled code blocks (#if 0) removed
-- Excessive trailing comments removed
-- Excessive blank lines removed
-- Files without actual code filtered out
-- Incomplete code (missing ENDCLASS, etc.) filtered out
-
-## Usage for Fine-tuning
+### With Ollama (merge + quantize first)
+
 ```bash
-# Using Ollama with Modelfile
-FROM qwen2.5-coder:14b
+# Export to GGUF
+python -m unsloth.save_pretrained_gguf model_output/ ./tokenizer/ q4_k_m
 
-# Training command
+# Then use with Ollama
 ollama create harbour-coder -f Modelfile
-
-# Or use with other training frameworks
-# The JSONL format is compatible with:
-# - OpenAI fine-tuning API
-# - Hugging Face transformers
-# - Axolotl
-# - LLaMA-Factory
 ```
 
-## File Structure
-- `harbour_train.jsonl` - Training set (896 entries)
-- `harbour_val.jsonl` - Validation set (100 entries)
-- `harbour_dataset_full.jsonl` - Full dataset with metadata
-- `dataset_stats.json` - Dataset statistics
-- `generate_dataset.py` - This script
+## Evaluation
+
+Evaluated on 100 Harbour programming tests (Arrays, OOP, Functions, Database, File I/O, Control flow):
+
+- **Compilation pass rate:** TBD (running test battery)
+- **Categories tested:** Arrays (48), OOP (22), Other (9), Functions (8), Database (7), File I/O (4), Control (2)
+
+## License
+
+Apache 2.0
+
+## Model Card Contact
 
-## Source
-The source files are from the Harbour project (https://harbour.github.io/),
-an open-source Clipper-compatible compiler.
+fivetech — https://github.com/fivetechsoft/finetune