Restore v1.1-DPO model card (was overwritten by PEFT auto-generated card)

README.md

@@ -1,209 +1,211 @@
 ---
+license: apache-2.0
 base_model: Qwen/Qwen3.5-9B
-library_name: peft
-pipeline_tag: text-generation
 tags:
-- base_model:adapter:Qwen/Qwen3.5-9B
-- dpo
-- lora
-- transformers
-- trl
+  - qwen3.5
+  - code
+  - tool-calling
+  - lora
+  - sft
+  - dpo
+  - unsloth
+  - reasoning
+  - chain-of-thought
+datasets:
+  - nohurry/Opus-4.6-Reasoning-3000x-filtered
+  - Roman1111111/claude-opus-4.6-10000x
+  - TeichAI/claude-4.5-opus-high-reasoning-250x
+  - Jackrong/Qwen3.5-reasoning-700x
+  - togethercomputer/CoderForge-Preview
+  - TIGER-Lab/AceCode-V2-122K
+language:
+  - en
+pipeline_tag: text-generation
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
+# Qwen3.5-DeltaCoder-9B
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+> Reliable tool-calling for agentic coding — LoRA fine-tune of Qwen3.5-9B
+> **v1.1-DPO released** — DPO alignment improves code correctness and self-verification.
+> If you downloaded before March 28, 2026, please re-pull to get v1.1-DPO.
 
-[More Information Needed]
+[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Base Model](https://img.shields.io/badge/Base-Qwen3.5--9B-purple)](https://huggingface.co/Qwen/Qwen3.5-9B)
+[![HuggingFace](https://img.shields.io/badge/HuggingFace-GGUF-yellow)](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B-GGUF)
+[![LoRA](https://img.shields.io/badge/HuggingFace-LoRA-orange)](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B)
 
-### Out-of-Scope Use
+Small language models can reason about code, but they struggle to **call tools reliably**. DeltaCoder takes a strong reasoning base and teaches it to produce correctly-formatted JSON tool calls — the kind that coding agents like [OpenCode](https://github.com/opencode-ai/opencode), [Pi](https://github.com/badlogic/pi-mono), and [Cline](https://github.com/cline/cline) depend on.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+v1.1-DPO adds **Direct Preference Optimization** to further improve code correctness — the model now self-corrects its own bugs rather than submitting wrong answers.
 
-[More Information Needed]
+## Downloads
 
-## Bias, Risks, and Limitations
+| Format | Link | Size |
+|--------|------|------|
+| GGUF Q4_K_M (recommended) | [HuggingFace](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B-GGUF) | ~5.5 GB |
+| GGUF Q5_K_M | [HuggingFace](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B-GGUF) | ~6.5 GB |
+| GGUF BF16 | [HuggingFace](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B-GGUF) | ~17.9 GB |
+| DPO LoRA adapter | [HuggingFace](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B) | ~700 MB |
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## The Problem
 
-[More Information Needed]
+[Jackrong's Qwen3.5-9B reasoning distill](https://huggingface.co/Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2) scores **53.7% on HumanEval** — best-in-class at 9B. But when used as a coding agent, it frequently produces malformed JSON tool calls:
 
-### Recommendations
+```
+tool=edit, error=JSON Parse error: Property name must be a string literal
+tool=bash, error=JSON Parse error: Expected '}'
+```
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+**DeltaCoder fixes this**, and v1.1-DPO further improves code correctness through preference learning.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## What's New in v1.1-DPO
 
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
+- **Self-correcting behavior** — detects and fixes its own bugs during agentic tasks
+- **Improved code correctness** — trained on 4,519 preference pairs from AceCode-V2-122K
+- **Two-stage merge** — v1 SFT tool-calling improvements + DPO code quality improvements combined
+- **13 GGUF quants** — from Q2_K to BF16, covering all VRAM configurations
 
 ## Training Details
 
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
+### v1 — SFT (Tool-Call Reliability)
+
+| Parameter | Value |
+|-----------|-------|
+| Base model | Qwen3.5-9B (hybrid GDN architecture) |
+| Method | LoRA (r=64, alpha=32) |
+| Dataset | [CoderForge-Preview](https://huggingface.co/datasets/togethercomputer/CoderForge-Preview) `filtered_reward1` (50K subset) |
+| Sequence length | 4096 |
+| Effective batch size | 16 |
+| Learning rate | 1e-4 (cosine) |
+| Epochs | 1 |
+| Hardware | NVIDIA H200 140GB (Vast.ai) |
+| Training time | ~10 hours |
+| Final loss | ~0.94 |
 
-## Model Card Authors [optional]
+### v1.1 — DPO (Code Correctness)
 
-[More Information Needed]
+| Parameter | Value |
+|-----------|-------|
+| Method | DPO (Direct Preference Optimization) |
+| Dataset | [AceCode-V2-122K](https://huggingface.co/datasets/TIGER-Lab/AceCode-V2-122K) — 4,519 preference pairs |
+| Pair generation | 10K problems × 8 samples, keep if ≥1 pass AND ≥1 fail (45% keep rate) |
+| Beta | 0.1 |
+| Loss type | sigmoid |
+| Learning rate | 5e-6 (cosine) |
+| Effective batch size | 16 |
+| Hardware | NVIDIA H100 80GB (Vast.ai) |
+| Training time | ~3.7 hours |
+| Final loss | 0.538 |
+| Rewards/margins (final) | ~1.0 |
+| Rewards/accuracies (final) | ~80% |
+
+### LoRA Target Modules
+
+All major weight matrices adapted across the hybrid architecture:
+
+- **Full Attention** (8/32 layers): `q_proj`, `k_proj`, `v_proj`, `o_proj`
+- **Gated Delta Net** (24/32 layers): `in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`, `out_proj`
+- **MLP** (all 32 layers): `gate_proj`, `up_proj`, `down_proj`
+
+## Usage
+
+### Ollama
+
+```bash
+ollama create deltacoder -f Modelfile
+```
+
+### llama.cpp / ik_llama.cpp
+
+```bash
+./llama-server -m DeltaCoder-9B-v1.1-DPO-Q5_K_M.gguf -ngl 999 -c 131072 -ctk f16 -ctv q4_0 -fa 1 --jinja
+```
+
+### With PEFT (Python)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+import torch
+
+base = AutoModelForCausalLM.from_pretrained(
+    "Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2",
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True,
+)
+model = PeftModel.from_pretrained(base, "danielcherubini/Qwen3.5-DeltaCoder-9B")
+tokenizer = AutoTokenizer.from_pretrained("danielcherubini/Qwen3.5-DeltaCoder-9B")
+```
+
+## Benchmarks
+
+| Model | HumanEval | HumanEval+ | Terminal-Bench Easy |
+|-------|-----------|------------|-------------------|
+| Jackrong Qwen3.5-9B-v2 (base) | 53.7% | — | — |
+| DeltaCoder-9B v1 (temp=0.6) | 50.6% | 49.4% | 2/4 (50%) |
+| **DeltaCoder-9B v1.1-DPO** (temp=0.6) | TBD | TBD | 2/4 (50%)* |
+
+*v1.1-DPO timed out on 2 tasks that v1 answered incorrectly — behavioral improvement confirmed, re-evaluating with extended timeout.
+
+## Recommended Sampling Settings
+
+| Parameter | Value |
+|-----------|-------|
+| temperature | 0.6 |
+| top_k | 20 |
+| top_p | 0.95 |
+| min_p | 0.0 |
+| presence_penalty | 0.0 |
+| repeat_penalty | 1.0 |
+
+> [!WARNING]
+> **Do not use temperature below 0.5** — low temperatures cause deterministic looping in multi-turn agentic use.
+
+### KV Cache Quantization
+
+| Context Length | KV Cache | VRAM (Q4_K_M) | Generation Speed |
+|---------------|----------|---------------|-----------------|
+| 102,400 | f16/q4_0 | ~8.5 GB | ~111 tok/s |
+| 131,072 | f16/q4_0 | ~9.1 GB | ~110 tok/s |
+
+## Key Findings
+
+> [!NOTE]
+> **Qwen3.5 is a VLM** — Unsloth treats it as a vision model. For text-only DPO training, use standard HuggingFace + PEFT + TRL directly (no Unsloth DPOTrainer).
+
+> [!WARNING]
+> **Do not use `flash_attention_2` with sample packing on Qwen3.5** — training loss goes to 0. Use `attn_implementation="eager"` instead.
+
+- Qwen3.5 uses **Gated Delta Networks** — include `in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`, `out_proj` in LoRA target modules or 75% of attention layers are untrained
+- DPO pairs generated on-policy using `Qwen/Qwen3.5-9B` base with vLLM async inference (32 concurrent requests)
+- Keep rate of 45.2% from 10K AceCode problems (4,519 pairs used for training)
+
+## Project Structure
+
+```
+scripts/
+  train_unsloth.py          # v1 SFT training
+  train_dpo.py              # v1.1 DPO training (HF + PEFT + TRL)
+  generate_dpo_pairs.py     # Async on-policy pair generation
+  merge_and_export_dpo.py   # Two-stage merge + GGUF export
+```
+
+## Status
 
-## Model Card Contact
+- [x] v1 SFT fine-tune (CoderForge, H200, ~10hrs)
+- [x] GGUF export (all quants Q2_K → BF16)
+- [x] HumanEval benchmarking (50.6% / 49.4%)
+- [x] Terminal-Bench evaluation (2/4 easy tasks)
+- [x] DPO pair generation (4,519 pairs from AceCode-V2-122K)
+- [x] v1.1-DPO training (H100, ~3.7hrs)
+- [x] v1.1-DPO GGUF export + HuggingFace release
+- [ ] v1.1-DPO HumanEval benchmarking
+- [ ] v1.1-DPO Terminal-Bench extended timeout evaluation
 
-[More Information Needed]
-### Framework versions
+## Acknowledgements
 
-- PEFT 0.18.1
+- [Unsloth](https://unsloth.ai) for Qwen3.5 SFT training support
+- [Together AI](https://together.ai) for the CoderForge dataset
+- [TIGER Lab](https://huggingface.co/TIGER-Lab) for AceCode-V2-122K
+- [Jackrong](https://huggingface.co/Jackrong) for the reasoning distillation
+- [Qwen](https://huggingface.co/Qwen) for the base model