Upload README.md with huggingface_hub

README.md

@@ -15,6 +15,10 @@ language:
 pipeline_tag: text-generation
 ---
 
+<p align="center">
+  <img src="knot-logo.png" width="110" alt="Knot AI">
+</p>
+
 # Knot AI - Commit Message Model
 
 **A task-specific model that reads a code diff and writes the commit message.**
@@ -46,7 +50,8 @@ model - feed it a diff, get a message. That focus is the point: a small, private
 
 - **Method:** QLoRA (4-bit), completion-only supervised fine-tuning (loss on the
   commit message only), 2 epochs, ~6h on a single **NVIDIA RTX 3060 (12 GB)**.
-- **Data:** 3,227 examples mined from real histories of permissively-licensed OSS -   **JavaScript/TypeScript** (vite, vitest, nx, pnpm, rollup, angular, changesets,
+- **Data:** 3,227 examples mined from real histories of permissively-licensed OSS -
+  **JavaScript/TypeScript** (vite, vitest, nx, pnpm, rollup, angular, changesets,
   husky, vitepress), **Python** (poetry, pydantic), **Rust** (starship, tauri),
   and **Go** (goreleaser). Type-balanced (fix 27% / feat 19% / refactor 11% /
   chore 8% / …).
@@ -54,21 +59,65 @@ model - feed it a diff, get a message. That focus is the point: a small, private
 
 ## Benchmark
 
-Evaluated on a held-out **gold set of 60 hand-reviewed cases** the model never saw,
-scored through the shipping GGUF path. "v3" is the prior release (trained on
-JavaScript/TypeScript only); "Knot AI" is this model, trained with added
-Python/Rust/Go diversity.
-
-| Metric | v3 (prior, JS/TS-only) | **Knot AI** |
-|---|---|---|
-| Type accuracy | 36.7% | **56.7%** |
-| Title word-overlap | 17.1% | **31.1%** |
-| Exact full message | 0% | **8.3%** |
-| Generation errors | 6 | **0** |
-
-**Cross-language generalization:** on a separate held-out set of **60 Python/Rust/Go
-commits** (languages this version added), Knot AI holds **56.7% type accuracy** - matching its JS/TS score, evidence the diversity generalized rather than
-memorized.
+Two held-out **gold sets of 60 hand-reviewed commits each** (the model never saw
+them), scored through the shipping GGUF path. Three models compared under one
+identical harness:
+
+- **Base Qwen** - Qwen2.5-Coder-7B, same prompt, no fine-tuning
+- **Knot AI v3** - the prior release (trained on JS/TS only)
+- **Knot AI (v4)** - this model (added Python / Rust / Go)
+
+![Commit-message quality vs baselines](bench-overall.png)
+
+### Full metrics (60 JS/TS gold)
+
+| Metric | Base Qwen | Knot AI v3 | **Knot AI (v4)** |
+|---|---|---|---|
+| Type accuracy | 0.0% | 40.0% | **55.0%** |
+| Title word-overlap | 0.4% | 18.3% | **30.3%** |
+| Exact title | 0.0% | 0.0% | **6.7%** |
+| Exact full message | 0.0% | 0.0% | **6.7%** |
+| Scope accuracy | 0.0% | 70.0% | **75.0%** |
+| Unparseable replies / 60 | 59 | 4 | **0** |
+
+The base model is not "bad at commits" - zero-shot with the same prompt it
+returns unstructured prose **59 times out of 60**, so it can't be wired into an
+automated pipeline at all. Fine-tuning's first job is to make the reply reliably
+parseable (59 → 4 → **0** errors); its second is to pick the right type.
+
+### By commit type - type accuracy (60 JS/TS gold)
+
+| Type | n | Base | v3 | **Knot AI (v4)** |
+|---|---|---|---|---|
+| chore | 6 | 0% | 83% | **100%** |
+| ci | 6 | 0% | 67% | **100%** |
+| feat | 6 | 0% | 33% | **83%** |
+| fix | 6 | 0% | 67% | **83%** |
+| test | 5 | 0% | 80% | **80%** |
+| docs | 6 | 0% | 83% | **67%** |
+| perf | 6 | 0% | 0% | **50%** |
+| build | 5 | 0% | 0% | **0%** |
+| refactor | 6 | 0% | 0% | **0%** |
+| style | 6 | 0% | 0% | **0%** |
+| revert | 2 | 0% | 0% | **0%** |
+
+Honest read: strong on the high-signal types (chore, ci, feat, fix, test); still
+developing on the inherently ambiguous ones - refactor vs feat, build vs chore,
+style, revert - which is exactly what v5 targets.
+
+### Generalization across languages - type accuracy (60 Rust / Go / Python gold)
+
+![Generalization across languages](bench-language.png)
+
+| Language | n | Base | v3 | **Knot AI (v4)** |
+|---|---|---|---|---|
+| Rust | 20 | 0% | 10% | **55%** |
+| Go | 20 | 0% | 55% | **45%** |
+| Python | 20 | 0% | 55% | **60%** |
+
+The added multilingual training pays off: Knot AI holds **~53% type accuracy**
+overall on this separate Rust/Go/Python set - close to its 55% JS/TS score - and
+lifts Rust from 10% → 55%, where the base model still scores 0% (unparseable).
 
 > Note: these are *single-reference* metrics - they compare to one human label,
 > though a commit has many valid messages and a change can honestly be `fix` *or*
@@ -84,6 +133,11 @@ memorized.
 
 ## Usage
 
+### Ollama
+```sh
+ollama run pointbreaklab/knot-ai "$(git diff --staged)"
+```
+
 ### llama.cpp (base + this LoRA)
 ```sh
 llama-server --model qwen2.5-coder-7b-instruct-q4_k_m.gguf \
@@ -101,21 +155,18 @@ model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
 model = PeftModel.from_pretrained(model, "pointbreaklab/knot-ai")
 ```
 
-### Ollama
-Coming with the Ollama release: `ollama run pointbreaklab/knot-ai`
-
 ## Limitations
 
-- Pattern completion over ~3k examples - it will miss on unusual diffs, and ~57%
-  type accuracy means it is wrong sometimes.
+- Pattern completion over ~3k examples - it will miss on unusual diffs, and ~55%
+  type accuracy means it is wrong sometimes (see the weaker types above).
 - Binary-only changes (images, PDFs) have no textual diff to read.
 - English commit conventions; not a general assistant.
 
 ## Roadmap
 
 This is **v1** (built on Knot AI's v4 adapter). A **v5** is planned - a larger,
-higher-quality corpus for better accuracy and closer matching. Knot AI improves
-in the open.
+higher-quality corpus for better accuracy and closer matching, with a focus on
+the ambiguous types (refactor / build / style / revert). Knot AI improves in the open.
 
 ## Citation / attribution