1337Hero commited on
Commit
56ad773
Β·
verified Β·
1 Parent(s): 499b3c4

Update README.md

Browse files

Updated to really explain what this is, what I was trying to do, and where I failed.

Files changed (1) hide show
  1. README.md +83 -16
README.md CHANGED
@@ -13,31 +13,98 @@ tags:
13
  - qwen3moe
14
  - quantized
15
  - code
 
 
 
16
  ---
17
 
18
- # qwen3-coder-30b-a3b-codemonkey-GGUF
19
 
20
- GGUF exports of the `qwen3-coder-30b-a3b-codemonkey` fine-tune.
21
 
22
- These files were produced by merging the LoRA adapter into the base weights,
23
- exporting a BF16 GGUF, and quantizing from that BF16 source.
24
 
25
- ## Files
26
 
27
- - `qwen3-coder-30b-a3b-codemonkey.Q4_K_M.gguf`: recommended release artifact
28
- - `qwen3-coder-30b-a3b-codemonkey.Q8_0.gguf`: larger reference GGUF
29
- - `qwen3-coder-30b-a3b-codemonkey.BF16.gguf`: optional full-size BF16 GGUF source
 
30
 
31
- ## Use with llama.cpp
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
  ```bash
34
  llama-cli \
35
  --model qwen3-coder-30b-a3b-codemonkey.Q4_K_M.gguf \
36
  --jinja \
37
- -p "Write a Python script that tails a log file and highlights errors."
38
- ```
39
-
40
- ## Notes
41
-
42
- - The `Q4_K_M` file in this repo was quantized from the merged BF16 GGUF, not requantized from Q8.
43
- - If you want the adapter-only release, use the companion adapter repo: `1337Hero/qwen3-coder-30b-a3b-codemonkey`
 
13
  - qwen3moe
14
  - quantized
15
  - code
16
+ - frontend
17
+ - react
18
+ - fine-tuned
19
  ---
20
 
21
+ # Qwen3-CodeMonkey v1
22
 
23
+ A fine-tuned version of [qwen3-coder-30b-a3b](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct), optimized for frontend development patterns and DHH-inspired simplicity.
24
 
25
+ **TL;DR:** I made Qwen ***faster* but dumber**. Q4_K_M hits 137 tps (63% faster gen than base) but drops 10 accuracy points. Your mileage may vary.
 
26
 
27
+ ## 🎯 Motivation
28
 
29
+ I wanted a model that *thinks* like I do about frontend code:
30
+ - **Simplicity over cleverness** β€” boring tech that works beats clever abstractions
31
+ - **Convention over configuration** β€” follow patterns, don't invent new ones
32
+ - **DHH/Ryan Florence-inspired** β€” majestic monolith vibes for the frontend stack
33
 
34
+ I work heavily in React/Preact + TanStack Query + Tailwind, and needed a model that could:
35
+ - Write clean, maintainable component code
36
+ - Avoid TypeScript ceremony and over-abstraction
37
+ - Default to proven patterns (no "clever" state management hacks)
38
+ - Generate code that a junior dev can understand 6 months later
39
+
40
+ ## πŸ“Š The Numbers (Honest Edition)
41
+
42
+ Benchmarked with [scaffold-bench](https://github.com/1337hero/scaffold-bench)
43
+ My own eval suite that tests real agentic coding work, not trivia or math puzzles. It throws 23 scenarios at the model (surgical edits, audits, scope discipline, verify-and-repair, implementation tasks) with actual tool use and real codebases.
44
+
45
+ **Results (mean of 2 runs, 50-task subset):**
46
+
47
+ | Model | Score | Gen TPS | Prompt TPS | Notes |
48
+ |-------|-------|---------|------------|-------|
49
+ | **qwen3-code-monkey (Q4_K_M)** | **66.0%** (33/50) | **137.3** | **1219** | Fastest, but -10pts accuracy |
50
+ | qwen3-coder-next (base) | 76.0% (38/50) | 84.4 | 953 | Baseline |
51
+ | qwen3-code-monkey-q8 | 33.0% (16.5/50) | 109.6 | 1444 | One run collapsed early β€” needs investigation |
52
+
53
+ **Translation:** Q4 is **63% faster at generation** but lost 10 accuracy points. If you're latency bound and can tolerate the trade-off, this might be your jam. If you need max accuracy, stick with base.
54
+
55
+ **What scaffold-bench tests:**
56
+ - **No LLM judge** - deterministic code-driven checks only
57
+ - **Real tool use** - read, ls, grep, glob, edit, write, bash
58
+ - **Real scenarios** - surgical edits, audits, scope discipline, implementation tasks against actual codebases
59
+ - Categories include: surgical edit, audit, scope discipline, read only analysis, verify and repair, implementation, responsiveness, long context
60
+
61
+ **Sample categories where code-monkey struggled vs base:**
62
+ - **verify-and-repair** - iterating through failing tests to green
63
+ - **implementation** - multi-file spec-to-code against existing stacks (wanted to rewrite everything my way instead of following the existing patterns)
64
+ - **scope-discipline** - "make this one change and nothing else" (my training data emphasized refactoring, which backfired)
65
+
66
+ **Q8 caveat:** One of the Q8 runs cratered to 8/50 after 64 requests - might be a harness issue, overfit collapse under tighter quantization, or early model failure. Take those numbers with salt until I re-run it.
67
+
68
+ **Full benchmark details:** Check out [scaffold-bench](https://github.com/1337hero/scaffold-bench) if you want to run your own evals. All results/*.json files from my runs are in the codemonkey repo.
69
+
70
+ πŸ› οΈ Training Process (And Where I Screwed Up)
71
+
72
+ - Dataset: ~20,000 Q&A pairs pulled from my frontend philosophy docs (DHH's "majestic monolith"
73
+ stuff + Ryan Florence's React patterns)
74
+ - Fine-tuning: LoRA adapter via Unsloth on Qwen3-Coder-30B-A3B-Instruct
75
+ - What I did wrong: Built the whole dataset as plain [question] β†’ [answer] pairs. No reasoning
76
+ shown, no "wait, should I even do this?" step in the middle. Turns out the model learned the shape
77
+ of my answers without learning how I got there.
78
+ - How it showed up: v1 benched at 32–34/50 vs the base model's 37–40/50. Worse than what I started
79
+ with. The failures were the giveaway β€” it would edit files when I asked read-only questions, jump
80
+ straight to edits without reading anything first, skip searching before acting. Not a knowledge
81
+ problem. A "doesn't stop to think" problem.
82
+ - The speed trap that fooled me: 138 tok/s vs base 84 tok/s looked like a win at first. It wasn't.
83
+ It was faster because it was thinking less. Trinity-mini sits at the end of that road - blazing
84
+ fast, fails everything, useless.
85
+ - What I'm doing for v2: Q&A alone teaches outputs, not process. Need explicit <think>...</think>
86
+ traces in the dataset - short, 2–4 sentences, enough to encode the taboos ("read-only means
87
+ read-only", "search before you edit", "don't re-add a feature that already exists"). Response-only
88
+ supervision. Less distilled CoT cosplay, more structural alignment.
89
+
90
+ **Lessons learned:**
91
+ 1. Unsloth works great once you get past the docs
92
+ 2. 20k rows might've been overkill for the domain specificity I wanted
93
+ 3. Need to balance domain adaptation with general capability retention
94
+ 4. Benchmarking early would've saved me a re-train (and $47)
95
+
96
+ ## πŸ“¦ Files
97
+
98
+ - `qwen3-coder-30b-a3b-codemonkey.Q4_K_M.gguf` β€” **recommended** release artifact (fastest)
99
+ - `qwen3-coder-30b-a3b-codemonkey.Q8_0.gguf` β€” larger reference GGUF (see caveat above)
100
+ - `qwen3-coder-30b-a3b-codemonkey.BF16.gguf` β€” optional full-size BF16 source
101
+
102
+ All quantized from the merged BF16 GGUF (not requantized from Q8).
103
+
104
+ ## πŸš€ Usage with llama.cpp
105
 
106
  ```bash
107
  llama-cli \
108
  --model qwen3-coder-30b-a3b-codemonkey.Q4_K_M.gguf \
109
  --jinja \
110
+ -p "Write a React component that fetches user data with TanStack Query and displays it in a card."