Hugging Face's logo

DominuZ
/
jq-coder-0.6B

Text Generation
GGUF
English
Portuguese
jq
json
text-to-jq
natural-language-to-code
code-generation
text-to-code
qwen3
ollama
llama.cpp
offline
privacy
structured-data
portuguese
Eval Results (legacy)
conversational
Model card Files
xet
 Community
DominuZ commited on
Commit
71b8215
·
verified ·
1 Parent(s): a8e9490

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +19 -13
README.md CHANGED
@@ -31,7 +31,7 @@ niche, [**jq-bench**](https://huggingface.co/datasets/DominuZ/jq-bench).
31
  With [llama.cpp](https://github.com/ggml-org/llama.cpp):
32
 
33
  ```bash
34
- llama-server -m jq-coder-v13-release-Q8_0.gguf --port 8091 -ngl 99
35
  ```
36
 
37
  ```bash
@@ -89,7 +89,7 @@ document correctly yields a different filter.
89
  **Reverse generation with ground truth by execution** — the training data was never
90
  written by an LLM guessing jq:
91
 
92
- 1. A grammar of **108 program families**, anchored in real-world usage (top-voted
93
  StackOverflow `[jq]` questions, the llm-jq / jiq / gpt-jq corpora, the official jq
94
  manual), samples candidate programs.
95
  2. Each program is **executed with real jq 1.8** against families of synthetic JSON
@@ -102,7 +102,7 @@ written by an LLM guessing jq:
102
  part). Teachers are locally served open-weights models (Apache 2.0, distillation
103
  permitted).
104
 
105
- Result: **27,480 verified pairs**, bilingual EN/PT-BR 50/50, full fine-tune (not LoRA)
106
  of Qwen3-0.6B-Base.
107
 
108
  ## Evaluation — jq-bench
@@ -115,18 +115,22 @@ The human slice of **jq-bench** is 30 real StackOverflow tasks, hand-curated,
115
  independent of the training grammar, never seen in training (CC BY-SA 4.0, per-item
116
  attribution).
117
 
118
- | Artifact (v13) | human slice, strict | human slice, task-solved¹ | in-distribution (400) |
119
  |---|---|---|---|
120
- | f16 | 9/30 | 10/30 | 394/400 |
121
- | **Q8_0 (recommended)** | **9/30** | **10/30** | **394/400** |
122
 
123
  ¹ *strict* = byte-identical to gold after canonicalization; *task-solved* additionally
124
  accepts equivalent output shapes (stream vs. array wrapper, etc.).
125
 
126
- An internal diagnostic of 204 realistic probes (execution-verified, authored
127
- independently of the training pipeline) measured the last data iteration at **52.5%**
128
- (up from 31.4%), with transformations under nested fields going from 12.5% to 54%. The
129
- human slice remains the canonical metric.
 
 
 
 
130
 
131
  ## Quantization
132
 
@@ -138,10 +142,12 @@ unharmed — we measured instead of assuming, and we don't publish what failed t
138
 
139
  ## Honest limitations
140
 
141
- - **Long compositions fail** (~2/3 of the human slice): `all(...)`, array subtraction
142
  (`. - [...]`), `to_entries` with object reconstruction + `tonumber`, compound merges
143
- with `del`. The model interpolates between neighboring compositions it saw in
144
- training; requests far from everything it saw come out wrong or hallucinated.
 
 
145
  - Aggregations under nested fields can come out as listings instead of sums.
146
  - The request must mention fields by their real names in the JSON; the JSON sample in
147
  the prompt is mandatory.
 
31
  With [llama.cpp](https://github.com/ggml-org/llama.cpp):
32
 
33
  ```bash
34
+ llama-server -m jq-coder-v14-release-Q8_0.gguf --port 8091 -ngl 99
35
  ```
36
 
37
  ```bash
 
89
  **Reverse generation with ground truth by execution** — the training data was never
90
  written by an LLM guessing jq:
91
 
92
+ 1. A grammar of **148 program families**, anchored in real-world usage (top-voted
93
  StackOverflow `[jq]` questions, the llm-jq / jiq / gpt-jq corpora, the official jq
94
  manual), samples candidate programs.
95
  2. Each program is **executed with real jq 1.8** against families of synthetic JSON
 
102
  part). Teachers are locally served open-weights models (Apache 2.0, distillation
103
  permitted).
104
 
105
+ Result: **36,879 verified pairs**, bilingual EN/PT-BR 50/50, full fine-tune (not LoRA)
106
  of Qwen3-0.6B-Base.
107
 
108
  ## Evaluation — jq-bench
 
115
  independent of the training grammar, never seen in training (CC BY-SA 4.0, per-item
116
  attribution).
117
 
118
+ | Artifact (v14) | human slice, strict | human slice, task-solved¹ | in-distribution (400) |
119
  |---|---|---|---|
120
+ | f16 | 10/30 | 11/30 | 394/400 |
121
+ | **Q8_0 (recommended)** | **10/30** | **11/30** | **394/400** |
122
 
123
  ¹ *strict* = byte-identical to gold after canonicalization; *task-solved* additionally
124
  accepts equivalent output shapes (stream vs. array wrapper, etc.).
125
 
126
+ Two internal diagnostics of ~200 realistic probes each (execution-verified, authored
127
+ independently of the training pipeline) track progress across data iterations: the v14
128
+ iteration scores **63.7%** on the older set (v13: 52.5%; v12: 31.4%) and **50.0%** on a
129
+ fresh, fully independent set (v13: 32.0%). The v14 data iteration targeted missing
130
+ *constructs* — `any`/`all`, filtered aggregations, `if-then-else` derived fields,
131
+ mid-level element paths (`.features[].properties.name`), string splitting — after the
132
+ v13 iteration had fixed the dominant envelope/root-shape failures. The human slice
133
+ remains the canonical metric.
134
 
135
  ## Quantization
136
 
 
142
 
143
  ## Honest limitations
144
 
145
+ - **Long compositions still fail** (~2/3 of the human slice): array subtraction
146
  (`. - [...]`), `to_entries` with object reconstruction + `tonumber`, compound merges
147
+ with `del`, recursive descent (`..`). The model interpolates between neighboring
148
+ compositions it saw in training; requests far from everything it saw come out wrong
149
+ or hallucinated. (v14 added `any`/`all`, filtered aggregations and conditional field
150
+ derivation — those now work in typical shapes but still break inside longer chains.)
151
  - Aggregations under nested fields can come out as listings instead of sums.
152
  - The request must mention fields by their real names in the JSON; the JSON sample in
153
  the prompt is mandatory.