Harness format-gate fix: Schemer 0.800 / int4 0.793 on the pooled eval
Browse files- README.md +9 -9
- manifest.json +5 -4
README.md
CHANGED
|
@@ -78,8 +78,8 @@ Three guarantees no generative extractor offers:
|
|
| 78 |
|
| 79 |
| File | Format | Size | Use |
|
| 80 |
|---|---|---:|---|
|
| 81 |
-
| `model-int8.pt` | int8 per-row | 218 MB | start here; full quality (0.
|
| 82 |
-
| `model-int4-awq.pt` | int4 AWQ | 111 MB | when download or RAM is tight (0.
|
| 83 |
| `model-fp32.pt` | fp32 | 850 MB | converting to other formats; not for deployment |
|
| 84 |
| `onnx/encoder_web4e4.onnx` | int4 ONNX | 111 MB | run in the browser or on Android |
|
| 85 |
| `onnx/*.onnx` | fp32 ONNX | 80 MB | decoding graphs; always ship with the encoder |
|
|
@@ -149,10 +149,10 @@ benched.
|
|
| 149 |
| Gemma 4 26B A4B (int4 AWQ) | 0.834 | 0.428 | 26.6B | 17.2 GB |
|
| 150 |
| Claude Haiku 4.5 (API) | 0.816 | 0.431 | n/a | API only |
|
| 151 |
| Qwen 3.5 9B | 0.812 | 0.434 | 9.1B | 9.1 GB |
|
|
|
|
|
|
|
| 152 |
| Ministral 3 8B | 0.790 | 0.430 | 8.0B | 17.8 GB |
|
| 153 |
| Gemma 4 E2B (official int4 QAT) | 0.780 | 0.427 | 5.1B | 8.3 GB |
|
| 154 |
-
| **Schemer** | **0.776** | **0.911** | 211M | 218 MB int8 |
|
| 155 |
-
| **Schemer (int4)** | **0.769** | 0.908 | 211M | 111 MB int4 |
|
| 156 |
| Qwen 3.5 0.8B | 0.651 | 0.351 | 0.87B | 1.75 GB |
|
| 157 |
| NuExtract-2.0-2B | 0.643 | 0.393 | 2.2B | 4.4 GB |
|
| 158 |
| GLiNER2-multi | 0.585 | 0.374 | 307M | 309 MB int8* |
|
|
@@ -168,13 +168,13 @@ within 0.004 of fp32).
|
|
| 168 |
|
| 169 |
| Model | Short | Long doc | Long + distractors | Register | Calendar | Factored | Nested |
|
| 170 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 171 |
-
| Gemma 4 26B A4B | **0.844** | **0.844** | 0.693 | 0.815 | 0.829 |
|
| 172 |
| Claude Haiku 4.5 | 0.838 | 0.759 | **0.762** | **0.831** | 0.802 | 0.817 | **1.000** |
|
| 173 |
| Qwen 3.5 9B | 0.828 | 0.807 | 0.717 | 0.791 | 0.774 | 0.891 | **1.000** |
|
|
|
|
|
|
|
| 174 |
| Ministral 3 8B | 0.818 | 0.811 | 0.711 | 0.720 | 0.796 | 0.790 | 0.998 |
|
| 175 |
| Gemma 4 E2B | 0.809 | 0.785 | 0.705 | 0.773 | 0.719 | 0.759 | **1.000** |
|
| 176 |
-
| **Schemer** | 0.759 | 0.708 | 0.685 | 0.829 | **0.946** | 0.705 | 0.988 |
|
| 177 |
-
| **Schemer (int4)** | 0.754 | 0.689 | 0.671 | 0.820 | 0.943 | 0.727 | 0.985 |
|
| 178 |
| Qwen 3.5 0.8B | 0.721 | 0.635 | 0.560 | 0.581 | 0.529 | 0.689 | 0.936 |
|
| 179 |
| NuExtract-2.0-2B | 0.763 | 0.704 | 0.542 | 0.648 | 0.496 | 0.805 | 0.000† |
|
| 180 |
| GLiNER2-multi | 0.683 | 0.650 | 0.388 | 0.662 | 0.468 | 0.783 | 0.000† |
|
|
@@ -196,7 +196,7 @@ Bold marks the best score in each row. Schemer's architecture shows plainly: it
|
|
| 196 |
| datetime | **0.769** | 0.752 | 0.641 | 0.652 |
|
| 197 |
| array | **0.766** | 0.750 | 0.731 | 0.686 |
|
| 198 |
| number | **0.750** | 0.743 | 0.670 | 0.679 |
|
| 199 |
-
| string | 0.
|
| 200 |
| label | 0.629 | 0.609 | **0.636** | 0.616 |
|
| 201 |
|
| 202 |
### Per language
|
|
@@ -221,7 +221,7 @@ distractors; ~600 records per language).
|
|
| 221 |
| Chinese | **0.683** | 0.640 | 0.408 |
|
| 222 |
| Polish | **0.670** | 0.609 | 0.597 |
|
| 223 |
|
| 224 |
-
Schemer scores 0.
|
| 225 |
parameters or an API. NuExtract-2.0-2B, the closest task-specific
|
| 226 |
extractor, scores 0.643 at ten times the parameters. Schemer wins all
|
| 227 |
13 languages against both the strongest sub-1B LLM and GLiNER2-multi,
|
|
|
|
| 78 |
|
| 79 |
| File | Format | Size | Use |
|
| 80 |
|---|---|---:|---|
|
| 81 |
+
| `model-int8.pt` | int8 per-row | 218 MB | start here; full quality (0.800 overall) |
|
| 82 |
+
| `model-int4-awq.pt` | int4 AWQ | 111 MB | when download or RAM is tight (0.793 overall) |
|
| 83 |
| `model-fp32.pt` | fp32 | 850 MB | converting to other formats; not for deployment |
|
| 84 |
| `onnx/encoder_web4e4.onnx` | int4 ONNX | 111 MB | run in the browser or on Android |
|
| 85 |
| `onnx/*.onnx` | fp32 ONNX | 80 MB | decoding graphs; always ship with the encoder |
|
|
|
|
| 149 |
| Gemma 4 26B A4B (int4 AWQ) | 0.834 | 0.428 | 26.6B | 17.2 GB |
|
| 150 |
| Claude Haiku 4.5 (API) | 0.816 | 0.431 | n/a | API only |
|
| 151 |
| Qwen 3.5 9B | 0.812 | 0.434 | 9.1B | 9.1 GB |
|
| 152 |
+
| **Schemer** | **0.800** | **0.911** | 211M | 218 MB int8 |
|
| 153 |
+
| **Schemer (int4)** | **0.793** | 0.908 | 211M | 111 MB int4 |
|
| 154 |
| Ministral 3 8B | 0.790 | 0.430 | 8.0B | 17.8 GB |
|
| 155 |
| Gemma 4 E2B (official int4 QAT) | 0.780 | 0.427 | 5.1B | 8.3 GB |
|
|
|
|
|
|
|
| 156 |
| Qwen 3.5 0.8B | 0.651 | 0.351 | 0.87B | 1.75 GB |
|
| 157 |
| NuExtract-2.0-2B | 0.643 | 0.393 | 2.2B | 4.4 GB |
|
| 158 |
| GLiNER2-multi | 0.585 | 0.374 | 307M | 309 MB int8* |
|
|
|
|
| 168 |
|
| 169 |
| Model | Short | Long doc | Long + distractors | Register | Calendar | Factored | Nested |
|
| 170 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 171 |
+
| Gemma 4 26B A4B | **0.844** | **0.844** | 0.693 | 0.815 | 0.829 | 0.954 | **1.000** |
|
| 172 |
| Claude Haiku 4.5 | 0.838 | 0.759 | **0.762** | **0.831** | 0.802 | 0.817 | **1.000** |
|
| 173 |
| Qwen 3.5 9B | 0.828 | 0.807 | 0.717 | 0.791 | 0.774 | 0.891 | **1.000** |
|
| 174 |
+
| **Schemer** | 0.759 | 0.708 | 0.685 | 0.829 | **0.946** | 0.946 | 0.988 |
|
| 175 |
+
| **Schemer (int4)** | 0.754 | 0.689 | 0.671 | 0.820 | 0.943 | **0.967** | 0.985 |
|
| 176 |
| Ministral 3 8B | 0.818 | 0.811 | 0.711 | 0.720 | 0.796 | 0.790 | 0.998 |
|
| 177 |
| Gemma 4 E2B | 0.809 | 0.785 | 0.705 | 0.773 | 0.719 | 0.759 | **1.000** |
|
|
|
|
|
|
|
| 178 |
| Qwen 3.5 0.8B | 0.721 | 0.635 | 0.560 | 0.581 | 0.529 | 0.689 | 0.936 |
|
| 179 |
| NuExtract-2.0-2B | 0.763 | 0.704 | 0.542 | 0.648 | 0.496 | 0.805 | 0.000† |
|
| 180 |
| GLiNER2-multi | 0.683 | 0.650 | 0.388 | 0.662 | 0.468 | 0.783 | 0.000† |
|
|
|
|
| 196 |
| datetime | **0.769** | 0.752 | 0.641 | 0.652 |
|
| 197 |
| array | **0.766** | 0.750 | 0.731 | 0.686 |
|
| 198 |
| number | **0.750** | 0.743 | 0.670 | 0.679 |
|
| 199 |
+
| string | 0.648 | 0.637 | **0.774** | 0.716 |
|
| 200 |
| label | 0.629 | 0.609 | **0.636** | 0.616 |
|
| 201 |
|
| 202 |
### Per language
|
|
|
|
| 221 |
| Chinese | **0.683** | 0.640 | 0.408 |
|
| 222 |
| Polish | **0.670** | 0.609 | 0.597 |
|
| 223 |
|
| 224 |
+
Schemer scores 0.800 overall; every model above it runs 9.1B to 26.6B
|
| 225 |
parameters or an API. NuExtract-2.0-2B, the closest task-specific
|
| 226 |
extractor, scores 0.643 at ten times the parameters. Schemer wins all
|
| 227 |
13 languages against both the strongest sub-1B LLM and GLiNER2-multi,
|
manifest.json
CHANGED
|
@@ -10,11 +10,12 @@
|
|
| 10 |
"review_register": 0.829,
|
| 11 |
"natural_temporal": 0.9323,
|
| 12 |
"nested_with_harness": 0.9875,
|
| 13 |
-
"pooled_usage_weighted_package": 0.
|
| 14 |
-
"pooled_usage_weighted_int8": 0.
|
| 15 |
-
"pooled_usage_weighted_int4_awq": 0.
|
| 16 |
"boolean_absence_pooled": 0.911,
|
| 17 |
-
"sob_truth_score": "pending"
|
|
|
|
| 18 |
},
|
| 19 |
"max_length": 1216,
|
| 20 |
"min_requirements": {
|
|
|
|
| 10 |
"review_register": 0.829,
|
| 11 |
"natural_temporal": 0.9323,
|
| 12 |
"nested_with_harness": 0.9875,
|
| 13 |
+
"pooled_usage_weighted_package": 0.8,
|
| 14 |
+
"pooled_usage_weighted_int8": 0.7998,
|
| 15 |
+
"pooled_usage_weighted_int4_awq": 0.7933,
|
| 16 |
"boolean_absence_pooled": 0.911,
|
| 17 |
+
"sob_truth_score": "pending",
|
| 18 |
+
"factored_domains_pooled": 0.9459
|
| 19 |
},
|
| 20 |
"max_length": 1216,
|
| 21 |
"min_requirements": {
|