pveugen commited on
Commit
a580ca0
·
verified ·
1 Parent(s): 3a40f8a

Harness format-gate fix: Schemer 0.800 / int4 0.793 on the pooled eval

Browse files
Files changed (2) hide show
  1. README.md +9 -9
  2. manifest.json +5 -4
README.md CHANGED
@@ -78,8 +78,8 @@ Three guarantees no generative extractor offers:
78
 
79
  | File | Format | Size | Use |
80
  |---|---|---:|---|
81
- | `model-int8.pt` | int8 per-row | 218 MB | start here; full quality (0.776 overall) |
82
- | `model-int4-awq.pt` | int4 AWQ | 111 MB | when download or RAM is tight (0.769 overall) |
83
  | `model-fp32.pt` | fp32 | 850 MB | converting to other formats; not for deployment |
84
  | `onnx/encoder_web4e4.onnx` | int4 ONNX | 111 MB | run in the browser or on Android |
85
  | `onnx/*.onnx` | fp32 ONNX | 80 MB | decoding graphs; always ship with the encoder |
@@ -149,10 +149,10 @@ benched.
149
  | Gemma 4 26B A4B (int4 AWQ) | 0.834 | 0.428 | 26.6B | 17.2 GB |
150
  | Claude Haiku 4.5 (API) | 0.816 | 0.431 | n/a | API only |
151
  | Qwen 3.5 9B | 0.812 | 0.434 | 9.1B | 9.1 GB |
 
 
152
  | Ministral 3 8B | 0.790 | 0.430 | 8.0B | 17.8 GB |
153
  | Gemma 4 E2B (official int4 QAT) | 0.780 | 0.427 | 5.1B | 8.3 GB |
154
- | **Schemer** | **0.776** | **0.911** | 211M | 218 MB int8 |
155
- | **Schemer (int4)** | **0.769** | 0.908 | 211M | 111 MB int4 |
156
  | Qwen 3.5 0.8B | 0.651 | 0.351 | 0.87B | 1.75 GB |
157
  | NuExtract-2.0-2B | 0.643 | 0.393 | 2.2B | 4.4 GB |
158
  | GLiNER2-multi | 0.585 | 0.374 | 307M | 309 MB int8* |
@@ -168,13 +168,13 @@ within 0.004 of fp32).
168
 
169
  | Model | Short | Long doc | Long + distractors | Register | Calendar | Factored | Nested |
170
  |---|---:|---:|---:|---:|---:|---:|---:|
171
- | Gemma 4 26B A4B | **0.844** | **0.844** | 0.693 | 0.815 | 0.829 | **0.954** | **1.000** |
172
  | Claude Haiku 4.5 | 0.838 | 0.759 | **0.762** | **0.831** | 0.802 | 0.817 | **1.000** |
173
  | Qwen 3.5 9B | 0.828 | 0.807 | 0.717 | 0.791 | 0.774 | 0.891 | **1.000** |
 
 
174
  | Ministral 3 8B | 0.818 | 0.811 | 0.711 | 0.720 | 0.796 | 0.790 | 0.998 |
175
  | Gemma 4 E2B | 0.809 | 0.785 | 0.705 | 0.773 | 0.719 | 0.759 | **1.000** |
176
- | **Schemer** | 0.759 | 0.708 | 0.685 | 0.829 | **0.946** | 0.705 | 0.988 |
177
- | **Schemer (int4)** | 0.754 | 0.689 | 0.671 | 0.820 | 0.943 | 0.727 | 0.985 |
178
  | Qwen 3.5 0.8B | 0.721 | 0.635 | 0.560 | 0.581 | 0.529 | 0.689 | 0.936 |
179
  | NuExtract-2.0-2B | 0.763 | 0.704 | 0.542 | 0.648 | 0.496 | 0.805 | 0.000† |
180
  | GLiNER2-multi | 0.683 | 0.650 | 0.388 | 0.662 | 0.468 | 0.783 | 0.000† |
@@ -196,7 +196,7 @@ Bold marks the best score in each row. Schemer's architecture shows plainly: it
196
  | datetime | **0.769** | 0.752 | 0.641 | 0.652 |
197
  | array | **0.766** | 0.750 | 0.731 | 0.686 |
198
  | number | **0.750** | 0.743 | 0.670 | 0.679 |
199
- | string | 0.641 | 0.631 | **0.774** | 0.716 |
200
  | label | 0.629 | 0.609 | **0.636** | 0.616 |
201
 
202
  ### Per language
@@ -221,7 +221,7 @@ distractors; ~600 records per language).
221
  | Chinese | **0.683** | 0.640 | 0.408 |
222
  | Polish | **0.670** | 0.609 | 0.597 |
223
 
224
- Schemer scores 0.776 overall; every model above it runs 5.1B to 26.6B
225
  parameters or an API. NuExtract-2.0-2B, the closest task-specific
226
  extractor, scores 0.643 at ten times the parameters. Schemer wins all
227
  13 languages against both the strongest sub-1B LLM and GLiNER2-multi,
 
78
 
79
  | File | Format | Size | Use |
80
  |---|---|---:|---|
81
+ | `model-int8.pt` | int8 per-row | 218 MB | start here; full quality (0.800 overall) |
82
+ | `model-int4-awq.pt` | int4 AWQ | 111 MB | when download or RAM is tight (0.793 overall) |
83
  | `model-fp32.pt` | fp32 | 850 MB | converting to other formats; not for deployment |
84
  | `onnx/encoder_web4e4.onnx` | int4 ONNX | 111 MB | run in the browser or on Android |
85
  | `onnx/*.onnx` | fp32 ONNX | 80 MB | decoding graphs; always ship with the encoder |
 
149
  | Gemma 4 26B A4B (int4 AWQ) | 0.834 | 0.428 | 26.6B | 17.2 GB |
150
  | Claude Haiku 4.5 (API) | 0.816 | 0.431 | n/a | API only |
151
  | Qwen 3.5 9B | 0.812 | 0.434 | 9.1B | 9.1 GB |
152
+ | **Schemer** | **0.800** | **0.911** | 211M | 218 MB int8 |
153
+ | **Schemer (int4)** | **0.793** | 0.908 | 211M | 111 MB int4 |
154
  | Ministral 3 8B | 0.790 | 0.430 | 8.0B | 17.8 GB |
155
  | Gemma 4 E2B (official int4 QAT) | 0.780 | 0.427 | 5.1B | 8.3 GB |
 
 
156
  | Qwen 3.5 0.8B | 0.651 | 0.351 | 0.87B | 1.75 GB |
157
  | NuExtract-2.0-2B | 0.643 | 0.393 | 2.2B | 4.4 GB |
158
  | GLiNER2-multi | 0.585 | 0.374 | 307M | 309 MB int8* |
 
168
 
169
  | Model | Short | Long doc | Long + distractors | Register | Calendar | Factored | Nested |
170
  |---|---:|---:|---:|---:|---:|---:|---:|
171
+ | Gemma 4 26B A4B | **0.844** | **0.844** | 0.693 | 0.815 | 0.829 | 0.954 | **1.000** |
172
  | Claude Haiku 4.5 | 0.838 | 0.759 | **0.762** | **0.831** | 0.802 | 0.817 | **1.000** |
173
  | Qwen 3.5 9B | 0.828 | 0.807 | 0.717 | 0.791 | 0.774 | 0.891 | **1.000** |
174
+ | **Schemer** | 0.759 | 0.708 | 0.685 | 0.829 | **0.946** | 0.946 | 0.988 |
175
+ | **Schemer (int4)** | 0.754 | 0.689 | 0.671 | 0.820 | 0.943 | **0.967** | 0.985 |
176
  | Ministral 3 8B | 0.818 | 0.811 | 0.711 | 0.720 | 0.796 | 0.790 | 0.998 |
177
  | Gemma 4 E2B | 0.809 | 0.785 | 0.705 | 0.773 | 0.719 | 0.759 | **1.000** |
 
 
178
  | Qwen 3.5 0.8B | 0.721 | 0.635 | 0.560 | 0.581 | 0.529 | 0.689 | 0.936 |
179
  | NuExtract-2.0-2B | 0.763 | 0.704 | 0.542 | 0.648 | 0.496 | 0.805 | 0.000† |
180
  | GLiNER2-multi | 0.683 | 0.650 | 0.388 | 0.662 | 0.468 | 0.783 | 0.000† |
 
196
  | datetime | **0.769** | 0.752 | 0.641 | 0.652 |
197
  | array | **0.766** | 0.750 | 0.731 | 0.686 |
198
  | number | **0.750** | 0.743 | 0.670 | 0.679 |
199
+ | string | 0.648 | 0.637 | **0.774** | 0.716 |
200
  | label | 0.629 | 0.609 | **0.636** | 0.616 |
201
 
202
  ### Per language
 
221
  | Chinese | **0.683** | 0.640 | 0.408 |
222
  | Polish | **0.670** | 0.609 | 0.597 |
223
 
224
+ Schemer scores 0.800 overall; every model above it runs 9.1B to 26.6B
225
  parameters or an API. NuExtract-2.0-2B, the closest task-specific
226
  extractor, scores 0.643 at ten times the parameters. Schemer wins all
227
  13 languages against both the strongest sub-1B LLM and GLiNER2-multi,
manifest.json CHANGED
@@ -10,11 +10,12 @@
10
  "review_register": 0.829,
11
  "natural_temporal": 0.9323,
12
  "nested_with_harness": 0.9875,
13
- "pooled_usage_weighted_package": 0.776,
14
- "pooled_usage_weighted_int8": 0.7757,
15
- "pooled_usage_weighted_int4_awq": 0.7693,
16
  "boolean_absence_pooled": 0.911,
17
- "sob_truth_score": "pending"
 
18
  },
19
  "max_length": 1216,
20
  "min_requirements": {
 
10
  "review_register": 0.829,
11
  "natural_temporal": 0.9323,
12
  "nested_with_harness": 0.9875,
13
+ "pooled_usage_weighted_package": 0.8,
14
+ "pooled_usage_weighted_int8": 0.7998,
15
+ "pooled_usage_weighted_int4_awq": 0.7933,
16
  "boolean_absence_pooled": 0.911,
17
+ "sob_truth_score": "pending",
18
+ "factored_domains_pooled": 0.9459
19
  },
20
  "max_length": 1216,
21
  "min_requirements": {