Update the model card
Browse files
README.md
CHANGED
|
@@ -25,11 +25,11 @@ model-index:
|
|
| 25 |
metrics:
|
| 26 |
- type: f1
|
| 27 |
name: Mean per-chart cell F1 (3% tolerance)
|
| 28 |
-
value: 0.
|
| 29 |
verified: false
|
| 30 |
- type: f1
|
| 31 |
name: Mean per-chart cell F1 (strict)
|
| 32 |
-
value: 0.
|
| 33 |
verified: false
|
| 34 |
---
|
| 35 |
|
|
@@ -59,7 +59,7 @@ within 0.1%. Every entry sees the same charts and the same prompt.
|
|
| 59 |
|
| 60 |
| System | F1 · 3% | Strict F1 | F1 · 5% | Shape | Type | Title | sec/chart |
|
| 61 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 62 |
-
| **Nutrient Chart Parsing** | **0.
|
| 63 |
| Gemini 3.8 Flash | 0.710 | 0.640 | 0.713 | 0.796 | 0.960 | 0.998 | — |
|
| 64 |
| Claude Opus 5 | 0.697 | 0.617 | 0.703 | 0.792 | 0.967 | 0.998 | — |
|
| 65 |
| GPT-6 Astra | 0.675 | 0.628 | 0.676 | 0.802 | 0.964 | 0.997 | — |
|
|
@@ -73,8 +73,8 @@ within 0.1%. Every entry sees the same charts and the same prompt.
|
|
| 73 |
| Qwen3-VL-4B-Instruct | 0.518 | 0.443 | 0.536 | 0.623 | 0.908 | 0.984 | 0.30 |
|
| 74 |
| Qwen3.5-4B | 0.169 | 0.143 | 0.171 | 0.200 | 0.290 | 0.312 | 1.00 |
|
| 75 |
|
| 76 |
-
The Nutrient model reaches **0.
|
| 77 |
-
and it recovers the right table shape on **
|
| 78 |
is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per
|
| 79 |
chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that
|
| 80 |
setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the
|
|
@@ -87,27 +87,27 @@ charts of each type.
|
|
| 87 |
|
| 88 |
| Chart type | n | Strict | 3% | 5% |
|
| 89 |
|---|---:|---:|---:|---:|
|
| 90 |
-
| line | 552 | 0.
|
| 91 |
-
| bar | 463 | 0.
|
| 92 |
-
| pie | 249 | 0.
|
| 93 |
-
| scatter | 235 | 0.
|
| 94 |
-
| area | 224 | 0.
|
| 95 |
-
| box | 171 | 0.
|
| 96 |
-
| bubble | 164 | 0.
|
| 97 |
-
| heatmap | 100 | 0.
|
| 98 |
-
| histogram | 54 | 0.
|
| 99 |
-
| treemap | 54 | 0.
|
| 100 |
-
| combo | 51 | 0.
|
| 101 |
-
| funnel | 49 | 0.
|
| 102 |
-
| radar | 46 | 0.
|
| 103 |
-
| violin | 33 | 0.
|
| 104 |
-
| stem | 25 | 0.
|
| 105 |
-
| step | 15 | 0.
|
| 106 |
-
| candlestick | 15 | 0.
|
| 107 |
-
|
| 108 |
-
Per-field accuracy on the same set: chart type 0.
|
| 109 |
-
axis scales 0.
|
| 110 |
-
column roles 0.
|
| 111 |
|
| 112 |
## Output
|
| 113 |
|
|
|
|
| 25 |
metrics:
|
| 26 |
- type: f1
|
| 27 |
name: Mean per-chart cell F1 (3% tolerance)
|
| 28 |
+
value: 0.888
|
| 29 |
verified: false
|
| 30 |
- type: f1
|
| 31 |
name: Mean per-chart cell F1 (strict)
|
| 32 |
+
value: 0.797
|
| 33 |
verified: false
|
| 34 |
---
|
| 35 |
|
|
|
|
| 59 |
|
| 60 |
| System | F1 · 3% | Strict F1 | F1 · 5% | Shape | Type | Title | sec/chart |
|
| 61 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 62 |
+
| **Nutrient Chart Parsing** | **0.888** | 0.797 | 0.897 | 0.963 | 0.983 | 0.989 | 0.27 |
|
| 63 |
| Gemini 3.8 Flash | 0.710 | 0.640 | 0.713 | 0.796 | 0.960 | 0.998 | — |
|
| 64 |
| Claude Opus 5 | 0.697 | 0.617 | 0.703 | 0.792 | 0.967 | 0.998 | — |
|
| 65 |
| GPT-6 Astra | 0.675 | 0.628 | 0.676 | 0.802 | 0.964 | 0.997 | — |
|
|
|
|
| 73 |
| Qwen3-VL-4B-Instruct | 0.518 | 0.443 | 0.536 | 0.623 | 0.908 | 0.984 | 0.30 |
|
| 74 |
| Qwen3.5-4B | 0.169 | 0.143 | 0.171 | 0.200 | 0.290 | 0.312 | 1.00 |
|
| 75 |
|
| 76 |
+
The Nutrient model reaches **0.888**, ahead of the strongest cloud model here, **Gemini 3.8 Flash (0.710)**,
|
| 77 |
+
and it recovers the right table shape on **96.3%** of charts against 80% for the best cloud model. Its lead
|
| 78 |
is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per
|
| 79 |
chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that
|
| 80 |
setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the
|
|
|
|
| 87 |
|
| 88 |
| Chart type | n | Strict | 3% | 5% |
|
| 89 |
|---|---:|---:|---:|---:|
|
| 90 |
+
| line | 552 | 0.831 | 0.924 | 0.928 |
|
| 91 |
+
| bar | 463 | 0.857 | 0.940 | 0.946 |
|
| 92 |
+
| pie | 249 | 0.762 | 0.765 | 0.765 |
|
| 93 |
+
| scatter | 235 | 0.819 | 0.859 | 0.860 |
|
| 94 |
+
| area | 224 | 0.822 | 0.889 | 0.912 |
|
| 95 |
+
| box | 171 | 0.753 | 0.980 | 0.991 |
|
| 96 |
+
| bubble | 164 | 0.665 | 0.750 | 0.761 |
|
| 97 |
+
| heatmap | 100 | 0.868 | 0.878 | 0.898 |
|
| 98 |
+
| histogram | 54 | 0.875 | 0.927 | 0.946 |
|
| 99 |
+
| treemap | 54 | 0.341 | 0.961 | 0.994 |
|
| 100 |
+
| combo | 51 | 0.784 | 0.937 | 0.945 |
|
| 101 |
+
| funnel | 49 | 0.953 | 0.953 | 0.953 |
|
| 102 |
+
| radar | 46 | 0.549 | 0.651 | 0.682 |
|
| 103 |
+
| violin | 33 | 0.691 | 0.958 | 0.980 |
|
| 104 |
+
| stem | 25 | 0.772 | 0.844 | 0.846 |
|
| 105 |
+
| step | 15 | 0.983 | 0.983 | 0.983 |
|
| 106 |
+
| candlestick | 15 | 0.693 | 0.807 | 0.813 |
|
| 107 |
+
|
| 108 |
+
Per-field accuracy on the same set: chart type 0.983, title 0.989, x and y axis labels 0.987 and 0.986,
|
| 109 |
+
axis scales 0.975 and 0.985, legend visibility 0.990, data-label visibility 0.989, column types 0.989,
|
| 110 |
+
column roles 0.988, axis bounds 0.835. Set F1 over encoding pairs 0.974 and over series pairs 0.993.
|
| 111 |
|
| 112 |
## Output
|
| 113 |
|