nicumog commited on
Commit
99116d2
·
verified ·
1 Parent(s): 81dc469

Update the model card

Browse files
Files changed (1) hide show
  1. README.md +26 -26
README.md CHANGED
@@ -25,11 +25,11 @@ model-index:
25
  metrics:
26
  - type: f1
27
  name: Mean per-chart cell F1 (3% tolerance)
28
- value: 0.857
29
  verified: false
30
  - type: f1
31
  name: Mean per-chart cell F1 (strict)
32
- value: 0.746
33
  verified: false
34
  ---
35
 
@@ -59,7 +59,7 @@ within 0.1%. Every entry sees the same charts and the same prompt.
59
 
60
  | System | F1 · 3% | Strict F1 | F1 · 5% | Shape | Type | Title | sec/chart |
61
  |---|---:|---:|---:|---:|---:|---:|---:|
62
- | **Nutrient Chart Parsing** | **0.857** | 0.746 | 0.871 | 0.946 | 0.972 | 0.983 | 0.27 |
63
  | Gemini 3.8 Flash | 0.710 | 0.640 | 0.713 | 0.796 | 0.960 | 0.998 | — |
64
  | Claude Opus 5 | 0.697 | 0.617 | 0.703 | 0.792 | 0.967 | 0.998 | — |
65
  | GPT-6 Astra | 0.675 | 0.628 | 0.676 | 0.802 | 0.964 | 0.997 | — |
@@ -73,8 +73,8 @@ within 0.1%. Every entry sees the same charts and the same prompt.
73
  | Qwen3-VL-4B-Instruct | 0.518 | 0.443 | 0.536 | 0.623 | 0.908 | 0.984 | 0.30 |
74
  | Qwen3.5-4B | 0.169 | 0.143 | 0.171 | 0.200 | 0.290 | 0.312 | 1.00 |
75
 
76
- The Nutrient model reaches **0.857**, ahead of the strongest cloud model here, **Gemini 3.8 Flash (0.710)**,
77
- and it recovers the right table shape on **94.6%** of charts against 80% for the best cloud model. Its lead
78
  is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per
79
  chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that
80
  setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the
@@ -87,27 +87,27 @@ charts of each type.
87
 
88
  | Chart type | n | Strict | 3% | 5% |
89
  |---|---:|---:|---:|---:|
90
- | line | 552 | 0.788 | 0.904 | 0.916 |
91
- | bar | 463 | 0.841 | 0.928 | 0.935 |
92
- | pie | 249 | 0.731 | 0.734 | 0.734 |
93
- | scatter | 235 | 0.749 | 0.814 | 0.820 |
94
- | area | 224 | 0.786 | 0.863 | 0.883 |
95
- | box | 171 | 0.671 | 0.968 | 0.981 |
96
- | bubble | 164 | 0.595 | 0.688 | 0.698 |
97
- | heatmap | 100 | 0.798 | 0.800 | 0.827 |
98
- | histogram | 54 | 0.723 | 0.840 | 0.870 |
99
- | treemap | 54 | 0.182 | 0.868 | 0.985 |
100
- | combo | 51 | 0.748 | 0.903 | 0.911 |
101
- | funnel | 49 | 0.901 | 0.903 | 0.923 |
102
- | radar | 46 | 0.496 | 0.602 | 0.638 |
103
- | violin | 33 | 0.526 | 0.942 | 0.981 |
104
- | stem | 25 | 0.766 | 0.866 | 0.876 |
105
- | step | 15 | 0.850 | 0.850 | 0.850 |
106
- | candlestick | 15 | 0.671 | 0.774 | 0.791 |
107
-
108
- Per-field accuracy on the same set: chart type 0.972, title 0.983, x and y axis labels 0.979 and 0.978,
109
- axis scales 0.967 and 0.976, legend visibility 0.985, data-label visibility 0.983, column types 0.981,
110
- column roles 0.979, axis bounds 0.747. Set F1 over encoding pairs 0.966 and over series pairs 0.986.
111
 
112
  ## Output
113
 
 
25
  metrics:
26
  - type: f1
27
  name: Mean per-chart cell F1 (3% tolerance)
28
+ value: 0.888
29
  verified: false
30
  - type: f1
31
  name: Mean per-chart cell F1 (strict)
32
+ value: 0.797
33
  verified: false
34
  ---
35
 
 
59
 
60
  | System | F1 · 3% | Strict F1 | F1 · 5% | Shape | Type | Title | sec/chart |
61
  |---|---:|---:|---:|---:|---:|---:|---:|
62
+ | **Nutrient Chart Parsing** | **0.888** | 0.797 | 0.897 | 0.963 | 0.983 | 0.989 | 0.27 |
63
  | Gemini 3.8 Flash | 0.710 | 0.640 | 0.713 | 0.796 | 0.960 | 0.998 | — |
64
  | Claude Opus 5 | 0.697 | 0.617 | 0.703 | 0.792 | 0.967 | 0.998 | — |
65
  | GPT-6 Astra | 0.675 | 0.628 | 0.676 | 0.802 | 0.964 | 0.997 | — |
 
73
  | Qwen3-VL-4B-Instruct | 0.518 | 0.443 | 0.536 | 0.623 | 0.908 | 0.984 | 0.30 |
74
  | Qwen3.5-4B | 0.169 | 0.143 | 0.171 | 0.200 | 0.290 | 0.312 | 1.00 |
75
 
76
+ The Nutrient model reaches **0.888**, ahead of the strongest cloud model here, **Gemini 3.8 Flash (0.710)**,
77
+ and it recovers the right table shape on **96.3%** of charts against 80% for the best cloud model. Its lead
78
  is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per
79
  chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that
80
  setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the
 
87
 
88
  | Chart type | n | Strict | 3% | 5% |
89
  |---|---:|---:|---:|---:|
90
+ | line | 552 | 0.831 | 0.924 | 0.928 |
91
+ | bar | 463 | 0.857 | 0.940 | 0.946 |
92
+ | pie | 249 | 0.762 | 0.765 | 0.765 |
93
+ | scatter | 235 | 0.819 | 0.859 | 0.860 |
94
+ | area | 224 | 0.822 | 0.889 | 0.912 |
95
+ | box | 171 | 0.753 | 0.980 | 0.991 |
96
+ | bubble | 164 | 0.665 | 0.750 | 0.761 |
97
+ | heatmap | 100 | 0.868 | 0.878 | 0.898 |
98
+ | histogram | 54 | 0.875 | 0.927 | 0.946 |
99
+ | treemap | 54 | 0.341 | 0.961 | 0.994 |
100
+ | combo | 51 | 0.784 | 0.937 | 0.945 |
101
+ | funnel | 49 | 0.953 | 0.953 | 0.953 |
102
+ | radar | 46 | 0.549 | 0.651 | 0.682 |
103
+ | violin | 33 | 0.691 | 0.958 | 0.980 |
104
+ | stem | 25 | 0.772 | 0.844 | 0.846 |
105
+ | step | 15 | 0.983 | 0.983 | 0.983 |
106
+ | candlestick | 15 | 0.693 | 0.807 | 0.813 |
107
+
108
+ Per-field accuracy on the same set: chart type 0.983, title 0.989, x and y axis labels 0.987 and 0.986,
109
+ axis scales 0.975 and 0.985, legend visibility 0.990, data-label visibility 0.989, column types 0.989,
110
+ column roles 0.988, axis bounds 0.835. Set F1 over encoding pairs 0.974 and over series pairs 0.993.
111
 
112
  ## Output
113