Hugging Face's logo

Spaces:
bardsai
/
performance-llm-board
Sleeping

App Files Community
piotr-szleg-bards-ai commited on
Commit
46c6a41
·
1 Parent(s): 362448c

Update

Browse files
Files changed (5) hide show
  1. data/2024-01-25 12:22:00.759762_general_plot.csv +398 -0
  2. data/2024-01-25 13:30:17.207984_summary.csv +59 -0
  3. data/2024-01-25 13:30:20.959750_time_of_day_comparison.csv +20 -0
  4. data/2024-01-25 13:34:46.224347_summary.csv +59 -0
  5. data → data/time_of_day_plot.json +0 -0
data/2024-01-25 12:22:00.759762_general_plot.csv ADDED
@@ -0,0 +1,398 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ path,plot_object,description,comment,plot_json
2
+ ./html/plots/response_times.html,"Figure({
3
+ 'data': [{'alignmentgroup': 'True',
4
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
5
+ 'legendgroup': 'Mistral-7B-Instruct-v0.2',
6
+ 'marker': {'color': '#636efa', 'pattern': {'shape': ''}},
7
+ 'name': 'Mistral-7B-Instruct-v0.2',
8
+ 'offsetgroup': 'Mistral-7B-Instruct-v0.2',
9
+ 'orientation': 'v',
10
+ 'showlegend': True,
11
+ 'textposition': 'auto',
12
+ 'type': 'bar',
13
+ 'x': array(['Mistral-7B-Instruct-v0.2'], dtype=object),
14
+ 'xaxis': 'x',
15
+ 'y': array([3.17454003]),
16
+ 'yaxis': 'y'},
17
+ {'alignmentgroup': 'True',
18
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
19
+ 'legendgroup': 'Mixtral-8x7B-Instruct-v0.1',
20
+ 'marker': {'color': '#EF553B', 'pattern': {'shape': ''}},
21
+ 'name': 'Mixtral-8x7B-Instruct-v0.1',
22
+ 'offsetgroup': 'Mixtral-8x7B-Instruct-v0.1',
23
+ 'orientation': 'v',
24
+ 'showlegend': True,
25
+ 'textposition': 'auto',
26
+ 'type': 'bar',
27
+ 'x': array(['Mixtral-8x7B-Instruct-v0.1'], dtype=object),
28
+ 'xaxis': 'x',
29
+ 'y': array([7.14219763]),
30
+ 'yaxis': 'y'},
31
+ {'alignmentgroup': 'True',
32
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
33
+ 'legendgroup': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
34
+ 'marker': {'color': '#00cc96', 'pattern': {'shape': ''}},
35
+ 'name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
36
+ 'offsetgroup': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
37
+ 'orientation': 'v',
38
+ 'showlegend': True,
39
+ 'textposition': 'auto',
40
+ 'type': 'bar',
41
+ 'x': array(['TinyLlama/TinyLlama-1.1B-Chat-v1.0'], dtype=object),
42
+ 'xaxis': 'x',
43
+ 'y': array([1.29026454]),
44
+ 'yaxis': 'y'},
45
+ {'alignmentgroup': 'True',
46
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
47
+ 'legendgroup': 'chat-bison',
48
+ 'marker': {'color': '#ab63fa', 'pattern': {'shape': ''}},
49
+ 'name': 'chat-bison',
50
+ 'offsetgroup': 'chat-bison',
51
+ 'orientation': 'v',
52
+ 'showlegend': True,
53
+ 'textposition': 'auto',
54
+ 'type': 'bar',
55
+ 'x': array(['chat-bison'], dtype=object),
56
+ 'xaxis': 'x',
57
+ 'y': array([3.89024363]),
58
+ 'yaxis': 'y'},
59
+ {'alignmentgroup': 'True',
60
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
61
+ 'legendgroup': 'chat-bison-32k',
62
+ 'marker': {'color': '#FFA15A', 'pattern': {'shape': ''}},
63
+ 'name': 'chat-bison-32k',
64
+ 'offsetgroup': 'chat-bison-32k',
65
+ 'orientation': 'v',
66
+ 'showlegend': True,
67
+ 'textposition': 'auto',
68
+ 'type': 'bar',
69
+ 'x': array(['chat-bison-32k'], dtype=object),
70
+ 'xaxis': 'x',
71
+ 'y': array([4.12817777]),
72
+ 'yaxis': 'y'},
73
+ {'alignmentgroup': 'True',
74
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
75
+ 'legendgroup': 'gemini-pro',
76
+ 'marker': {'color': '#19d3f3', 'pattern': {'shape': ''}},
77
+ 'name': 'gemini-pro',
78
+ 'offsetgroup': 'gemini-pro',
79
+ 'orientation': 'v',
80
+ 'showlegend': True,
81
+ 'textposition': 'auto',
82
+ 'type': 'bar',
83
+ 'x': array(['gemini-pro'], dtype=object),
84
+ 'xaxis': 'x',
85
+ 'y': array([4.72138064]),
86
+ 'yaxis': 'y'},
87
+ {'alignmentgroup': 'True',
88
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
89
+ 'legendgroup': 'gpt-3.5-turbo',
90
+ 'marker': {'color': '#FF6692', 'pattern': {'shape': ''}},
91
+ 'name': 'gpt-3.5-turbo',
92
+ 'offsetgroup': 'gpt-3.5-turbo',
93
+ 'orientation': 'v',
94
+ 'showlegend': True,
95
+ 'textposition': 'auto',
96
+ 'type': 'bar',
97
+ 'x': array(['gpt-3.5-turbo'], dtype=object),
98
+ 'xaxis': 'x',
99
+ 'y': array([5.28237349]),
100
+ 'yaxis': 'y'},
101
+ {'alignmentgroup': 'True',
102
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
103
+ 'legendgroup': 'gpt-4',
104
+ 'marker': {'color': '#B6E880', 'pattern': {'shape': ''}},
105
+ 'name': 'gpt-4',
106
+ 'offsetgroup': 'gpt-4',
107
+ 'orientation': 'v',
108
+ 'showlegend': True,
109
+ 'textposition': 'auto',
110
+ 'type': 'bar',
111
+ 'x': array(['gpt-4'], dtype=object),
112
+ 'xaxis': 'x',
113
+ 'y': array([15.52184566]),
114
+ 'yaxis': 'y'},
115
+ {'alignmentgroup': 'True',
116
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
117
+ 'legendgroup': 'gpt-4-turbo',
118
+ 'marker': {'color': '#FF97FF', 'pattern': {'shape': ''}},
119
+ 'name': 'gpt-4-turbo',
120
+ 'offsetgroup': 'gpt-4-turbo',
121
+ 'orientation': 'v',
122
+ 'showlegend': True,
123
+ 'textposition': 'auto',
124
+ 'type': 'bar',
125
+ 'x': array(['gpt-4-turbo'], dtype=object),
126
+ 'xaxis': 'x',
127
+ 'y': array([14.00852903]),
128
+ 'yaxis': 'y'},
129
+ {'alignmentgroup': 'True',
130
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
131
+ 'legendgroup': 'llama-2-70b-chat',
132
+ 'marker': {'color': '#FECB52', 'pattern': {'shape': ''}},
133
+ 'name': 'llama-2-70b-chat',
134
+ 'offsetgroup': 'llama-2-70b-chat',
135
+ 'orientation': 'v',
136
+ 'showlegend': True,
137
+ 'textposition': 'auto',
138
+ 'type': 'bar',
139
+ 'x': array(['llama-2-70b-chat'], dtype=object),
140
+ 'xaxis': 'x',
141
+ 'y': array([6.0658799]),
142
+ 'yaxis': 'y'},
143
+ {'alignmentgroup': 'True',
144
+ 'hovertemplate': 'Model=%{x}<br>Execution time (s)=%{y}<extra></extra>',
145
+ 'legendgroup': 'zephyr-7b-beta',
146
+ 'marker': {'color': '#636efa', 'pattern': {'shape': ''}},
147
+ 'name': 'zephyr-7b-beta',
148
+ 'offsetgroup': 'zephyr-7b-beta',
149
+ 'orientation': 'v',
150
+ 'showlegend': True,
151
+ 'textposition': 'auto',
152
+ 'type': 'bar',
153
+ 'x': array(['zephyr-7b-beta'], dtype=object),
154
+ 'xaxis': 'x',
155
+ 'y': array([3.23879932]),
156
+ 'yaxis': 'y'}],
157
+ 'layout': {'barmode': 'relative',
158
+ 'legend': {'title': {'text': 'Model'}, 'tracegroupgap': 0},
159
+ 'template': '...',
160
+ 'title': {'text': 'Model endpoints response times'},
161
+ 'xaxis': {'anchor': 'y',
162
+ 'categoryarray': [Mistral-7B-Instruct-v0.2,
163
+ Mixtral-8x7B-Instruct-v0.1,
164
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,
165
+ chat-bison, chat-bison-32k, gemini-pro,
166
+ gpt-3.5-turbo, gpt-4, gpt-4-turbo,
167
+ llama-2-70b-chat, zephyr-7b-beta],
168
+ 'categoryorder': 'array',
169
+ 'domain': [0.0, 1.0],
170
+ 'title': {'text': 'Model'}},
171
+ 'yaxis': {'anchor': 'x', 'domain': [0.0, 1.0], 'title': {'text': 'Execution time (s)'}}}
172
+ })",Model endpoints response times,,"{""data"":[{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""Mistral-7B-Instruct-v0.2"",""marker"":{""color"":""#636efa"",""pattern"":{""shape"":""""}},""name"":""Mistral-7B-Instruct-v0.2"",""offsetgroup"":""Mistral-7B-Instruct-v0.2"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""Mistral-7B-Instruct-v0.2""],""xaxis"":""x"",""y"":[3.174540030956268],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""Mixtral-8x7B-Instruct-v0.1"",""marker"":{""color"":""#EF553B"",""pattern"":{""shape"":""""}},""name"":""Mixtral-8x7B-Instruct-v0.1"",""offsetgroup"":""Mixtral-8x7B-Instruct-v0.1"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""Mixtral-8x7B-Instruct-v0.1""],""xaxis"":""x"",""y"":[7.142197625471814],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""marker"":{""color"":""#00cc96"",""pattern"":{""shape"":""""}},""name"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""offsetgroup"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0""],""xaxis"":""x"",""y"":[1.2902645373344421],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""chat-bison"",""marker"":{""color"":""#ab63fa"",""pattern"":{""shape"":""""}},""name"":""chat-bison"",""offsetgroup"":""chat-bison"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""chat-bison""],""xaxis"":""x"",""y"":[3.890243631601334],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""chat-bison-32k"",""marker"":{""color"":""#FFA15A"",""pattern"":{""shape"":""""}},""name"":""chat-bison-32k"",""offsetgroup"":""chat-bison-32k"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""chat-bison-32k""],""xaxis"":""x"",""y"":[4.128177767992019],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gemini-pro"",""marker"":{""color"":""#19d3f3"",""pattern"":{""shape"":""""}},""name"":""gemini-pro"",""offsetgroup"":""gemini-pro"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gemini-pro""],""xaxis"":""x"",""y"":[4.72138064004937],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gpt-3.5-turbo"",""marker"":{""color"":""#FF6692"",""pattern"":{""shape"":""""}},""name"":""gpt-3.5-turbo"",""offsetgroup"":""gpt-3.5-turbo"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gpt-3.5-turbo""],""xaxis"":""x"",""y"":[5.282373493889353],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gpt-4"",""marker"":{""color"":""#B6E880"",""pattern"":{""shape"":""""}},""name"":""gpt-4"",""offsetgroup"":""gpt-4"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gpt-4""],""xaxis"":""x"",""y"":[15.5218456586202],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gpt-4-turbo"",""marker"":{""color"":""#FF97FF"",""pattern"":{""shape"":""""}},""name"":""gpt-4-turbo"",""offsetgroup"":""gpt-4-turbo"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gpt-4-turbo""],""xaxis"":""x"",""y"":[14.008529031276703],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""llama-2-70b-chat"",""marker"":{""color"":""#FECB52"",""pattern"":{""shape"":""""}},""name"":""llama-2-70b-chat"",""offsetgroup"":""llama-2-70b-chat"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""llama-2-70b-chat""],""xaxis"":""x"",""y"":[6.06587989590749],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution time (s)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""zephyr-7b-beta"",""marker"":{""color"":""#636efa"",""pattern"":{""shape"":""""}},""name"":""zephyr-7b-beta"",""offsetgroup"":""zephyr-7b-beta"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""zephyr-7b-beta""],""xaxis"":""x"",""y"":[3.2387993240356447],""yaxis"":""y"",""type"":""bar""}],""layout"":{""template"":{""data"":{""histogram2dcontour"":[{""type"":""histogram2dcontour"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""choropleth"":[{""type"":""choropleth"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""histogram2d"":[{""type"":""histogram2d"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""heatmap"":[{""type"":""heatmap"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""heatmapgl"":[{""type"":""heatmapgl"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""contourcarpet"":[{""type"":""contourcarpet"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""contour"":[{""type"":""contour"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""surface"":[{""type"":""surface"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""mesh3d"":[{""type"":""mesh3d"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""scatter"":[{""fillpattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2},""type"":""scatter""}],""parcoords"":[{""type"":""parcoords"",""line"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterpolargl"":[{""type"":""scatterpolargl"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""bar"":[{""error_x"":{""color"":""#2a3f5f""},""error_y"":{""color"":""#2a3f5f""},""marker"":{""line"":{""color"":""#E5ECF6"",""width"":0.5},""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""bar""}],""scattergeo"":[{""type"":""scattergeo"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterpolar"":[{""type"":""scatterpolar"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""histogram"":[{""marker"":{""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""histogram""}],""scattergl"":[{""type"":""scattergl"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatter3d"":[{""type"":""scatter3d"",""line"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}},""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scattermapbox"":[{""type"":""scattermapbox"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterternary"":[{""type"":""scatterternary"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scattercarpet"":[{""type"":""scattercarpet"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""carpet"":[{""aaxis"":{""endlinecolor"":""#2a3f5f"",""gridcolor"":""white"",""linecolor"":""white"",""minorgridcolor"":""white"",""startlinecolor"":""#2a3f5f""},""baxis"":{""endlinecolor"":""#2a3f5f"",""gridcolor"":""white"",""linecolor"":""white"",""minorgridcolor"":""white"",""startlinecolor"":""#2a3f5f""},""type"":""carpet""}],""table"":[{""cells"":{""fill"":{""color"":""#EBF0F8""},""line"":{""color"":""white""}},""header"":{""fill"":{""color"":""#C8D4E3""},""line"":{""color"":""white""}},""type"":""table""}],""barpolar"":[{""marker"":{""line"":{""color"":""#E5ECF6"",""width"":0.5},""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""barpolar""}],""pie"":[{""automargin"":true,""type"":""pie""}]},""layout"":{""autotypenumbers"":""strict"",""colorway"":[""#636efa"",""#EF553B"",""#00cc96"",""#ab63fa"",""#FFA15A"",""#19d3f3"",""#FF6692"",""#B6E880"",""#FF97FF"",""#FECB52""],""font"":{""color"":""#2a3f5f""},""hovermode"":""closest"",""hoverlabel"":{""align"":""left""},""paper_bgcolor"":""white"",""plot_bgcolor"":""#E5ECF6"",""polar"":{""bgcolor"":""#E5ECF6"",""angularaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""radialaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""}},""ternary"":{""bgcolor"":""#E5ECF6"",""aaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""baxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""caxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""}},""coloraxis"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}},""colorscale"":{""sequential"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]],""sequentialminus"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]],""diverging"":[[0,""#8e0152""],[0.1,""#c51b7d""],[0.2,""#de77ae""],[0.3,""#f1b6da""],[0.4,""#fde0ef""],[0.5,""#f7f7f7""],[0.6,""#e6f5d0""],[0.7,""#b8e186""],[0.8,""#7fbc41""],[0.9,""#4d9221""],[1,""#276419""]]},""xaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":"""",""title"":{""standoff"":15},""zerolinecolor"":""white"",""automargin"":true,""zerolinewidth"":2},""yaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":"""",""title"":{""standoff"":15},""zerolinecolor"":""white"",""automargin"":true,""zerolinewidth"":2},""scene"":{""xaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2},""yaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2},""zaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2}},""shapedefaults"":{""line"":{""color"":""#2a3f5f""}},""annotationdefaults"":{""arrowcolor"":""#2a3f5f"",""arrowhead"":0,""arrowwidth"":1},""geo"":{""bgcolor"":""white"",""landcolor"":""#E5ECF6"",""subunitcolor"":""white"",""showland"":true,""showlakes"":true,""lakecolor"":""white""},""title"":{""x"":0.05},""mapbox"":{""style"":""light""}}},""xaxis"":{""anchor"":""y"",""domain"":[0.0,1.0],""title"":{""text"":""Model""},""categoryorder"":""array"",""categoryarray"":[""Mistral-7B-Instruct-v0.2"",""Mixtral-8x7B-Instruct-v0.1"",""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""chat-bison"",""chat-bison-32k"",""gemini-pro"",""gpt-3.5-turbo"",""gpt-4"",""gpt-4-turbo"",""llama-2-70b-chat"",""zephyr-7b-beta""]},""yaxis"":{""anchor"":""x"",""domain"":[0.0,1.0],""title"":{""text"":""Execution time (s)""}},""legend"":{""title"":{""text"":""Model""},""tracegroupgap"":0},""title"":{""text"":""Model endpoints response times""},""barmode"":""relative""}}"
173
+ ./html/plots/execution_costs.html,"Figure({
174
+ 'data': [{'alignmentgroup': 'True',
175
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
176
+ 'legendgroup': 'gpt-3.5-turbo',
177
+ 'marker': {'color': '#636efa', 'pattern': {'shape': ''}},
178
+ 'name': 'gpt-3.5-turbo',
179
+ 'offsetgroup': 'gpt-3.5-turbo',
180
+ 'orientation': 'v',
181
+ 'showlegend': True,
182
+ 'textposition': 'auto',
183
+ 'type': 'bar',
184
+ 'x': array(['gpt-3.5-turbo', 'gpt-3.5-turbo'], dtype=object),
185
+ 'xaxis': 'x',
186
+ 'y': array([0.144438, 0.144438]),
187
+ 'yaxis': 'y'},
188
+ {'alignmentgroup': 'True',
189
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
190
+ 'legendgroup': 'gpt-4-turbo',
191
+ 'marker': {'color': '#EF553B', 'pattern': {'shape': ''}},
192
+ 'name': 'gpt-4-turbo',
193
+ 'offsetgroup': 'gpt-4-turbo',
194
+ 'orientation': 'v',
195
+ 'showlegend': True,
196
+ 'textposition': 'auto',
197
+ 'type': 'bar',
198
+ 'x': array(['gpt-4-turbo'], dtype=object),
199
+ 'xaxis': 'x',
200
+ 'y': array([0.44014]),
201
+ 'yaxis': 'y'},
202
+ {'alignmentgroup': 'True',
203
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
204
+ 'legendgroup': 'gpt-4',
205
+ 'marker': {'color': '#00cc96', 'pattern': {'shape': ''}},
206
+ 'name': 'gpt-4',
207
+ 'offsetgroup': 'gpt-4',
208
+ 'orientation': 'v',
209
+ 'showlegend': True,
210
+ 'textposition': 'auto',
211
+ 'type': 'bar',
212
+ 'x': array(['gpt-4'], dtype=object),
213
+ 'xaxis': 'x',
214
+ 'y': array([1.1724]),
215
+ 'yaxis': 'y'},
216
+ {'alignmentgroup': 'True',
217
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
218
+ 'legendgroup': 'llama-2-70b-chat',
219
+ 'marker': {'color': '#ab63fa', 'pattern': {'shape': ''}},
220
+ 'name': 'llama-2-70b-chat',
221
+ 'offsetgroup': 'llama-2-70b-chat',
222
+ 'orientation': 'v',
223
+ 'showlegend': True,
224
+ 'textposition': 'auto',
225
+ 'type': 'bar',
226
+ 'x': array(['llama-2-70b-chat'], dtype=object),
227
+ 'xaxis': 'x',
228
+ 'y': array([0.0186864]),
229
+ 'yaxis': 'y'},
230
+ {'alignmentgroup': 'True',
231
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
232
+ 'legendgroup': 'Mixtral-8x7B-Instruct-v0.1',
233
+ 'marker': {'color': '#FFA15A', 'pattern': {'shape': ''}},
234
+ 'name': 'Mixtral-8x7B-Instruct-v0.1',
235
+ 'offsetgroup': 'Mixtral-8x7B-Instruct-v0.1',
236
+ 'orientation': 'v',
237
+ 'showlegend': True,
238
+ 'textposition': 'auto',
239
+ 'type': 'bar',
240
+ 'x': array(['Mixtral-8x7B-Instruct-v0.1'], dtype=object),
241
+ 'xaxis': 'x',
242
+ 'y': array([0.0781551]),
243
+ 'yaxis': 'y'},
244
+ {'alignmentgroup': 'True',
245
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
246
+ 'legendgroup': 'zephyr-7b-beta',
247
+ 'marker': {'color': '#19d3f3', 'pattern': {'shape': ''}},
248
+ 'name': 'zephyr-7b-beta',
249
+ 'offsetgroup': 'zephyr-7b-beta',
250
+ 'orientation': 'v',
251
+ 'showlegend': True,
252
+ 'textposition': 'auto',
253
+ 'type': 'bar',
254
+ 'x': array(['zephyr-7b-beta'], dtype=object),
255
+ 'xaxis': 'x',
256
+ 'y': array([55.45703704]),
257
+ 'yaxis': 'y'},
258
+ {'alignmentgroup': 'True',
259
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
260
+ 'legendgroup': 'Mistral-7B-Instruct-v0.2',
261
+ 'marker': {'color': '#FF6692', 'pattern': {'shape': ''}},
262
+ 'name': 'Mistral-7B-Instruct-v0.2',
263
+ 'offsetgroup': 'Mistral-7B-Instruct-v0.2',
264
+ 'orientation': 'v',
265
+ 'showlegend': True,
266
+ 'textposition': 'auto',
267
+ 'type': 'bar',
268
+ 'x': array(['Mistral-7B-Instruct-v0.2'], dtype=object),
269
+ 'xaxis': 'x',
270
+ 'y': array([55.45703704]),
271
+ 'yaxis': 'y'},
272
+ {'alignmentgroup': 'True',
273
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
274
+ 'legendgroup': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
275
+ 'marker': {'color': '#B6E880', 'pattern': {'shape': ''}},
276
+ 'name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
277
+ 'offsetgroup': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
278
+ 'orientation': 'v',
279
+ 'showlegend': True,
280
+ 'textposition': 'auto',
281
+ 'type': 'bar',
282
+ 'x': array(['TinyLlama/TinyLlama-1.1B-Chat-v1.0'], dtype=object),
283
+ 'xaxis': 'x',
284
+ 'y': array([25.59555556]),
285
+ 'yaxis': 'y'},
286
+ {'alignmentgroup': 'True',
287
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
288
+ 'legendgroup': 'gemini-pro',
289
+ 'marker': {'color': '#FF97FF', 'pattern': {'shape': ''}},
290
+ 'name': 'gemini-pro',
291
+ 'offsetgroup': 'gemini-pro',
292
+ 'orientation': 'v',
293
+ 'showlegend': True,
294
+ 'textposition': 'auto',
295
+ 'type': 'bar',
296
+ 'x': array(['gemini-pro'], dtype=object),
297
+ 'xaxis': 'x',
298
+ 'y': array([0.022971]),
299
+ 'yaxis': 'y'},
300
+ {'alignmentgroup': 'True',
301
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
302
+ 'legendgroup': 'chat-bison',
303
+ 'marker': {'color': '#FECB52', 'pattern': {'shape': ''}},
304
+ 'name': 'chat-bison',
305
+ 'offsetgroup': 'chat-bison',
306
+ 'orientation': 'v',
307
+ 'showlegend': True,
308
+ 'textposition': 'auto',
309
+ 'type': 'bar',
310
+ 'x': array(['chat-bison'], dtype=object),
311
+ 'xaxis': 'x',
312
+ 'y': array([0.008985]),
313
+ 'yaxis': 'y'},
314
+ {'alignmentgroup': 'True',
315
+ 'hovertemplate': 'Model=%{x}<br>Execution cost ($)=%{y}<extra></extra>',
316
+ 'legendgroup': 'chat-bison-32k',
317
+ 'marker': {'color': '#636efa', 'pattern': {'shape': ''}},
318
+ 'name': 'chat-bison-32k',
319
+ 'offsetgroup': 'chat-bison-32k',
320
+ 'orientation': 'v',
321
+ 'showlegend': True,
322
+ 'textposition': 'auto',
323
+ 'type': 'bar',
324
+ 'x': array(['chat-bison-32k'], dtype=object),
325
+ 'xaxis': 'x',
326
+ 'y': array([0.0089655]),
327
+ 'yaxis': 'y'}],
328
+ 'layout': {'barmode': 'relative',
329
+ 'legend': {'title': {'text': 'Model'}, 'tracegroupgap': 0},
330
+ 'template': '...',
331
+ 'title': {'text': 'Costs of execution of 80 test queries per model'},
332
+ 'xaxis': {'anchor': 'y', 'domain': [0.0, 1.0], 'title': {'text': 'Model'}},
333
+ 'yaxis': {'anchor': 'x', 'domain': [0.0, 1.0], 'title': {'text': 'Execution cost ($)'}}}
334
+ })",Costs of execution of 80 test queries per model,Note that some providers bill by token count and some by execution time. In the second case we used averaged pausing and resuming time to calculate the cost and assumed that all queries have been done in one session. Also note that for now the Gemini-Pro model is free for under 60 queries per minute.,"{""data"":[{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gpt-3.5-turbo"",""marker"":{""color"":""#636efa"",""pattern"":{""shape"":""""}},""name"":""gpt-3.5-turbo"",""offsetgroup"":""gpt-3.5-turbo"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gpt-3.5-turbo"",""gpt-3.5-turbo""],""xaxis"":""x"",""y"":[0.144438,0.144438],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gpt-4-turbo"",""marker"":{""color"":""#EF553B"",""pattern"":{""shape"":""""}},""name"":""gpt-4-turbo"",""offsetgroup"":""gpt-4-turbo"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gpt-4-turbo""],""xaxis"":""x"",""y"":[0.44014],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gpt-4"",""marker"":{""color"":""#00cc96"",""pattern"":{""shape"":""""}},""name"":""gpt-4"",""offsetgroup"":""gpt-4"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gpt-4""],""xaxis"":""x"",""y"":[1.1724],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""llama-2-70b-chat"",""marker"":{""color"":""#ab63fa"",""pattern"":{""shape"":""""}},""name"":""llama-2-70b-chat"",""offsetgroup"":""llama-2-70b-chat"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""llama-2-70b-chat""],""xaxis"":""x"",""y"":[0.0186864],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""Mixtral-8x7B-Instruct-v0.1"",""marker"":{""color"":""#FFA15A"",""pattern"":{""shape"":""""}},""name"":""Mixtral-8x7B-Instruct-v0.1"",""offsetgroup"":""Mixtral-8x7B-Instruct-v0.1"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""Mixtral-8x7B-Instruct-v0.1""],""xaxis"":""x"",""y"":[0.07815509999999999],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""zephyr-7b-beta"",""marker"":{""color"":""#19d3f3"",""pattern"":{""shape"":""""}},""name"":""zephyr-7b-beta"",""offsetgroup"":""zephyr-7b-beta"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""zephyr-7b-beta""],""xaxis"":""x"",""y"":[55.45703703703705],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""Mistral-7B-Instruct-v0.2"",""marker"":{""color"":""#FF6692"",""pattern"":{""shape"":""""}},""name"":""Mistral-7B-Instruct-v0.2"",""offsetgroup"":""Mistral-7B-Instruct-v0.2"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""Mistral-7B-Instruct-v0.2""],""xaxis"":""x"",""y"":[55.45703703703705],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""marker"":{""color"":""#B6E880"",""pattern"":{""shape"":""""}},""name"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""offsetgroup"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0""],""xaxis"":""x"",""y"":[25.59555555555556],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""gemini-pro"",""marker"":{""color"":""#FF97FF"",""pattern"":{""shape"":""""}},""name"":""gemini-pro"",""offsetgroup"":""gemini-pro"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""gemini-pro""],""xaxis"":""x"",""y"":[0.022971],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""chat-bison"",""marker"":{""color"":""#FECB52"",""pattern"":{""shape"":""""}},""name"":""chat-bison"",""offsetgroup"":""chat-bison"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""chat-bison""],""xaxis"":""x"",""y"":[0.008985],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eExecution cost ($)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""chat-bison-32k"",""marker"":{""color"":""#636efa"",""pattern"":{""shape"":""""}},""name"":""chat-bison-32k"",""offsetgroup"":""chat-bison-32k"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""chat-bison-32k""],""xaxis"":""x"",""y"":[0.0089655],""yaxis"":""y"",""type"":""bar""}],""layout"":{""template"":{""data"":{""histogram2dcontour"":[{""type"":""histogram2dcontour"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""choropleth"":[{""type"":""choropleth"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""histogram2d"":[{""type"":""histogram2d"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""heatmap"":[{""type"":""heatmap"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""heatmapgl"":[{""type"":""heatmapgl"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""contourcarpet"":[{""type"":""contourcarpet"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""contour"":[{""type"":""contour"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""surface"":[{""type"":""surface"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""mesh3d"":[{""type"":""mesh3d"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""scatter"":[{""fillpattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2},""type"":""scatter""}],""parcoords"":[{""type"":""parcoords"",""line"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterpolargl"":[{""type"":""scatterpolargl"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""bar"":[{""error_x"":{""color"":""#2a3f5f""},""error_y"":{""color"":""#2a3f5f""},""marker"":{""line"":{""color"":""#E5ECF6"",""width"":0.5},""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""bar""}],""scattergeo"":[{""type"":""scattergeo"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterpolar"":[{""type"":""scatterpolar"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""histogram"":[{""marker"":{""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""histogram""}],""scattergl"":[{""type"":""scattergl"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatter3d"":[{""type"":""scatter3d"",""line"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}},""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scattermapbox"":[{""type"":""scattermapbox"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterternary"":[{""type"":""scatterternary"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scattercarpet"":[{""type"":""scattercarpet"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""carpet"":[{""aaxis"":{""endlinecolor"":""#2a3f5f"",""gridcolor"":""white"",""linecolor"":""white"",""minorgridcolor"":""white"",""startlinecolor"":""#2a3f5f""},""baxis"":{""endlinecolor"":""#2a3f5f"",""gridcolor"":""white"",""linecolor"":""white"",""minorgridcolor"":""white"",""startlinecolor"":""#2a3f5f""},""type"":""carpet""}],""table"":[{""cells"":{""fill"":{""color"":""#EBF0F8""},""line"":{""color"":""white""}},""header"":{""fill"":{""color"":""#C8D4E3""},""line"":{""color"":""white""}},""type"":""table""}],""barpolar"":[{""marker"":{""line"":{""color"":""#E5ECF6"",""width"":0.5},""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""barpolar""}],""pie"":[{""automargin"":true,""type"":""pie""}]},""layout"":{""autotypenumbers"":""strict"",""colorway"":[""#636efa"",""#EF553B"",""#00cc96"",""#ab63fa"",""#FFA15A"",""#19d3f3"",""#FF6692"",""#B6E880"",""#FF97FF"",""#FECB52""],""font"":{""color"":""#2a3f5f""},""hovermode"":""closest"",""hoverlabel"":{""align"":""left""},""paper_bgcolor"":""white"",""plot_bgcolor"":""#E5ECF6"",""polar"":{""bgcolor"":""#E5ECF6"",""angularaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""radialaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""}},""ternary"":{""bgcolor"":""#E5ECF6"",""aaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""baxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""caxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""}},""coloraxis"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}},""colorscale"":{""sequential"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]],""sequentialminus"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]],""diverging"":[[0,""#8e0152""],[0.1,""#c51b7d""],[0.2,""#de77ae""],[0.3,""#f1b6da""],[0.4,""#fde0ef""],[0.5,""#f7f7f7""],[0.6,""#e6f5d0""],[0.7,""#b8e186""],[0.8,""#7fbc41""],[0.9,""#4d9221""],[1,""#276419""]]},""xaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":"""",""title"":{""standoff"":15},""zerolinecolor"":""white"",""automargin"":true,""zerolinewidth"":2},""yaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":"""",""title"":{""standoff"":15},""zerolinecolor"":""white"",""automargin"":true,""zerolinewidth"":2},""scene"":{""xaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2},""yaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2},""zaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2}},""shapedefaults"":{""line"":{""color"":""#2a3f5f""}},""annotationdefaults"":{""arrowcolor"":""#2a3f5f"",""arrowhead"":0,""arrowwidth"":1},""geo"":{""bgcolor"":""white"",""landcolor"":""#E5ECF6"",""subunitcolor"":""white"",""showland"":true,""showlakes"":true,""lakecolor"":""white""},""title"":{""x"":0.05},""mapbox"":{""style"":""light""}}},""xaxis"":{""anchor"":""y"",""domain"":[0.0,1.0],""title"":{""text"":""Model""}},""yaxis"":{""anchor"":""x"",""domain"":[0.0,1.0],""title"":{""text"":""Execution cost ($)""}},""legend"":{""title"":{""text"":""Model""},""tracegroupgap"":0},""title"":{""text"":""Costs of execution of 80 test queries per model""},""barmode"":""relative""}}"
335
+ ./html/plots/model_sizes.html,"Figure({
336
+ 'data': [{'alignmentgroup': 'True',
337
+ 'hovertemplate': 'Model=%{x}<br>Model size (B params)=%{y}<extra></extra>',
338
+ 'legendgroup': 'Mixtral-8x7B-Instruct-v0.1',
339
+ 'marker': {'color': '#636efa', 'pattern': {'shape': ''}},
340
+ 'name': 'Mixtral-8x7B-Instruct-v0.1',
341
+ 'offsetgroup': 'Mixtral-8x7B-Instruct-v0.1',
342
+ 'orientation': 'v',
343
+ 'showlegend': True,
344
+ 'textposition': 'auto',
345
+ 'type': 'bar',
346
+ 'x': array(['Mixtral-8x7B-Instruct-v0.1'], dtype=object),
347
+ 'xaxis': 'x',
348
+ 'y': array([56.]),
349
+ 'yaxis': 'y'},
350
+ {'alignmentgroup': 'True',
351
+ 'hovertemplate': 'Model=%{x}<br>Model size (B params)=%{y}<extra></extra>',
352
+ 'legendgroup': 'zephyr-7b-beta',
353
+ 'marker': {'color': '#EF553B', 'pattern': {'shape': ''}},
354
+ 'name': 'zephyr-7b-beta',
355
+ 'offsetgroup': 'zephyr-7b-beta',
356
+ 'orientation': 'v',
357
+ 'showlegend': True,
358
+ 'textposition': 'auto',
359
+ 'type': 'bar',
360
+ 'x': array(['zephyr-7b-beta'], dtype=object),
361
+ 'xaxis': 'x',
362
+ 'y': array([7.]),
363
+ 'yaxis': 'y'},
364
+ {'alignmentgroup': 'True',
365
+ 'hovertemplate': 'Model=%{x}<br>Model size (B params)=%{y}<extra></extra>',
366
+ 'legendgroup': 'Mistral-7B-Instruct-v0.2',
367
+ 'marker': {'color': '#00cc96', 'pattern': {'shape': ''}},
368
+ 'name': 'Mistral-7B-Instruct-v0.2',
369
+ 'offsetgroup': 'Mistral-7B-Instruct-v0.2',
370
+ 'orientation': 'v',
371
+ 'showlegend': True,
372
+ 'textposition': 'auto',
373
+ 'type': 'bar',
374
+ 'x': array(['Mistral-7B-Instruct-v0.2'], dtype=object),
375
+ 'xaxis': 'x',
376
+ 'y': array([7.]),
377
+ 'yaxis': 'y'},
378
+ {'alignmentgroup': 'True',
379
+ 'hovertemplate': 'Model=%{x}<br>Model size (B params)=%{y}<extra></extra>',
380
+ 'legendgroup': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
381
+ 'marker': {'color': '#ab63fa', 'pattern': {'shape': ''}},
382
+ 'name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
383
+ 'offsetgroup': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0',
384
+ 'orientation': 'v',
385
+ 'showlegend': True,
386
+ 'textposition': 'auto',
387
+ 'type': 'bar',
388
+ 'x': array(['TinyLlama/TinyLlama-1.1B-Chat-v1.0'], dtype=object),
389
+ 'xaxis': 'x',
390
+ 'y': array([1.1]),
391
+ 'yaxis': 'y'}],
392
+ 'layout': {'barmode': 'relative',
393
+ 'legend': {'title': {'text': 'Model'}, 'tracegroupgap': 0},
394
+ 'template': '...',
395
+ 'title': {'text': 'Model sizes in billions of parameters'},
396
+ 'xaxis': {'anchor': 'y', 'domain': [0.0, 1.0], 'title': {'text': 'Model'}},
397
+ 'yaxis': {'anchor': 'x', 'domain': [0.0, 1.0], 'title': {'text': 'Model size (B params)'}}}
398
+ })",Model sizes in billions of parameters,Note that some models have undisclosed sizes,"{""data"":[{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eModel size (B params)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""Mixtral-8x7B-Instruct-v0.1"",""marker"":{""color"":""#636efa"",""pattern"":{""shape"":""""}},""name"":""Mixtral-8x7B-Instruct-v0.1"",""offsetgroup"":""Mixtral-8x7B-Instruct-v0.1"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""Mixtral-8x7B-Instruct-v0.1""],""xaxis"":""x"",""y"":[56.0],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eModel size (B params)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""zephyr-7b-beta"",""marker"":{""color"":""#EF553B"",""pattern"":{""shape"":""""}},""name"":""zephyr-7b-beta"",""offsetgroup"":""zephyr-7b-beta"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""zephyr-7b-beta""],""xaxis"":""x"",""y"":[7.0],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eModel size (B params)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""Mistral-7B-Instruct-v0.2"",""marker"":{""color"":""#00cc96"",""pattern"":{""shape"":""""}},""name"":""Mistral-7B-Instruct-v0.2"",""offsetgroup"":""Mistral-7B-Instruct-v0.2"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""Mistral-7B-Instruct-v0.2""],""xaxis"":""x"",""y"":[7.0],""yaxis"":""y"",""type"":""bar""},{""alignmentgroup"":""True"",""hovertemplate"":""Model=%{x}\u003cbr\u003eModel size (B params)=%{y}\u003cextra\u003e\u003c\u002fextra\u003e"",""legendgroup"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""marker"":{""color"":""#ab63fa"",""pattern"":{""shape"":""""}},""name"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""offsetgroup"":""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0"",""orientation"":""v"",""showlegend"":true,""textposition"":""auto"",""x"":[""TinyLlama\u002fTinyLlama-1.1B-Chat-v1.0""],""xaxis"":""x"",""y"":[1.1],""yaxis"":""y"",""type"":""bar""}],""layout"":{""template"":{""data"":{""histogram2dcontour"":[{""type"":""histogram2dcontour"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""choropleth"":[{""type"":""choropleth"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""histogram2d"":[{""type"":""histogram2d"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""heatmap"":[{""type"":""heatmap"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""heatmapgl"":[{""type"":""heatmapgl"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""contourcarpet"":[{""type"":""contourcarpet"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""contour"":[{""type"":""contour"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""surface"":[{""type"":""surface"",""colorbar"":{""outlinewidth"":0,""ticks"":""""},""colorscale"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]]}],""mesh3d"":[{""type"":""mesh3d"",""colorbar"":{""outlinewidth"":0,""ticks"":""""}}],""scatter"":[{""fillpattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2},""type"":""scatter""}],""parcoords"":[{""type"":""parcoords"",""line"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterpolargl"":[{""type"":""scatterpolargl"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""bar"":[{""error_x"":{""color"":""#2a3f5f""},""error_y"":{""color"":""#2a3f5f""},""marker"":{""line"":{""color"":""#E5ECF6"",""width"":0.5},""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""bar""}],""scattergeo"":[{""type"":""scattergeo"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterpolar"":[{""type"":""scatterpolar"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""histogram"":[{""marker"":{""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""histogram""}],""scattergl"":[{""type"":""scattergl"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatter3d"":[{""type"":""scatter3d"",""line"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}},""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scattermapbox"":[{""type"":""scattermapbox"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scatterternary"":[{""type"":""scatterternary"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""scattercarpet"":[{""type"":""scattercarpet"",""marker"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}}}],""carpet"":[{""aaxis"":{""endlinecolor"":""#2a3f5f"",""gridcolor"":""white"",""linecolor"":""white"",""minorgridcolor"":""white"",""startlinecolor"":""#2a3f5f""},""baxis"":{""endlinecolor"":""#2a3f5f"",""gridcolor"":""white"",""linecolor"":""white"",""minorgridcolor"":""white"",""startlinecolor"":""#2a3f5f""},""type"":""carpet""}],""table"":[{""cells"":{""fill"":{""color"":""#EBF0F8""},""line"":{""color"":""white""}},""header"":{""fill"":{""color"":""#C8D4E3""},""line"":{""color"":""white""}},""type"":""table""}],""barpolar"":[{""marker"":{""line"":{""color"":""#E5ECF6"",""width"":0.5},""pattern"":{""fillmode"":""overlay"",""size"":10,""solidity"":0.2}},""type"":""barpolar""}],""pie"":[{""automargin"":true,""type"":""pie""}]},""layout"":{""autotypenumbers"":""strict"",""colorway"":[""#636efa"",""#EF553B"",""#00cc96"",""#ab63fa"",""#FFA15A"",""#19d3f3"",""#FF6692"",""#B6E880"",""#FF97FF"",""#FECB52""],""font"":{""color"":""#2a3f5f""},""hovermode"":""closest"",""hoverlabel"":{""align"":""left""},""paper_bgcolor"":""white"",""plot_bgcolor"":""#E5ECF6"",""polar"":{""bgcolor"":""#E5ECF6"",""angularaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""radialaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""}},""ternary"":{""bgcolor"":""#E5ECF6"",""aaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""baxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""},""caxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":""""}},""coloraxis"":{""colorbar"":{""outlinewidth"":0,""ticks"":""""}},""colorscale"":{""sequential"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]],""sequentialminus"":[[0.0,""#0d0887""],[0.1111111111111111,""#46039f""],[0.2222222222222222,""#7201a8""],[0.3333333333333333,""#9c179e""],[0.4444444444444444,""#bd3786""],[0.5555555555555556,""#d8576b""],[0.6666666666666666,""#ed7953""],[0.7777777777777778,""#fb9f3a""],[0.8888888888888888,""#fdca26""],[1.0,""#f0f921""]],""diverging"":[[0,""#8e0152""],[0.1,""#c51b7d""],[0.2,""#de77ae""],[0.3,""#f1b6da""],[0.4,""#fde0ef""],[0.5,""#f7f7f7""],[0.6,""#e6f5d0""],[0.7,""#b8e186""],[0.8,""#7fbc41""],[0.9,""#4d9221""],[1,""#276419""]]},""xaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":"""",""title"":{""standoff"":15},""zerolinecolor"":""white"",""automargin"":true,""zerolinewidth"":2},""yaxis"":{""gridcolor"":""white"",""linecolor"":""white"",""ticks"":"""",""title"":{""standoff"":15},""zerolinecolor"":""white"",""automargin"":true,""zerolinewidth"":2},""scene"":{""xaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2},""yaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2},""zaxis"":{""backgroundcolor"":""#E5ECF6"",""gridcolor"":""white"",""linecolor"":""white"",""showbackground"":true,""ticks"":"""",""zerolinecolor"":""white"",""gridwidth"":2}},""shapedefaults"":{""line"":{""color"":""#2a3f5f""}},""annotationdefaults"":{""arrowcolor"":""#2a3f5f"",""arrowhead"":0,""arrowwidth"":1},""geo"":{""bgcolor"":""white"",""landcolor"":""#E5ECF6"",""subunitcolor"":""white"",""showland"":true,""showlakes"":true,""lakecolor"":""white""},""title"":{""x"":0.05},""mapbox"":{""style"":""light""}}},""xaxis"":{""anchor"":""y"",""domain"":[0.0,1.0],""title"":{""text"":""Model""}},""yaxis"":{""anchor"":""x"",""domain"":[0.0,1.0],""title"":{""text"":""Model size (B params)""}},""legend"":{""title"":{""text"":""Model""},""tracegroupgap"":0},""title"":{""text"":""Model sizes in billions of parameters""},""barmode"":""relative""}}"
data/2024-01-25 13:30:17.207984_summary.csv ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model,language,template_name,execution_time,characters_count,words_count,chunk_sizes_mean,chunk_sizes_min,chunk_sizes_max,chunk_generation_times_mean,chunk_generation_times_min,chunk_generation_times_max
2
+ Mistral-7B-Instruct-v0.2,english,call,,,,,,,,,
3
+ Mistral-7B-Instruct-v0.2,english,json,7.318650853633881,222.2,35.95,4.157156220767072,1.0,14.0,0.10865272596910582,0.00024437904357910156,60.7178213596344
4
+ Mistral-7B-Instruct-v0.2,english,markdown,2.8044421672821045,238.15,38.35,4.148954703832753,1.0,14.0,0.03463017442085187,0.020945072174072266,0.25538039207458496
5
+ Mistral-7B-Instruct-v0.2,japanese,call,,,,,,,,,
6
+ Mistral-7B-Instruct-v0.2,japanese,json,1.929928070306778,37.9,2.95,1.3882783882783882,1.0,11.0,0.039187116718990896,0.020466089248657227,0.23994874954223633
7
+ Mistral-7B-Instruct-v0.2,japanese,markdown,1.8897509932518006,28.85,1.15,1.088679245283019,1.0,4.0,0.0396594097029488,0.0004932880401611328,0.2205650806427002
8
+ Mixtral-8x7B-Instruct-v0.1,english,call,,,,,,,,,
9
+ Mixtral-8x7B-Instruct-v0.1,english,json,3.181837737560272,394.95,61.5,4.355665839536807,1.0,15.0,0.015196637342151676,0.0002071857452392578,1.011472463607788
10
+ Mixtral-8x7B-Instruct-v0.1,english,markdown,3.399144798517227,654.4,105.05,4.374331550802139,1.0,14.0,0.013296874289844125,0.00021576881408691406,0.5904912948608398
11
+ Mixtral-8x7B-Instruct-v0.1,japanese,call,,,,,,,,,
12
+ Mixtral-8x7B-Instruct-v0.1,japanese,json,8.82588157138309,532.425,56.35,2.0712896323672436,1.0,15.0,0.027260187998098102,0.0002186298370361328,76.45081758499146
13
+ Mixtral-8x7B-Instruct-v0.1,japanese,markdown,12.566328430175782,460.6,5.7,1.0741604477611941,1.0,9.0,0.01511352992991903,0.00021529197692871094,0.8098442554473877
14
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,english,call,,,,,,,,,
15
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,english,json,1.3998499751091003,203.15,34.6,4.120689655172414,1.0,13.0,0.01180358711652775,0.0003421306610107422,0.13397955894470215
16
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,english,markdown,1.4005955815315247,179.5,30.65,3.847802786709539,1.0,14.0,0.01265179331535184,0.0003273487091064453,0.13492584228515625
17
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,japanese,call,,,,,,,,,
18
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,japanese,json,1.2260903775691987,48.35,5.8,1.95748987854251,1.0,13.0,0.01474765992840292,0.00041413307189941406,0.18952107429504395
19
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,japanese,markdown,1.1986963748931885,51.55,6.85,2.0415841584158416,1.0,10.0,0.014000740617808729,0.0005300045013427734,0.16827392578125
20
+ chat-bison,english,call,,,,,,,,,
21
+ chat-bison,english,json,3.446989095211029,550.05,86.0,89.4390243902439,1.0,138.0,0.4258628240445765,0.0006544589996337891,2.2015340328216553
22
+ chat-bison,japanese,call,,,,,,,,,
23
+ chat-bison,japanese,json,4.333498167991638,246.1,12.0,33.48299319727891,3.0,124.0,0.47770044917152044,0.0005426406860351562,3.5425846576690674
24
+ chat-bison-32k,english,call,,,,,,,,,
25
+ chat-bison-32k,english,json,3.74513863325119,538.4,84.25,88.26229508196721,1.0,134.0,0.47832588094179745,0.0006556510925292969,3.0114917755126953
26
+ chat-bison-32k,japanese,call,,,,,,,,,
27
+ chat-bison-32k,japanese,json,4.5112169027328495,241.45,11.8,35.77037037037037,6.0,124.0,0.5467217816246881,0.0005869865417480469,4.387756824493408
28
+ gemini-pro,english,call,,,,,,,,,
29
+ gemini-pro,english,json,3.6837018529574075,557.475,87.225,175.58267716535434,4.0,988.0,1.0232121174729716,0.0006723403930664062,8.669384241104126
30
+ gemini-pro,english,markdown,4.207570552825928,663.8,109.05,139.74736842105264,19.0,472.0,0.7348335742950439,0.0007510185241699219,2.7830429077148438
31
+ gemini-pro,japanese,call,,,,,,,,,
32
+ gemini-pro,japanese,json,4.794884290013995,189.55,4.05,64.2542372881356,4.0,215.0,1.1180692850533178,0.0006062984466552734,5.698127031326294
33
+ gemini-pro,japanese,markdown,7.311563323525822,392.7,7.75,62.832,15.0,203.0,0.9826514282226563,0.0006639957427978516,13.754565954208374
34
+ gpt-3.5-turbo,english,call,3.409479808062315,671.15625,108.59375,,,,,,
35
+ gpt-3.5-turbo,english,json,2.650409721559094,564.6451612903226,90.98387096774194,5.037122302158274,1.0,16.0,0.017705622920029456,0.0005590915679931641,0.4925534725189209
36
+ gpt-3.5-turbo,english,markdown,3.1277945737044015,649.3333333333334,103.75,5.186023294509152,1.0,16.0,0.019346605402459322,0.0006084442138671875,1.127073049545288
37
+ gpt-3.5-turbo,japanese,call,8.916680586142618,375.1875,4.9375,,,,,,
38
+ gpt-3.5-turbo,japanese,json,6.141976299932447,244.75806451612902,3.306451612903226,1.1217474866942638,1.0,7.0,0.0232258041112258,0.0005545616149902344,13.75982117652893
39
+ gpt-3.5-turbo,japanese,markdown,8.25130424045381,315.5,1.3636363636363635,1.1153784348385023,1.0,10.0,0.025426305415027686,0.0005831718444824219,13.71114993095398
40
+ gpt-4,english,call,10.59899697303772,538.45,83.7,,,,,,
41
+ gpt-4,english,json,9.713180017471313,558.35,87.5,4.971950133570792,1.0,16.0,0.07619813660479824,0.0005440711975097656,1.9534873962402344
42
+ gpt-4,japanese,call,22.77469485684445,242.1,1.2,,,,,,
43
+ gpt-4,japanese,json,19.565327330639487,217.85,4.1,1.1505149194613151,1.0,7.0,0.09166746716213503,0.0005655288696289062,2.1393778324127197
44
+ gpt-4-turbo,english,call,9.345100498199463,550.9,84.9,,,,,,
45
+ gpt-4-turbo,english,json,8.325499892234802,686.55,105.4,5.042600073448402,1.0,17.0,0.05410054322774035,0.0007143020629882812,0.755082368850708
46
+ gpt-4-turbo,japanese,call,18.703535676002502,222.35,3.6,,,,,,
47
+ gpt-4-turbo,japanese,json,19.659980058670044,272.5,6.0,1.181956191715463,1.0,8.0,0.07962745783567377,0.0006952285766601562,13.252992153167725
48
+ llama-2-70b-chat,english,call,,,,,,,,,
49
+ llama-2-70b-chat,english,json,3.55666184425354,427.6363636363636,65.81818181818181,4.172062084257206,1.0,14.0,0.01772472985303587,0.00021791458129882812,0.5100884437561035
50
+ llama-2-70b-chat,english,markdown,6.183407843112946,1169.7727272727273,184.1818181818182,4.07263807564488,1.0,16.0,0.01361775794423896,0.0002124309539794922,0.5661592483520508
51
+ llama-2-70b-chat,japanese,call,,,,,,,,,
52
+ llama-2-70b-chat,japanese,json,4.945841180650811,350.4761904761905,42.642857142857146,2.46483590087073,1.0,15.0,0.01905980879878806,0.00021982192993164062,0.35228896141052246
53
+ llama-2-70b-chat,japanese,markdown,14.294062587949965,1299.1818181818182,172.5,2.8476636445152934,1.0,16.0,0.02320206142747451,0.0002117156982421875,76.44498753547668
54
+ zephyr-7b-beta,english,call,,,,,,,,,
55
+ zephyr-7b-beta,english,json,2.845248854160309,238.1,39.2,4.274685816876122,1.0,14.0,0.03622233246986485,0.0003070831298828125,0.4896988868713379
56
+ zephyr-7b-beta,english,markdown,7.618257820606232,238.65,38.75,4.358904109589041,1.0,15.0,0.1235110485390441,0.00040459632873535156,75.15552616119385
57
+ zephyr-7b-beta,japanese,call,,,,,,,,,
58
+ zephyr-7b-beta,japanese,json,1.921021181344986,29.75,3.3,1.1829025844930416,1.0,7.0,0.042037644869764565,0.018090486526489258,0.4599947929382324
59
+ zephyr-7b-beta,japanese,markdown,1.8884475827217102,26.6,1.75,1.0350194552529184,1.0,4.0,0.04097196471366437,0.02583909034729004,0.4606599807739258
data/2024-01-25 13:30:20.959750_time_of_day_comparison.csv ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model,time_of_day,execution_time,characters_count,words_count
2
+ Mistral-7B-Instruct-v0.2,afternoon,,,
3
+ Mistral-7B-Instruct-v0.2,evening,,,
4
+ Mistral-7B-Instruct-v0.2,morning,3.174540030956268,113.0,16.27
5
+ Mixtral-8x7B-Instruct-v0.1,morning,7.142197625471814,494.9583333333333,57.74166666666667
6
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,morning,1.2902645373344421,106.18,16.74
7
+ chat-bison,morning,3.890243631601334,398.075,49.0
8
+ chat-bison-32k,morning,4.128177767992019,389.925,48.025
9
+ gemini-pro,morning,4.72138064004937,425.09166666666664,49.891666666666666
10
+ gpt-3.5-turbo,afternoon,4.666725277900696,302.1666666666667,32.0
11
+ gpt-3.5-turbo,evening,4.351089119911194,305.25,28.5
12
+ gpt-3.5-turbo,morning,5.336839800789242,479.2785714285714,53.92142857142857
13
+ gpt-4,morning,15.5218456586202,389.1875,44.125
14
+ gpt-4-turbo,morning,14.008529031276703,433.075,49.975
15
+ llama-2-70b-chat,afternoon,2.1692867279052734,292.0,43.5
16
+ llama-2-70b-chat,evening,5.552149415016174,967.5,141.25
17
+ llama-2-70b-chat,morning,6.175082007679371,662.7416666666667,94.30833333333334
18
+ zephyr-7b-beta,afternoon,,,
19
+ zephyr-7b-beta,evening,,,
20
+ zephyr-7b-beta,morning,3.2387993240356447,112.57,17.26
data/2024-01-25 13:34:46.224347_summary.csv ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model,language,template_name,execution_time,characters_count,words_count,chunk_sizes_mean,chunk_sizes_min,chunk_sizes_max,chunk_generation_times_mean,chunk_generation_times_min,chunk_generation_times_max
2
+ Mistral-7B-Instruct-v0.2,english,call,,,,,,,,,
3
+ Mistral-7B-Instruct-v0.2,english,json,7.318650853633881,222.2,35.95,4.157156220767072,1.0,14.0,0.10865272596910582,0.00024437904357910156,60.7178213596344
4
+ Mistral-7B-Instruct-v0.2,english,markdown,2.8044421672821045,238.15,38.35,4.148954703832753,1.0,14.0,0.03463017442085187,0.020945072174072266,0.25538039207458496
5
+ Mistral-7B-Instruct-v0.2,japanese,call,,,,,,,,,
6
+ Mistral-7B-Instruct-v0.2,japanese,json,1.929928070306778,37.9,2.95,1.3882783882783882,1.0,11.0,0.039187116718990896,0.020466089248657227,0.23994874954223633
7
+ Mistral-7B-Instruct-v0.2,japanese,markdown,1.8897509932518006,28.85,1.15,1.088679245283019,1.0,4.0,0.0396594097029488,0.0004932880401611328,0.2205650806427002
8
+ Mixtral-8x7B-Instruct-v0.1,english,call,,,,,,,,,
9
+ Mixtral-8x7B-Instruct-v0.1,english,json,3.181837737560272,394.95,61.5,4.355665839536807,1.0,15.0,0.015196637342151676,0.0002071857452392578,1.011472463607788
10
+ Mixtral-8x7B-Instruct-v0.1,english,markdown,3.399144798517227,654.4,105.05,4.374331550802139,1.0,14.0,0.013296874289844125,0.00021576881408691406,0.5904912948608398
11
+ Mixtral-8x7B-Instruct-v0.1,japanese,call,,,,,,,,,
12
+ Mixtral-8x7B-Instruct-v0.1,japanese,json,8.82588157138309,532.425,56.35,2.0712896323672436,1.0,15.0,0.027260187998098102,0.0002186298370361328,76.45081758499146
13
+ Mixtral-8x7B-Instruct-v0.1,japanese,markdown,12.566328430175782,460.6,5.7,1.0741604477611941,1.0,9.0,0.01511352992991903,0.00021529197692871094,0.8098442554473877
14
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,english,call,,,,,,,,,
15
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,english,json,1.3998499751091003,203.15,34.6,4.120689655172414,1.0,13.0,0.01180358711652775,0.0003421306610107422,0.13397955894470215
16
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,english,markdown,1.4005955815315247,179.5,30.65,3.847802786709539,1.0,14.0,0.01265179331535184,0.0003273487091064453,0.13492584228515625
17
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,japanese,call,,,,,,,,,
18
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,japanese,json,1.2260903775691987,48.35,5.8,1.95748987854251,1.0,13.0,0.01474765992840292,0.00041413307189941406,0.18952107429504395
19
+ TinyLlama/TinyLlama-1.1B-Chat-v1.0,japanese,markdown,1.1986963748931885,51.55,6.85,2.0415841584158416,1.0,10.0,0.014000740617808729,0.0005300045013427734,0.16827392578125
20
+ chat-bison,english,call,,,,,,,,,
21
+ chat-bison,english,json,3.446989095211029,550.05,86.0,89.4390243902439,1.0,138.0,0.4258628240445765,0.0006544589996337891,2.2015340328216553
22
+ chat-bison,japanese,call,,,,,,,,,
23
+ chat-bison,japanese,json,4.333498167991638,246.1,12.0,33.48299319727891,3.0,124.0,0.47770044917152044,0.0005426406860351562,3.5425846576690674
24
+ chat-bison-32k,english,call,,,,,,,,,
25
+ chat-bison-32k,english,json,3.74513863325119,538.4,84.25,88.26229508196721,1.0,134.0,0.47832588094179745,0.0006556510925292969,3.0114917755126953
26
+ chat-bison-32k,japanese,call,,,,,,,,,
27
+ chat-bison-32k,japanese,json,4.5112169027328495,241.45,11.8,35.77037037037037,6.0,124.0,0.5467217816246881,0.0005869865417480469,4.387756824493408
28
+ gemini-pro,english,call,,,,,,,,,
29
+ gemini-pro,english,json,3.6837018529574075,557.475,87.225,175.58267716535434,4.0,988.0,1.0232121174729716,0.0006723403930664062,8.669384241104126
30
+ gemini-pro,english,markdown,4.207570552825928,663.8,109.05,139.74736842105264,19.0,472.0,0.7348335742950439,0.0007510185241699219,2.7830429077148438
31
+ gemini-pro,japanese,call,,,,,,,,,
32
+ gemini-pro,japanese,json,4.794884290013995,189.55,4.05,64.2542372881356,4.0,215.0,1.1180692850533178,0.0006062984466552734,5.698127031326294
33
+ gemini-pro,japanese,markdown,7.311563323525822,392.7,7.75,62.832,15.0,203.0,0.9826514282226563,0.0006639957427978516,13.754565954208374
34
+ gpt-3.5-turbo,english,call,3.409479808062315,671.15625,108.59375,,,,,,
35
+ gpt-3.5-turbo,english,json,2.650409721559094,564.6451612903226,90.98387096774194,5.037122302158274,1.0,16.0,0.017705622920029456,0.0005590915679931641,0.4925534725189209
36
+ gpt-3.5-turbo,english,markdown,3.1277945737044015,649.3333333333334,103.75,5.186023294509152,1.0,16.0,0.019346605402459322,0.0006084442138671875,1.127073049545288
37
+ gpt-3.5-turbo,japanese,call,8.916680586142618,375.1875,4.9375,,,,,,
38
+ gpt-3.5-turbo,japanese,json,6.141976299932447,244.75806451612902,3.306451612903226,1.1217474866942638,1.0,7.0,0.0232258041112258,0.0005545616149902344,13.75982117652893
39
+ gpt-3.5-turbo,japanese,markdown,8.25130424045381,315.5,1.3636363636363635,1.1153784348385023,1.0,10.0,0.025426305415027686,0.0005831718444824219,13.71114993095398
40
+ gpt-4,english,call,10.59899697303772,538.45,83.7,,,,,,
41
+ gpt-4,english,json,9.713180017471313,558.35,87.5,4.971950133570792,1.0,16.0,0.07619813660479824,0.0005440711975097656,1.9534873962402344
42
+ gpt-4,japanese,call,22.77469485684445,242.1,1.2,,,,,,
43
+ gpt-4,japanese,json,19.565327330639487,217.85,4.1,1.1505149194613151,1.0,7.0,0.09166746716213503,0.0005655288696289062,2.1393778324127197
44
+ gpt-4-turbo,english,call,9.345100498199463,550.9,84.9,,,,,,
45
+ gpt-4-turbo,english,json,8.325499892234802,686.55,105.4,5.042600073448402,1.0,17.0,0.05410054322774035,0.0007143020629882812,0.755082368850708
46
+ gpt-4-turbo,japanese,call,18.703535676002502,222.35,3.6,,,,,,
47
+ gpt-4-turbo,japanese,json,19.659980058670044,272.5,6.0,1.181956191715463,1.0,8.0,0.07962745783567377,0.0006952285766601562,13.252992153167725
48
+ llama-2-70b-chat,english,call,,,,,,,,,
49
+ llama-2-70b-chat,english,json,3.55666184425354,427.6363636363636,65.81818181818181,4.172062084257206,1.0,14.0,0.01772472985303587,0.00021791458129882812,0.5100884437561035
50
+ llama-2-70b-chat,english,markdown,6.183407843112946,1169.7727272727273,184.1818181818182,4.07263807564488,1.0,16.0,0.01361775794423896,0.0002124309539794922,0.5661592483520508
51
+ llama-2-70b-chat,japanese,call,,,,,,,,,
52
+ llama-2-70b-chat,japanese,json,4.945841180650811,350.4761904761905,42.642857142857146,2.46483590087073,1.0,15.0,0.01905980879878806,0.00021982192993164062,0.35228896141052246
53
+ llama-2-70b-chat,japanese,markdown,14.294062587949965,1299.1818181818182,172.5,2.8476636445152934,1.0,16.0,0.02320206142747451,0.0002117156982421875,76.44498753547668
54
+ zephyr-7b-beta,english,call,,,,,,,,,
55
+ zephyr-7b-beta,english,json,2.845248854160309,238.1,39.2,4.274685816876122,1.0,14.0,0.03622233246986485,0.0003070831298828125,0.4896988868713379
56
+ zephyr-7b-beta,english,markdown,7.618257820606232,238.65,38.75,4.358904109589041,1.0,15.0,0.1235110485390441,0.00040459632873535156,75.15552616119385
57
+ zephyr-7b-beta,japanese,call,,,,,,,,,
58
+ zephyr-7b-beta,japanese,json,1.921021181344986,29.75,3.3,1.1829025844930416,1.0,7.0,0.042037644869764565,0.018090486526489258,0.4599947929382324
59
+ zephyr-7b-beta,japanese,markdown,1.8884475827217102,26.6,1.75,1.0350194552529184,1.0,4.0,0.04097196471366437,0.02583909034729004,0.4606599807739258
data → data/time_of_day_plot.json RENAMED
File without changes