code-daemon-ner-v1 / token_probes.json
faxenoff's picture
Add token_probes.json: tokenizer ids the graph expects for five probe strings (checked by the code-daemon worker at start)
b3319f9 verified
Raw History Blame Contribute Delete
1.24 kB
{
"schema": 1,
"ids": "hf",
"source": "tokenizer.json",
"probes": [
{
"text": "parse the config file",
"ids": [
0,
366,
184,
70,
6,
153837,
11435,
2
]
},
{
"text": "fn build_index(allocator: std.mem.Allocator) !void",
"ids": [
0,
6,
14783,
45367,
454,
4711,
132,
289,
116292,
748,
12,
6138,
71,
5,
37138,
5,
6210,
116292,
748,
16,
711,
965,
532,
2
]
},
{
"text": "Разбор конфигурации и загрузка индекса",
"ids": [
0,
7720,
10698,
211783,
89,
35,
61,
49371,
415,
76404,
59,
2
]
},
{
"text": "HTTP/2 connection pool — keep-alive timeout=30s",
"ids": [
0,
135913,
12477,
94878,
19361,
292,
13695,
9,
11,
24056,
1733,
6056,
1369,
1197,
7,
2
]
},
{
"text": "src/inference/tokenizer.zig:589 encodePair",
"ids": [
0,
91,
19437,
64,
73,
69988,
13,
64,
6448,
33,
52825,
5,
38170,
12,
192531,
22,
40899,
8335,
481,
2
]
}
]
}