faxenoff commited on
Commit
2d7f959
·
verified ·
1 Parent(s): f496094

Add token_probes.json: tokenizer ids the graph expects for five probe strings (checked by the code-daemon worker at start)

Browse files
Files changed (1) hide show
  1. token_probes.json +112 -0
token_probes.json ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": 1,
3
+ "ids": "hf",
4
+ "source": "tokenizer.json",
5
+ "probes": [
6
+ {
7
+ "text": "parse the config file",
8
+ "ids": [
9
+ 0,
10
+ 366,
11
+ 184,
12
+ 70,
13
+ 6,
14
+ 153837,
15
+ 11435,
16
+ 2
17
+ ]
18
+ },
19
+ {
20
+ "text": "fn build_index(allocator: std.mem.Allocator) !void",
21
+ "ids": [
22
+ 0,
23
+ 6,
24
+ 14783,
25
+ 45367,
26
+ 454,
27
+ 4711,
28
+ 132,
29
+ 289,
30
+ 116292,
31
+ 748,
32
+ 12,
33
+ 6138,
34
+ 71,
35
+ 5,
36
+ 37138,
37
+ 5,
38
+ 6210,
39
+ 116292,
40
+ 748,
41
+ 16,
42
+ 711,
43
+ 965,
44
+ 532,
45
+ 2
46
+ ]
47
+ },
48
+ {
49
+ "text": "Разбор конфигурации и загрузка индекса",
50
+ "ids": [
51
+ 0,
52
+ 7720,
53
+ 10698,
54
+ 211783,
55
+ 89,
56
+ 35,
57
+ 61,
58
+ 49371,
59
+ 415,
60
+ 76404,
61
+ 59,
62
+ 2
63
+ ]
64
+ },
65
+ {
66
+ "text": "HTTP/2 connection pool — keep-alive timeout=30s",
67
+ "ids": [
68
+ 0,
69
+ 135913,
70
+ 12477,
71
+ 94878,
72
+ 19361,
73
+ 292,
74
+ 13695,
75
+ 9,
76
+ 11,
77
+ 24056,
78
+ 1733,
79
+ 6056,
80
+ 1369,
81
+ 1197,
82
+ 7,
83
+ 2
84
+ ]
85
+ },
86
+ {
87
+ "text": "src/inference/tokenizer.zig:589 encodePair",
88
+ "ids": [
89
+ 0,
90
+ 91,
91
+ 19437,
92
+ 64,
93
+ 73,
94
+ 69988,
95
+ 13,
96
+ 64,
97
+ 6448,
98
+ 33,
99
+ 52825,
100
+ 5,
101
+ 38170,
102
+ 12,
103
+ 192531,
104
+ 22,
105
+ 40899,
106
+ 8335,
107
+ 481,
108
+ 2
109
+ ]
110
+ }
111
+ ]
112
+ }