maxgfr commited on
Commit
a8366f1
·
verified ·
1 Parent(s): f2c4bfc

Publish s6-route

Browse files
Files changed (1) hide show
  1. README.md +7 -0
README.md CHANGED
@@ -134,6 +134,13 @@ Paired bootstrap (slates): +0.0942 [0.0720, 0.1185] over the model alone, +0.148
134
  of the intents group's (chance 0.3721); what the model did with them is not measured
135
  ([v4's result](https://github.com/maxgfr/skycmd/blob/main/docs/s6-route4-prereg.md)).
136
  - **Read with the request hidden** (18/11/2026, files already read; [diagnostic](https://github.com/maxgfr/skycmd/blob/main/runs/s6-route6/diagnostic.md)): with the request replaced by `?` (every BM25 score 0.00), the pair picks the right tool among the tools on 0.1104 [0.0906, 0.1310] of `v_tools` and 0.1402 [0.1158, 0.1658] of T2-tools (clusters [0.0658, 0.2311]), against a chance of 0.1429, and below chance under masked names (0.0699 and 0.0503); its *none* answer does not depend on whether *none* is right (gaps +0.0461 [−0.0066, +0.0969] and +0.0119 [−0.0204, +0.0432]): it does not answer from the options alone in that reading, which does not measure a cue used together with the request.
 
 
 
 
 
 
 
137
  - **MiniLM is more accurate on every file.** If 91.6 MB and about 12 ms of CPU per request are
138
  affordable, all-MiniLM-L6-v2 with cosine is the better router on these tests.
139
  - **Outside the tool lists of its training families it stays near its embedding part, below it on MetaTool and above it on CLINC150 and MASSIVE:** MetaTool
 
134
  of the intents group's (chance 0.3721); what the model did with them is not measured
135
  ([v4's result](https://github.com/maxgfr/skycmd/blob/main/docs/s6-route4-prereg.md)).
136
  - **Read with the request hidden** (18/11/2026, files already read; [diagnostic](https://github.com/maxgfr/skycmd/blob/main/runs/s6-route6/diagnostic.md)): with the request replaced by `?` (every BM25 score 0.00), the pair picks the right tool among the tools on 0.1104 [0.0906, 0.1310] of `v_tools` and 0.1402 [0.1158, 0.1658] of T2-tools (clusters [0.0658, 0.2311]), against a chance of 0.1429, and below chance under masked names (0.0699 and 0.0503); its *none* answer does not depend on whether *none* is right (gaps +0.0461 [−0.0066, +0.0969] and +0.0119 [−0.0204, +0.0432]): it does not answer from the options alone in that reading, which does not measure a cue used together with the request.
137
+ - **On tools of a new family** (19/11/2026, reported with s6-route v7, deciding nothing about this
138
+ pass; [result](https://github.com/maxgfr/skycmd/blob/main/docs/s6-route7-prereg.md)): on 2,000
139
+ fresh requests about the tools of 262 MCP servers (Toucan), which no training slate holds, it
140
+ picks the right option on 0.5530, where BM25 reads 0.5675 and all-MiniLM-L6-v2 0.6630; with the
141
+ names masked 0.2810; with the request hidden 0.1794 (servers [0.1466, 0.2138], chance 0.1429).
142
+ Its retrained successor, [s6-route7-nano](https://huggingface.co/maxgfr/s6-route7-nano), failed
143
+ its gate on the same requests, and the line is closed: this file stays the published router.
144
  - **MiniLM is more accurate on every file.** If 91.6 MB and about 12 ms of CPU per request are
145
  affordable, all-MiniLM-L6-v2 with cosine is the better router on these tests.
146
  - **Outside the tool lists of its training families it stays near its embedding part, below it on MetaTool and above it on CLINC150 and MASSIVE:** MetaTool