k-l-lambda commited on
Commit
a57617e
·
verified ·
1 Parent(s): dc68113

README: add full_acc@0 row + cond_acc@0 note (complete per-position table)

Browse files
Files changed (1) hide show
  1. README.md +9 -2
README.md CHANGED
@@ -37,17 +37,24 @@ replacement in the same vLLM serving path.
37
 
38
  Per-position draft accuracy on a fixed held-out val split, measured during
39
  training (teacher-forced argmax match; this is a *training-time* metric and is
40
- not directly comparable to runtime accept-length). Continual FT lifts the
41
- weaker later positions most:
 
 
 
42
 
43
  | metric | init | final | rel. |
44
  | --- | --- | --- | --- |
45
  | val_loss | 7.69 | 5.62 | -27% |
 
46
  | full_acc@1 | 0.253 | 0.367 | +45% |
47
  | full_acc@2 | 0.085 | 0.184 | +116% |
48
  | cond_acc@1 | 0.412 | 0.559 | +36% |
49
  | cond_acc@2 | 0.336 | 0.499 | +49% |
50
 
 
 
 
51
  A runtime `accept_length` benchmark (vLLM 0.20, `num_speculative_tokens=3`) is
52
  pending and will be added once measured.
53
 
 
37
 
38
  Per-position draft accuracy on a fixed held-out val split, measured during
39
  training (teacher-forced argmax match; this is a *training-time* metric and is
40
+ not directly comparable to runtime accept-length). `acc@i` is the accuracy at
41
+ TTT position `i` (`i = 0, 1, 2`). `full_acc@i` requires positions `0..i` all
42
+ correct; `cond_acc@i` is conditioned on `0..i-1` being correct. Continual FT
43
+ lifts the weaker later positions most, while the already-strong position 0
44
+ holds:
45
 
46
  | metric | init | final | rel. |
47
  | --- | --- | --- | --- |
48
  | val_loss | 7.69 | 5.62 | -27% |
49
+ | full_acc@0 | 0.614 | 0.656 | +7% |
50
  | full_acc@1 | 0.253 | 0.367 | +45% |
51
  | full_acc@2 | 0.085 | 0.184 | +116% |
52
  | cond_acc@1 | 0.412 | 0.559 | +36% |
53
  | cond_acc@2 | 0.336 | 0.499 | +49% |
54
 
55
+ (`cond_acc@0` equals `full_acc@0` by definition — position 0 has no prior to
56
+ condition on — so it is the same 0.614 -> 0.656.)
57
+
58
  A runtime `accept_length` benchmark (vLLM 0.20, `num_speculative_tokens=3`) is
59
  pending and will be added once measured.
60