HelloSun commited on
Commit
bd32da7
·
verified ·
1 Parent(s): 26d741f

Stage 1: runtime knob sweep (poll, cpu-strict, load mode) and -ub sweep: no gains

Browse files
README.md CHANGED
@@ -82,6 +82,13 @@ Interleaved runs on a shared host (see `benchmarks/ab_results.csv`; 12-thread re
82
  much slower. `ngram-mod` costs little on novel text and helps when output repeats earlier text (editing files, echoing tool
83
  output). "Warm" means the same text had already been generated once in the session, so it is a best case.
84
 
 
 
 
 
 
 
 
85
  Suggested server flags on a quota-limited shared host: `-t 12 -fa on -lm mlock --spec-type ngram-mod`.
86
 
87
  ## Accuracy
 
82
  much slower. `ngram-mod` costs little on novel text and helps when output repeats earlier text (editing files, echoing tool
83
  output). "Warm" means the same text had already been generated once in the session, so it is a best case.
84
 
85
+ - **Runtime knobs: nothing to gain** (`benchmarks/knobs_decode.csv`, tg64, 12 threads, 3 interleaved rounds, medians):
86
+ default `--poll 50` 10.37, `--poll 0` 10.16, `--poll 100` 10.39, `--cpu-strict 1` 9.96, `-lm none` 10.20 t/s. All differences are
87
+ inside the run-to-run noise (single outliers of ~6 t/s appeared once for `--cpu-strict 1` and `-lm none`). `--prio 2` did not run
88
+ in this container. Transparent hugepages are `madvise` system-wide and were not changed.
89
+ - **`-ub` for prefill** (`benchmarks/ubatch_prefill.csv`, pp512, 12 threads, 2 rounds): 128 -> 33-35 t/s, 256 -> 37, 512 -> 40,
90
+ 1024 -> 39-41 t/s. The default 512 is already the best choice.
91
+
92
  Suggested server flags on a quota-limited shared host: `-t 12 -fa on -lm mlock --spec-type ngram-mod`.
93
 
94
  ## Accuracy
benchmarks/knobs_decode.csv ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ round,config,tg64_tps
2
+ 1,base,10.370985
3
+ 1,poll0,10.163646
4
+ 1,poll100,10.389921
5
+ 1,strict,6.191479
6
+ 1,lmnone,6.667170
7
+ 1,prio2,
8
+ 2,poll0,9.785995
9
+ 2,poll100,10.493260
10
+ 2,strict,9.958393
11
+ 2,lmnone,10.200113
12
+ 2,prio2,
13
+ 2,base,9.529010
14
+ 3,poll100,9.595270
15
+ 3,strict,9.990220
16
+ 3,lmnone,10.460284
17
+ 3,prio2,
18
+ 3,base,10.366802
19
+ 3,poll0,10.329963
20
+ KNOBS_DONE
benchmarks/ubatch_prefill.csv ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ round,ub,pp512_tps
2
+ 1,128,33.298578
3
+ 1,256,37.306879
4
+ 1,512,40.081132
5
+ 1,1024,39.106427
6
+ 2,128,34.787931
7
+ 2,256,37.228463
8
+ 2,512,39.742511
9
+ 2,1024,40.676948
10
+ UB_DONE