Hugging Face's logo

rpDungeon
/
instruct-mask-tools

Model card Files
xet
 Community
instruct-mask-tools / docs /EVAL_GUIDELINES.md
ToastyPigeon's picture
ToastyPigeon
Initial release: instruct-preserving masked training & merging core tools + guidelines (recovered from dev logs, sanitized)
a4df021 verified
preview code
|
Raw History Blame Contribute Delete
11.2 kB

RECOVERED 2026-05-23T21:29:56 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

Eval Guidelines

Status: PROPOSED — these are the ideas from the 2026-05-23 throughput-optimization analysis. Once we have actual benchmark numbers (smoke runs of each change against a known model) we lock in the recipe.

Companion docs:

  • INSTRUCT_MASKING_TECHNIQUES.md — masking theory + recipes
  • MERGE_MADNESS.md — community-model merge recipe cookbook
  • This doc — how to eval the outputs of the above efficiently

1. Current full-eval breakdown (baseline, 2026-05-23)

Runner: run_ifeval.py + run_vampire_eval.py + loft-t5/eval/style_quality.py, served via llama-server (llama.cpp).

Stage N Wall Per-item Why
IFEval gen 541 ~2.5 h 16.6 s serial-ish gen, cache_prompt=False, no flash-attn flag
IFEval score 541 ~30 m 3.3 s external Google evaluation_main subprocess
Style gen 40 ~45 m 67 s --parallel 1 forced (8k ctx), single-stream
Style score 40 ~3 m — SBERT paraphrase mining per-sample, no batching
Vampire gen 48 ~15 m 19 s triple-nested serial loop (weakness × scenario × variant)
Vampire score 48 ~1 m — regex; already fine
Total ~4 h IFEval is 60% of it

2. Root causes of slowness (priority order)

  1. cache_prompt: False in run_ifeval.py ~line 69. Every IFEval call re-prefills KV from scratch even though the chat-template prefix is identical. With cache_prompt: True llama-server reuses the longest matching prefix in the slot's KV. ~10-15% per call thrown away.

  2. No --flash-attn flag on llama-server launch. llama.cpp doesn't auto-enable FA for Q4_K_M on CUDA — has to be explicit. Empirically ~20-30% gen speedup on 30B-class models at 8k ctx on 3090.

  3. --parallel 4 but slots can't actually saturate. ThreadPoolExecutor submits 4 at a time, but llama-server splits the KV across slots and each slot still generates serially on the same GPUs. Need to verify VRAM headroom before bumping to --parallel 6-8 on 31B at 8k ctx.

  4. No -ctk q8_0 -ctv q8_0 (kv-cache quant). Default kv-cache is f16. Quantizing the KV to q8_0 frees ~30% KV memory → lets us bump --parallel and/or --ctx-size. Quality hit on IFEval expected negligible.

  5. Style is locked to --parallel 1 because the inline runner uses 8k ctx. Most style prompts are < 4k tokens — drop ctx to 4k and bump to --parallel 2-3, halving the 45 min. Gate with a length check; fall back to 8k+parallel-1 if any prompt exceeds 4k.

  6. Vampire is fully serial. 48 cells × 3 samples = 144 generations in a triple for loop. Parallelize cells with a ThreadPoolExecutor against existing --parallel 4 slots → ~4× speedup → ~4 min instead of 15.

  7. Per-prompt max_tokens=1536 on IFEval. Actual gen is usually 500-800 tokens. Server respects the cap, but EOS regressions (saw this on aux-loss EOS bug) make one runaway prompt hold a slot for full 1536. Worth instrumenting per-prompt token-out length to confirm.

  8. Style scoring runs SBERT paraphrase_mining per sample instead of batching all 40 outputs through one mining call. Easy 5-10× cut on style-score wall.

3. Proposed fix order (lowest-risk first)

A. cache_prompt: True + --flash-attn + -ctk q8_0 -ctv q8_0 on run_ifeval.py + the served llama-server.

  • Estimated combined: 2.5 h → ~1.4-1.6 h.
  • Risk: low.
  • Validation: re-run on a known model (V3 baseline), compare IFEval score; expect ≤ 0.5pp drift from kv-quant.

B. Bump --parallel to 6-8 after KV quant frees memory.

  • Estimated: another 1.6 h → ~1.1-1.3 h.
  • Risk: OOM if memory check is wrong; mitigate by running a 50-prompt smoke at the new parallel count first.

C. Parallelize Vampire cells with a ThreadPoolExecutor (same llama-server, existing --parallel 4).

  • 15 min → ~4 min.
  • Risk: low; scoring is already cell-local.

D. Style: drop --ctx-size to 4096, set --parallel 2.

  • 45 min → ~22 min.
  • Risk: any style prompt > 4k tokens fails; gate with a length check, fall back to 8k+parallel-1 if any prompt exceeds.

E. Batch SBERT paraphrase mining across all 40 style samples in one call.

  • 3 min → ~30 s. Trivial.

Bottom line for 31B full eval: 4 h → ~1.5-2 h with A+B+C+D (~50-60% cut). E gets us another ~2 min, essentially free.

4. E4B differences

Same scripts; smaller model (4.3 B params, single-3090). E4B IFEval bottlenecks are the same as 31B (cache_prompt, flash-attn, --parallel), so the same fixes apply with even more headroom because the model only takes ~5-7 GB at Q4_K_M, leaving plenty of room for --parallel 8 and larger KV. E4B should be in the ~30-45 min range for full IFEval after fixes (currently ~1 h).

E4B humanness scoring (score_gemma4_e4b_humanness.py) is a separate beast — runs transformers.generate serially per prompt instead of going through llama-server, which is the wrong tool. Switching E4B humanness gen to llama-server with --parallel would cut it from ~20 min to ~3 min. Same script structure as IFEval; well-trodden path.

5. GPU layout — single-GPU vs dual-GPU

31B Q4_K_M (~18.7 GB on disk):

  • Dual-GPU tensor-split (current default): model spread ~9.4 GB / GPU. Per-GPU ~14 GB headroom for KV + slots. Comfortably --parallel 4 at 8k ctx today; with -ctk q8_0 -ctv q8_0 → --parallel 6-8. Per-token gen is ~16-22% slower than single-GPU (PCIe layer hops on NVLink-less 3090 pair), but more concurrent slots → higher total throughput.
  • Single-GPU 31B: 18.7 GB leaves only ~5 GB for KV. At 8k ctx that's --parallel 1, maybe --parallel 2 if KV is q8_0 and we drop ctx to 4k. Per-token ~20% faster than tensor-split, but loses all slot-level parallelism — net loss vs. dual-GPU --parallel 4.
  • Conclusion: 31B should always run dual-GPU when it's the only model.

E4B Q4_K_M (~5-7 GB):

  • Single-GPU E4B: model 6 GB, leaves ~17 GB for KV. Comfortably --parallel 6-8 at 8k ctx with default kv, --parallel 12+ with kv-q8. Per-token fast (no PCIe tax).
  • Conclusion: E4B always single-GPU. Tensor-splitting E4B across both GPUs is worse — PCIe overhead, no concurrency gain.

6. Side-by-side eval: 2+ models simultaneously

Pattern: one llama-server per GPU, different ports. Each gets CUDA_VISIBLE_DEVICES=N. The eval runner splits prompts across the two servers and aggregates responses at the end.

Pair Feasibility Speedup vs. serial Caveat
E4B + E4B Easy ✓ ~2× (each on own GPU, no contention) None — go for it whenever ≥2 E4Bs are queued
E4B + 31B Tight but doable ~1.5-1.8× 31B alone on its GPU = --parallel 1-2 at 4k ctx + kv-q8. Slower IFEval on 31B side, but reclaims entire E4B run.
31B + 31B NOT recommended (slower than serial dual-GPU) Each needs 18.7 GB → no room for --parallel >1 or full 8k ctx. Run them back-to-back, each using dual-GPU at --parallel 6-8.

Mode decision matrix:

have N models queued:
  if all E4B:  pair them 2-at-a-time, single-GPU each (8090, 8095)
  if 1 E4B + 1 31B: e4b+31b mode — 31B reduced parallel/ctx on GPU 1, E4B on GPU 0
  if all 31B: serial — each runs dual-GPU at --parallel 6-8 (no co-residency)
  if mixed >2: prefer pairing E4B+E4B first, then E4B+31B, then any leftover 31B serial

Expected throughput gains for typical queues (post fixes A-E + side-by-side):

  • 4 E4B models eval (currently 4 h × 4 = 16 h serial): pair-up = 2 pairs × 2-3 h = **5-6 h**. ~3× cut.
  • 4 jaxxks 31B recipes (currently 4 h × 4 = 16 h serial, dual-GPU each): no side-by-side benefit → **7-8 h** serial after the IFEval+vampire+style fixes alone.
  • 2 E4B + 2 31B queued: one E4B+E4B pass (2.5 h) + two 31B serial (3.5 h each) = ~9.5 h vs. 16 h serial.

7. Implementation sketch

Wrapper script: run_eval_dual.sh (not yet written).

Inputs: MODEL_A_GGUF, MODEL_B_GGUF, LABEL_A, LABEL_B, mode (e4b+e4b, e4b+31b, serial-31b).

Behavior:

  1. Launch llama-server A with CUDA_VISIBLE_DEVICES=0 --port 8090 --no-mmap --flash-attn -ctk q8_0 -ctv q8_0 --parallel <N_A> --ctx-size <C_A> against MODEL_A_GGUF. For E4B: N_A=8, C_A=8192. For 31B-single-GPU (the e4b+31b mode): N_A=1-2, C_A=4096.
  2. Launch llama-server B with CUDA_VISIBLE_DEVICES=1 --port 8095 --parallel <N_B> --ctx-size <C_B> against MODEL_B_GGUF.
  3. Wait for both /health endpoints to return 200.
  4. Run run_ifeval_gpu0.py --port 8090 --label $LABEL_A and run_ifeval_gpu1.py --port 8095 --label $LABEL_B concurrently. (Both scripts already exist; need port/parallel-arg parameterization.)
  5. After IFEval, run style + vampire against each server in parallel.
  6. Tear down both servers.
  7. Discord ping with both labels + summary metrics.

Eval queue: eval_queue.json — list of (model_path, label, expected_size) tuples; wrapper picks pairs by the mode decision matrix.

Smoke test: 2 small known models (e.g., 2 E4B variants we already have) at 50 IFEval prompts to validate parallel run produces same scores as serial.

8. Validation protocol (before locking any change as default)

For each proposed change (A, B, C, D, E, side-by-side):

  1. Pick a known reference model with a known eval scoreboard (e.g., 31B-V3 fisher-layer-test or E4B IT-base).
  2. Run a smoke (50-100 prompts) with the change applied. Compare per-prompt latency and any output drift vs. baseline.
  3. Run the full eval (541 IFEval + style + vampire) with the change applied. Compare:
    • Wall-clock vs. baseline (the throughput claim)
    • IFEval strict prompt-level score (must be within 0.5pp of baseline; larger drift = quality regression)
    • Style metrics (h_delta, slop, rep3g — must match within noise)
    • Vampire pass-rate (regex-scored, deterministic; must match exactly)
  4. Document the actual wall-clock delta in this doc's section 9 (changelog).
  5. Lock the change in defaults only after the comparison shows the throughput gain without quality regression.

9. Changelog (verified results — fill in as tests land)

Empty pending validation runs.

10. Open questions / things to research

  • Does --flash-attn work cleanly with --tensor-split on dual-3090 at Q4_K_M? (Some flash-attn paths required compile flags in older llama.cpp builds.)
  • KV quant + Gemma4's hybrid attention — does q8_0 KV interact badly with the sliding-window full-attn layer mix? Need an A/B with same prompts.
  • For E4B+E4B co-eval, is there contention through the host on RAM bandwidth or NVMe (if model is mmap'd)? Probably negligible at Q4_K_M sizes, but worth a check.
  • Can we share a single Google IFEval scoring subprocess between two parallel runs to cut the 30 min × 2 → 30 min total? (Scorer is CPU-bound, single-threaded — could batch.)
  • For Vampire side-by-side, the conversation_*.md cards are shared assets; concurrent reads are fine but worth confirming nothing writes back.