# Re-validation protocol

`REVALIDATION.json` records the current decision. `VALIDATION.json` preserves the
initial run, including its unsaved Q5_1 failure and its superseded greedy-equality
gate. Neither record omits a failed case.

The model is MiniCPM-V 4.7 35B-A3B Q4_K_M, SHA256
`9ecf91e814d7e9adc64325b1b65f8628a9b1bb8da4f61a92869ca7489bfd888e`.
The fixed original-text corpus has SHA256
`68c3808a113f74773f6c3c278173171358f2402d4e38e116ae6a03aca0263055`.
This is a numerical comparison over four 512-token windows, scoring 255 positions
per window, or 1,020 positions total. It is not a representative model-quality
benchmark and covers text inference only.

## Model agreement

Use the same executable, model, corpus, context, batch, microbatch, cache formats,
and Flash Attention setting for both commands. `$BUILD_DIR` is the build
directory, `$MODEL` the local model file, and `$CPU_SET` an assigned set of at most
40 logical CPUs. The GPU command requires separately arranged GPU admission and
an appropriate `CUDA_VISIBLE_DEVICES`.

```sh
CUDA_VISIBLE_DEVICES='' nice -n 15 taskset -c "$CPU_SET" \
  "$BUILD_DIR/bin/llama-perplexity" \
  -m "$MODEL" -c 512 -b 512 -ub 128 -t 40 -tb 40 \
  -fa on -ctk f16 -ctv f16 --fit off --seed 20261016 \
  --no-context-shift --verbosity 4 \
  -ngl 0 --device none --no-op-offload \
  -f validation-corpus.txt --chunks 4 \
  --kl-divergence-base "$BUILD_DIR/reference-logits.bin"

nice -n 15 taskset -c "$CPU_SET" "$BUILD_DIR/bin/llama-perplexity" \
  -m "$MODEL" -c 512 -b 512 -ub 128 -t 40 -tb 40 \
  -fa on -ctk f16 -ctv f16 --fit off --seed 20261016 \
  --no-context-shift --verbosity 4 -ngl 999 --device CUDA0 \
  -f validation-corpus.txt --chunks 4 \
  --kl-divergence-base "$BUILD_DIR/reference-logits.bin" --kl-divergence
```

The acceptance limits are mean KLD ≤ 0.001, same-top agreement ≥ 99.0%, and
absolute relative perplexity difference ≤ 0.5%. The 99th-percentile KLD is
reported without a separate cutoff. “Same top p” is the fraction of scored
positions with the same highest-probability token, not a sampling top-p setting.

The upstream base file stores scaled 16-bit log probabilities rather than
uncompressed FP32 logits. Its KLD estimate includes base probabilities above
`exp(-16)`; tiny negative values can result from representation and arithmetic
error. Quantiles use the upstream tool's linear interpolation. The measured
perplexity difference uses the base and candidate values from the same KLD run;
it is more precise than subtracting the initial four-decimal PPL display.

Greedy first-divergence positions from the initial three 64-token prompts remain
informational: reasoning 26, prose 12, and no divergence for code. Positions are
one-based. CPU and GPU floating-point summation order can change near-tied token
choices, so greedy sequences can diverge despite passing aggregate limits.

## Kernel repetitions and failure evidence

Run three complete repetitions with no parameter filter:

```sh
test-backend-ops -b CUDA0 \
  -o 'MUL_MAT.*,FLASH_ATTN_EXT,SSM_CONV.*,GATED_DELTA_NET.*,RMS_NORM,ROPE,SOFT_MAX' \
  --output csv -j 1
```

The re-validation uses a separate copy of this upstream test harness with capture
calls inserted only at comparison failures. The production executables, shared
libraries, CUDA kernels, random initialization, numerical comparisons, and
tolerances remain unchanged. Each repetition has a distinct capture prefix.
Source and binary hashes are recorded in the current result.

Supported rows with no error are passes. Unsupported combinations are reported
separately and excluded from the executed-case denominator. Unexpected process
exits, missing families, incomplete output, nonfinite results, and failures
outside quantized MUL_MAT edge cases fail the gate. Random inputs come from
`std::random_device`; separate invocations generate fresh tensors.

Any allowed numerical edge case must have NMSE below twice its own threshold,
an observed MUL_MAT failure fraction below 0.001, retained exact operands, and a
CPU replay with enough evidence to establish its cause. CPU replay alone does
not prove input conditioning or rule out a CUDA-specific arithmetic defect.
An independent high-precision calculation, and where applicable the same-input
comparison on another architecture, are needed to support that attribution.
Observed pooled failure fractions are sample statistics, not guarantees about
every quantization, tensor shape, model, or input.

The historical Q5_1 failure was 0.000740110 against 0.000500000, or 1.48022 times
the threshold. Its original input was not saved. Passing subsequent random
cases cannot retroactively reproduce or establish the cause of that exact
failure. The release decision explicitly accounts for this limitation.

## Flash Attention and relocation

The exact former crash case is:

```sh
test-backend-ops -b CUDA0 -o FLASH_ATTN_EXT \
  -p '^hsk=128,hsv=128,nh=1,nr23=\[8,1\],kv=5003,nb=4,' --output csv -j 1
```

It is run with the unmodified packaged test executable. All three complete
repetitions also include the entire FLASH_ATTN_EXT family.

Relocation checks compare all 18 unpacked ELF files to the tested build, resolve
dependencies for all six tools, and run their supported version or usage probes
with `LD_LIBRARY_PATH` unset and a runtime-only `PATH`. The host has a CUDA
toolkit installed; this verifies runtime library dependencies and independence
from the original build directory, not a separately provisioned runtime-only OS
image. No compiler is needed to launch the package.

Raw execution logs, machine-specific plans, workboard records, and captured
tensor files remain private and are not package contents.