REVALIDATION.md

Re-validation protocol

REVALIDATION.json records the current decision. VALIDATION.json preserves the initial run, including its unsaved Q5_1 failure and its superseded greedy-equality gate. Neither record omits a failed case.

The model is MiniCPM-V 4.7 35B-A3B Q4_K_M, SHA256 9ecf91e814d7e9adc64325b1b65f8628a9b1bb8da4f61a92869ca7489bfd888e. The fixed original-text corpus has SHA256 68c3808a113f74773f6c3c278173171358f2402d4e38e116ae6a03aca0263055. This is a numerical comparison over four 512-token windows, scoring 255 positions per window, or 1,020 positions total. It is not a representative model-quality benchmark and covers text inference only.

Model agreement

Use the same executable, model, corpus, context, batch, microbatch, cache formats, and Flash Attention setting for both commands. $BUILD_DIR is the build directory, $MODEL the local model file, and $CPU_SET an assigned set of at most 40 logical CPUs. The GPU command requires separately arranged GPU admission and an appropriate CUDA_VISIBLE_DEVICES.

CUDA_VISIBLE_DEVICES='' nice -n 15 taskset -c "$CPU_SET" \
  "$BUILD_DIR/bin/llama-perplexity" \
  -m "$MODEL" -c 512 -b 512 -ub 128 -t 40 -tb 40 \
  -fa on -ctk f16 -ctv f16 --fit off --seed 20261016 \
  --no-context-shift --verbosity 4 \
  -ngl 0 --device none --no-op-offload \
  -f validation-corpus.txt --chunks 4 \
  --kl-divergence-base "$BUILD_DIR/reference-logits.bin"

nice -n 15 taskset -c "$CPU_SET" "$BUILD_DIR/bin/llama-perplexity" \
  -m "$MODEL" -c 512 -b 512 -ub 128 -t 40 -tb 40 \
  -fa on -ctk f16 -ctv f16 --fit off --seed 20261016 \
  --no-context-shift --verbosity 4 -ngl 999 --device CUDA0 \
  -f validation-corpus.txt --chunks 4 \
  --kl-divergence-base "$BUILD_DIR/reference-logits.bin" --kl-divergence

The acceptance limits are mean KLD ≤ 0.001, same-top agreement ≥ 99.0%, and absolute relative perplexity difference ≤ 0.5%. The 99th-percentile KLD is reported without a separate cutoff. “Same top p” is the fraction of scored positions with the same highest-probability token, not a sampling top-p setting.

The upstream base file stores scaled 16-bit log probabilities rather than uncompressed FP32 logits. Its KLD estimate includes base probabilities above exp(-16); tiny negative values can result from representation and arithmetic error. Quantiles use the upstream tool's linear interpolation. The measured perplexity difference uses the base and candidate values from the same KLD run; it is more precise than subtracting the initial four-decimal PPL display.

Greedy first-divergence positions from the initial three 64-token prompts remain informational: reasoning 26, prose 12, and no divergence for code. Positions are one-based. CPU and GPU floating-point summation order can change near-tied token choices, so greedy sequences can diverge despite passing aggregate limits.

Kernel repetitions and failure evidence

Run three complete repetitions with no parameter filter:

test-backend-ops -b CUDA0 \
  -o 'MUL_MAT.*,FLASH_ATTN_EXT,SSM_CONV.*,GATED_DELTA_NET.*,RMS_NORM,ROPE,SOFT_MAX' \
  --output csv -j 1

The re-validation uses a separate copy of this upstream test harness with capture calls inserted only at comparison failures. The production executables, shared libraries, CUDA kernels, random initialization, numerical comparisons, and tolerances remain unchanged. Each repetition has a distinct capture prefix. Source and binary hashes are recorded in the current result.

Supported rows with no error are passes. Unsupported combinations are reported separately and excluded from the executed-case denominator. Unexpected process exits, missing families, incomplete output, nonfinite results, and failures outside quantized MUL_MAT edge cases fail the gate. Random inputs come from std::random_device; separate invocations generate fresh tensors.

Any allowed numerical edge case must have NMSE below twice its own threshold, an observed MUL_MAT failure fraction below 0.001, retained exact operands, and a CPU replay with enough evidence to establish its cause. CPU replay alone does not prove input conditioning or rule out a CUDA-specific arithmetic defect. An independent high-precision calculation, and where applicable the same-input comparison on another architecture, are needed to support that attribution. Observed pooled failure fractions are sample statistics, not guarantees about every quantization, tensor shape, model, or input.

The historical Q5_1 failure was 0.000740110 against 0.000500000, or 1.48022 times the threshold. Its original input was not saved. Passing subsequent random cases cannot retroactively reproduce or establish the cause of that exact failure. The release decision explicitly accounts for this limitation.

Flash Attention and relocation

The exact former crash case is:

test-backend-ops -b CUDA0 -o FLASH_ATTN_EXT \
  -p '^hsk=128,hsv=128,nh=1,nr23=\[8,1\],kv=5003,nb=4,' --output csv -j 1

It is run with the unmodified packaged test executable. All three complete repetitions also include the entire FLASH_ATTN_EXT family.

Relocation checks compare all 18 unpacked ELF files to the tested build, resolve dependencies for all six tools, and run their supported version or usage probes with LD_LIBRARY_PATH unset and a runtime-only PATH. The host has a CUDA toolkit installed; this verifies runtime library dependencies and independence from the original build directory, not a separately provisioned runtime-only OS image. No compiler is needed to launch the package.

Raw execution logs, machine-specific plans, workboard records, and captured tensor files remain private and are not package contents.