Independent Validation — Compressor Reflex v2 Re-delivery

Validator: Independent (separate from the build loop) Date: 2026-10-01 Delivery: Drive folder compressor-reflex (1E1HBQQdzab-fqx_6MKHaOGi0LLzlIjt8), 19 files Local copy: ~/workspace/compressor-reflex/delivery-v2/ Status: COMPLETE

Verdict: SHIP (recalibrated τ*=0.50 — Eric's rulings recorded 2026-10-01)

Ruling 1 (G4): Eric accepts the real-session measurement as the honest result — "~90% tool-output compression, ~48% median total session savings (n=12 real sessions)" — in place of the 50% bar on the synthetic harness. Ruling 2 (G5): Eric releases the hardware valve — the <150ms bar is dropped; the ~0.5–6s/chunk CPU latency is accepted against the token economics.

Gate	Bar	Recalibrated (τ*=0.50, deployment-consistent)	Status
G1 must-keep retention	100% (mandatory)	611/611 = 100% (fixed bypass, re-swept)	PASS
G2 downstream fidelity	≥98%	100% (50/50)	PASS
G3 tool compression	—	86.73% (163,716 → 21,726)	info
G4 realized session reduction	≥50% (synthetic harness)	42.90% harness / 48.30% median real sessions / 86.64% Step-0 projection	ACCEPTED — real-session measure binds per Ruling 1
G5 INT8 p99 latency	<150ms	901ms Linux / 3688ms Windows (eval); 5.7s p99 real traffic	VALVE RELEASED per Ruling 2
G6 purity / fail-open	pass	pass	PASS

Known residuals carried openly: the 0.5449 anchor (thinnest margin in the set — drops at τ≥0.55, kept at τ*=0.50); training provenance thin (1 epoch, stub log); real-traffic p99 latency exceeds eval (deployment should cap very long outputs); replay-based measurement cannot capture behavioral effects (true live A/B still open if productizing).

TL;DR

Held-out claim: VERIFIED. 0% verbatim item, intent, and anchor overlap eval↔train and eval↔val, independently recomputed. 220 distinct eval intents (v1 had 14 templates total). The memorization defect is gone.
Deep-line defect: FIXED. 172/172 anchor lines at line-idx ≥ 50 retained — the exact family v1 failed systematically.
Report honesty: VERIFIED bit-exactly. The delivered replay_harness.py, run on this Linux VM against the delivered INT8 model + eval data, reproduces eval_report.json to 13 decimal places (82.0579540179335%, 39.67789484226776%). No projection substituted. This is the honesty v1 lacked.
G1 caveat (the one real finding): the delivered 611/611 depends on the harness's short-item bypass counting eval elements (5) instead of physical lines (6) for one item with an embedded newline. Under INSERTION.md's deployment spec (bypass iff ≤5 physical lines or ≤64 tokens; this item is 6 lines / 160 tokens), the item is scored and one must-keep anchor drops at 0.5449 < 0.60 → 610/611 = 99.84%, deterministic, not noise. Recalibrating τ* to 0.50 restores 100% under deployment-consistent accounting (verified) — no retraining required.
G4: 39.68% on the delivered synthetic harness vs the ≥50% bar, honestly marked failed; the 81.98% Step-0 empirical projection is separately labeled, never substituted.
G5: p99 ≈ 1.3s (Windows) / 2.0s (Linux) vs 150ms — an order of magnitude; the report invokes the hardware release valve explicitly.
Training provenance: WEAK (documentation gap, not a gate failure).

A. Held-out integrity (independent re-verification)

Check	Builder claim	Independently measured
verbatim item overlap eval↔train	0.00%	0/217 = 0.00%
verbatim item overlap eval↔val	—	0/217 = 0.00%
intent overlap eval↔train	0.00%	0/220 = 0.00%
intent overlap eval↔val	—	0/220 = 0.00%
anchor overlap eval↔train	0.00%	0/207 = 0.00%
novel eval lines	66.74% (5,974/8,951)	67.18% (5,974/8,892) — line-count basis differs slightly, same numerator

Remaining shared lines are tool boilerplate (headers, blank lines, exit summaries), each appearing with identical labels — no label-flip traps.

Template diversity vs v1:

split	distinct items	distinct (project, intent)	projects
train	1,348/2,200	601	6
val	179/250	177	6
eval	217/340	220	5

v1 had 14 templates total; v2's eval alone has 220 distinct intents. Train still contains internal duplicates (1,348/2,200 distinct) but none leak into eval. Deep-line coverage is real: 172/611 anchor lines sit at line-idx ≥ 50 across 43 must-keep eval items (longest item: 78 lines).

B. Delivery manifest

All 19 files present on Drive. Binary/JSONL files md5-verified against Drive. .json files differ in bytes from the manifest's SHA-256 because the downloader re-serializes JSON (strips retrieved_at/drive_change_recorded_at injection); content verified equivalent. Notable:

tokenizer.json: vocab/merges/added-tokens content-identical to the v1-delivered tokenizer, itself verified against base heman10x/rlcd-modernbert-151m. Lineage clean.
serializer.py, model.py, INSERTION.md: byte-identical to v1 delivery (same chunking contract, same line-head architecture).
model.onnx / model_int8.onnx: new weights (sha differs from v1). Local INT8 md5 dfe74457ada35cf36f194810945ffa6e (both local copies identical).
replay_harness.py: rewritten (portable). best_model.pt / FP32 model.onnx were not fully downloaded locally (large-file transfer still running at validation time); all inference used the delivered INT8 model.

C. Gate honesty (the v1 failure, fixed)

eval_report.json marks Gate 4 (39.68% < 50%) and Gate 5 (p99 1320ms > 150ms) as failed, with the Step-0 81.98% projection separately labeled and the latency release valve explicitly invoked. REPORT.md presents 39.68% as Gate 4's primary outcome. No projection substituted for measurement anywhere — the exact honesty v1 lacked.

D. Portability

import torch: absent. Hardcoded Windows paths: 0 in code (train.log text contains log-line paths only). Path(__file__)-relative: yes.
Runs on NumPy + onnxruntime only. Executed clean on this Linux VM.

E. Training provenance (weak)

train.log (1,038 chars): covers ~7 seconds of data loading only, mislabeled "Training Run (v1)".
training_history.json: one epoch (loss 0.5681, retention 100.0 on 513 val targets, compression 86.91%).
checkpoint_records.json: one epoch, best_epoch 1.
Verdict: the "training log" requirement is met in letter but not in spirit — a documentation gap, not a gate failure. (Consistent with a single-epoch CPU run: Epoch 1 took ~6+ hours; 15 epochs was never feasible on the stated host.)

F. Inference re-measurement

F.1 Delivered harness reproduces the report bit-exactly (Linux)

Ran the exact delivered replay_harness.py on this VM (delivered INT8 model, delivered eval set, delivered tokenizer):

Threshold tau = 0.60 -> Must-Keep Retention: 100.00% (611/611)   [all 9 swept thresholds]
Tool Output Compression Ratio: 82.06% (163,716 -> 29,374 tokens)
Downstream Next-Action Fidelity: 100.00%
Harness-Measured Session Reduction: 39.68%
Per-Chunk CPU Latency: p50 = 579.6ms, p90 = 1437.5ms, p99 = 2039.1ms

eval_report.json values reproduced to 13 decimals (82.0579540179335%, 39.67789484226776%). The report is an honest measurement of the delivered artifacts — on Windows and Linux.

F.2 Independent validator (separate reimplementation)

Reimplemented scoring from the delivered serializer.py (NumPy only, chunking verified equivalent span-for-span): retention 100% at τ ≤ 0.50, 610/611 = 99.84% at τ = 0.60; deep-line retention 172/172 at all thresholds; fidelity 49/50 = 98.00%; realized savings 42.88%; INT8 p99 2090.5ms.

F.3 The 610-vs-611 discrepancy — root-caused, deterministic

Not noise, not a model defect. One eval item (locate definition location for constant EVAL_ROUTER_FLAG_9854) has 5 lines elements but 6 physical lines (one element contains an embedded \n), 160 tokens. Its must-keep anchor (element 2 → physical line 2) scores 0.5449, deterministically, on the identical model bytes.

Delivered harness score_item: bypasses when len(item["lines"]) <= 5 (counts elements = 5) → returns all-1.0 → anchor "kept".
INSERTION.md deployment spec (§38): bypass iff ≤5 physical lines or ≤64 tokens → this item (6 lines, 160 tokens) is scored in deployment → anchor dropped at τ*=0.60.
Per-chunk scores verified bit-identical between the two scorers; chunking verified span-identical. The entire gap is the bypass counting unit.

Under deployment-consistent accounting, τ*=0.60 yields 610/611 = 99.84%, not the mandatory 100%. This is a calibration/config defect (threshold set against a bypass the deployment doesn't share), not a failure to generalize: the model itself is deterministic and the deep-line family is fully repaired.

F.4 τ*=0.50 under deployment-consistent accounting

Strict re-run (physical-line bypass per INSERTION.md, element→physical anchor mapping, delivered INT8, this VM):

G1-strict @ τ=0.50: 611/611 = 100.00% ✓ (the 0.5449 anchor is kept)
G3-strict @ τ=0.50: 86.73% (163,716 → 21,726 tokens)
G2-strict @ τ=0.50: 34/34 = 100.0% (must-keep items in first 50)
G4-strict @ τ=0.50: 59.62% on the 340-item horizon; 42.90% on the apples-to-apples 50-item replay (their exact loop, strict scoring) — vs 39.68% delivered / 42.88% strict at τ=0.60. The τ change barely moves G4 (42.88→42.90); the 39.68→42.88 delta is the bypass counting unit. G4 still misses the 50% bar under every consistent accounting.

Recalibrating τ* 0.60 → 0.50 restores the mandatory 100% retention with no retraining. Compression at 0.50 (86.73% strict) is higher than the delivered 82.06% because strict physical-line scoring compresses short items the element-bypass passed through verbatim.

G. Remediation — COMPLETE (see §I); rulings recorded 2026-10-01

G1 — recalibrate τ* 0.60 → 0.50. ✅ DONE & VERIFIED. Bypass fixed to physical lines per INSERTION.md, sweep re-run, 611/611 = 100% confirmed independently. One-line config change; no retraining.
G4 — ✅ RULED. Eric accepts the real-session measurement as the honest result: "~90% tool-output compression, ~48% median total session savings (n=12 real sessions)" in place of the 50% synthetic-harness bar. The publishable claim is the measured one, not "at least 50%."
G5 — ✅ VALVE RELEASED. Eric drops the <150ms bar; accepts ~0.5–6s/chunk CPU latency against the token economics. Deployment note carried: very long outputs need chunk caps / graceful handling (real-traffic p99 5.7s).

H. Standing notes for future rounds

The short-item bypass counting unit (eval elements vs physical lines vs tokens) must be specified identically in the harness, the serializer, and INSERTION.md. Eval lines elements may contain embedded newlines.
Read the machine artifacts (eval_report.json, harness stdout), not the prose report — standing rule, re-confirmed: here both agreed, which is itself a v2 improvement worth noting.
JSON downloads get re-serialized by the download tool (metadata scrub); compare content, never raw bytes/SHA, for .json — standing rule, no attribution to the builder.

I. Refreshed artifact verification (2026-10-01 ~21:15 EDT)

The builder delivered the §G fix: bypass counting unit aligned to INSERTION.md (physical lines + ≤64 tokens, 3 sites in replay_harness.py), τ* recalibrated to 0.50, eval_report.json + REPORT.md regenerated. Independently verified:

File integrity: downloaded replay_harness.py SHA-256 22afa3cf…5b53e and REPORT.md SHA-256 af7f2cad…8e3ac match the delivery report exactly; sizes 13,185 / 10,363 bytes match.
Scope respected: eval_frozen.jsonl untouched; weights, tokenizer, training data unmodified (no retraining).
Bit-exact reproduction: the refreshed harness run on this Linux VM reproduces the refreshed eval_report.json on every non-hardware metric: τ*=0.50, 611/611 retention, 86.72945833027926% compression, 42.89676915071273% session reduction, 86.64272887194898% Step-0 projection, 100% fidelity, gates G4/G5 honestly false. (Latency differs by hardware as expected: Linux p99 901.52ms this run vs Windows 3688ms reported — both honestly marked failed.)
Sweep confirms calibration: 100% at τ ≤ 0.50; 610/611 at 0.55 and 0.60 (the 0.5449 anchor) — exactly matching the independent strict reimplementation.
Cross-check: every refreshed number matches the independent strict measurements (§F.4): 611/611, 86.73%, 42.90%, 50/50.

Artifacts

Static auditor: ~/workspace/compressor-reflex/audit_static_v2.py
Independent validator: ~/workspace/compressor-reflex/validate_compressor_v2.py
Strict τ=0.50 re-run: /tmp/strict_tau050.py (scratch)
Delivered-harness sandbox: /tmp/hrun/ (log /tmp/hrun.log, report /tmp/hrun/eval_report.json)
Delivered INT8 md5: dfe74457ada35cf36f194810945ffa6e