docs/weight_audit.md

Weight Audit

The weight_audit/ directory contains a structural audit script and a generated report comparing the final distilled checkpoint against Qwen/Qwen3-1.7B-Base.

The audit is not a behavioral benchmark. It answers a narrower question: is the checkpoint structurally intact, same-architecture, and plausibly modified by training without signs of collapse?

What Was Checked

The audit verifies:

• Base and distilled checkpoint commits.
• Architecture and config compatibility.
• Parameter counts and tensor keys.
• Weight tying between embeddings and LM head.
• Per-tensor statistics.
• Layer-type aggregate statistics.
• Isotropy of 2D weight matrices.
• Base-vs-distilled divergence for all shared tensors.
• Sparsity, dead rows, low cosine similarity, and low SNR warnings.

Headline Result

The final report shows:

shared tensors                   : 311
tensors changed vs base          : 277 / 311
cosine similarity                : mean = 0.999991 | median = 0.999992
relative error                   : mean = 0.001093 | median = 0.001293
SNR dB                           : mean = 81.86 | median = 47.79
high-sparsity layers (>10%)      : 0
heavy-tail layers (|kurt_d|>5.0) : 0
dead-row layers                  : 0
low-cos layers (<0.95)           : 0
low-SNR layers (<20 dB)          : 0

Interpretation

This is a healthy pattern for light-touch distillation:

• The architecture is unchanged.
• Most tensors changed.
• The changes are small relative to the original base weights.
• Projection matrices, embeddings, and MLP/attention layers moved.
• Some normalization tensors remained unchanged or changed only slightly.
• No layer shows obvious structural collapse.

The unchanged tensors are primarily normalization-related weights. That is not concerning by itself. It suggests the main semantic projection weights absorbed the training signal while basic scaling structure stayed stable.

Why Isotropy Matters

The report's global isotropy score is close to zero. Near-zero average pairwise row cosine means the weight rows are not collapsing into one shared direction.

This is useful as a sanity check after KD. A collapsed model can sometimes load and produce text, but its internal geometry becomes degenerate. The audit does not show that pattern.

What The Audit Does Not Prove

The weight audit does not prove that answers are correct, safe, or well calibrated. It should be read alongside:

• Standard benchmarks.
• Open-ended qualitative evaluations.
• SFT evaluation outputs.
• Manual regression prompts.

The audit says the checkpoint is structurally ready for downstream evaluation and release packaging.