docs/weight_audit.md

# Weight Audit

The `weight_audit/` directory contains a structural audit script and a generated report comparing the final distilled checkpoint against `Qwen/Qwen3-1.7B-Base`.

The audit is not a behavioral benchmark. It answers a narrower question: is the checkpoint structurally intact, same-architecture, and plausibly modified by training without signs of collapse?

## What Was Checked

The audit verifies:

- Base and distilled checkpoint commits.
- Architecture and config compatibility.
- Parameter counts and tensor keys.
- Weight tying between embeddings and LM head.
- Per-tensor statistics.
- Layer-type aggregate statistics.
- Isotropy of 2D weight matrices.
- Base-vs-distilled divergence for all shared tensors.
- Sparsity, dead rows, low cosine similarity, and low SNR warnings.

## Headline Result

The final report shows:

```text
shared tensors                   : 311
tensors changed vs base          : 277 / 311
cosine similarity                : mean = 0.999991 | median = 0.999992
relative error                   : mean = 0.001093 | median = 0.001293
SNR dB                           : mean = 81.86 | median = 47.79
high-sparsity layers (>10%)      : 0
heavy-tail layers (|kurt_d|>5.0) : 0
dead-row layers                  : 0
low-cos layers (<0.95)           : 0
low-SNR layers (<20 dB)          : 0
```

## Interpretation

This is a healthy pattern for light-touch distillation:

- The architecture is unchanged.
- Most tensors changed.
- The changes are small relative to the original base weights.
- Projection matrices, embeddings, and MLP/attention layers moved.
- Some normalization tensors remained unchanged or changed only slightly.
- No layer shows obvious structural collapse.

The unchanged tensors are primarily normalization-related weights. That is not concerning by itself. It suggests the main semantic projection weights absorbed the training signal while basic scaling structure stayed stable.

## Why Isotropy Matters

The report's global isotropy score is close to zero. Near-zero average pairwise row cosine means the weight rows are not collapsing into one shared direction.

This is useful as a sanity check after KD. A collapsed model can sometimes load and produce text, but its internal geometry becomes degenerate. The audit does not show that pattern.

## What The Audit Does Not Prove

The weight audit does not prove that answers are correct, safe, or well calibrated. It should be read alongside:

- Standard benchmarks.
- Open-ended qualitative evaluations.
- SFT evaluation outputs.
- Manual regression prompts.

The audit says the checkpoint is structurally ready for downstream evaluation and release packaging.