robust/exp_4_implementation_stability/README.md

Robustness Exp 4: Implementation Stability

Status:

• Complete

Goal:

• Check whether the method is robust to implementation path.
• In the current study, the concrete axis is:
  • non-streaming whitening / recovery
  • streaming compression / aligned recovery

Source experiment:

• ablation/exp_4_streaming_vs_nonstreaming

Why this is complete:

• Logs exist for all three ratios:
  • r0.8
  • r0.6
  • r0.4
• And for all four variants:
  • nonstream/whitening_only
  • nonstream/fullseq_actlora
  • streaming/compression_only
  • streaming/recovery_aligned

Key r0.8 results:

path	variant	raw avg	mixed7	wt2	ptb	c4
nonstream	whitening_only	48.56	52.83	13.98	14.31	27.23
nonstream	fullseq_actlora	50.72	54.92	9.08	12.98	19.84
streaming	compression_only	29.40	32.06	2298.77	883.79	2725.46
streaming	recovery_aligned	38.14	40.72	15.48	35.38	51.52

Sources:

• nonstream r0p8 whitening_only mcq
• nonstream r0p8 whitening_only ppl
• nonstream r0p8 fullseq_actlora mcq
• nonstream r0p8 fullseq_actlora ppl
• streaming r0p8 compression_only mcq
• streaming r0p8 compression_only ppl
• streaming r0p8 recovery_aligned mcq
• streaming r0p8 recovery_aligned ppl

Conclusion:

• The current method is not robust to replacing the non-streaming path with the current streaming path.
• Non-streaming clearly dominates at r0.8, both before and after recovery.
• This robustness axis is already answered strongly enough; no immediate rerun is needed unless the streaming implementation changes.

Recommended use in writeup:

• Put this in the robustness section, not the core ablation section.
• The main claim is implementation sensitivity:
  • current results depend strongly on the non-streaming path.