| # Robustness Exp 4: Implementation Stability |
|
|
| Status: |
| - Complete |
|
|
| Goal: |
| - Check whether the method is robust to implementation path. |
| - In the current study, the concrete axis is: |
| - non-streaming whitening / recovery |
| - streaming compression / aligned recovery |
|
|
| Source experiment: |
| - [ablation/exp_4_streaming_vs_nonstreaming](../../ablation/exp_4_streaming_vs_nonstreaming) |
|
|
| Why this is complete: |
| - Logs exist for all three ratios: |
| - `r0.8` |
| - `r0.6` |
| - `r0.4` |
| - And for all four variants: |
| - `nonstream/whitening_only` |
| - `nonstream/fullseq_actlora` |
| - `streaming/compression_only` |
| - `streaming/recovery_aligned` |
|
|
| Key `r0.8` results: |
|
|
| | path | variant | raw avg | mixed7 | wt2 | ptb | c4 | |
| |---|---|---:|---:|---:|---:|---:| |
| | nonstream | whitening_only | 48.56 | 52.83 | 13.98 | 14.31 | 27.23 | |
| | nonstream | fullseq_actlora | 50.72 | 54.92 | 9.08 | 12.98 | 19.84 | |
| | streaming | compression_only | 29.40 | 32.06 | 2298.77 | 883.79 | 2725.46 | |
| | streaming | recovery_aligned | 38.14 | 40.72 | 15.48 | 35.38 | 51.52 | |
|
|
| Sources: |
| - [nonstream r0p8 whitening_only mcq](../../ablation/exp_4_streaming_vs_nonstreaming/logs/nonstream/r0p8/whitening_only/mcq.log) |
| - [nonstream r0p8 whitening_only ppl](../../ablation/exp_4_streaming_vs_nonstreaming/logs/nonstream/r0p8/whitening_only/ppl.log) |
| - [nonstream r0p8 fullseq_actlora mcq](../../ablation/exp_4_streaming_vs_nonstreaming/logs/nonstream/r0p8/fullseq_actlora/mcq.log) |
| - [nonstream r0p8 fullseq_actlora ppl](../../ablation/exp_4_streaming_vs_nonstreaming/logs/nonstream/r0p8/fullseq_actlora/ppl.log) |
| - [streaming r0p8 compression_only mcq](../../ablation/exp_4_streaming_vs_nonstreaming/logs/streaming/r0p8/compression_only/mcq.log) |
| - [streaming r0p8 compression_only ppl](../../ablation/exp_4_streaming_vs_nonstreaming/logs/streaming/r0p8/compression_only/ppl.log) |
| - [streaming r0p8 recovery_aligned mcq](../../ablation/exp_4_streaming_vs_nonstreaming/logs/streaming/r0p8/recovery_aligned/mcq.log) |
| - [streaming r0p8 recovery_aligned ppl](../../ablation/exp_4_streaming_vs_nonstreaming/logs/streaming/r0p8/recovery_aligned/ppl.log) |
|
|
| Conclusion: |
| - The current method is not robust to replacing the non-streaming path with the current streaming path. |
| - Non-streaming clearly dominates at `r0.8`, both before and after recovery. |
| - This robustness axis is already answered strongly enough; no immediate rerun is needed unless the streaming implementation changes. |
|
|
| Recommended use in writeup: |
| - Put this in the robustness section, not the core ablation section. |
| - The main claim is implementation sensitivity: |
| - current results depend strongly on the non-streaming path. |
|
|
