| bits lost to the block-16 scale, mean over channels, median over layers (312 layers) |
|
|
| modality lam=1 lam=all lam=video lam=text |
| ---------------------------------------------------------- |
| video 1.274 0.979 0.837 1.232 |
| text 1.087 0.850 1.303 0.949 |
| audio 2.085 2.031 2.327 2.144 |
|
|
| change vs no smoothing (negative = smoothing helps that modality) |
| modality lam=1 lam=all lam=video lam=text |
| ---------------------------------------------------------- |
| video -- -0.294 -0.437 -0.042 |
| text -- -0.237 +0.216 -0.138 |
| audio -- -0.053 +0.243 +0.060 |
|
|
| layers where lambda=video costs text the most, vs lambda=all: |
| transformer_blocks.33.attn.to_q text 0.691 -> 1.846 video 0.948 -> 0.756 |
| transformer_blocks.32.attn.to_k text 0.589 -> 1.718 video 0.810 -> 0.675 |
| transformer_blocks.32.attn.to_q text 0.637 -> 1.718 video 0.914 -> 0.675 |
| transformer_blocks.1.ff.net.2 text 2.941 -> 4.014 video 3.195 -> 1.438 |
| transformer_blocks.30.attn.to_k text 0.587 -> 1.590 video 0.739 -> 0.687 |
| transformer_blocks.34.attn.to_k text 0.674 -> 1.660 video 0.977 -> 0.789 |
| transformer_blocks.33.attn.to_k text 0.756 -> 1.716 video 1.004 -> 0.824 |
| transformer_blocks.35.attn.to_k text 0.705 -> 1.630 video 0.822 -> 0.577 |
|
|
