rank causality screen: rkb finals (bf16), ledger, README
Browse files
rank_causality/README.md
CHANGED
|
@@ -40,10 +40,28 @@ Held-out bits per byte are the mean of the last five census points in each windo
|
|
| 40 |
|
| 41 |
## What it shows
|
| 42 |
|
| 43 |
-
1. Swapping only the first three mixers barely moves the early erank. The erank of the stream at a block boundary follows the type of the block that *reads* it: a hub block feeding softmax sits near 10-
|
| 44 |
2. At this scale, next-byte capacity follows how many blocks are softmax, not the early erank. The two mixed arms land within .02 of each other although their early eranks differ about 2x.
|
| 45 |
3. Caption-geometry alignment with T5-XXL runs the other way: the all-hub arms align best.
|
| 46 |
|
| 47 |
-
## Follow-up
|
| 48 |
|
| 49 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
## What it shows
|
| 42 |
|
| 43 |
+
1. Swapping only the first three mixers barely moves the early erank. The erank of the stream at a block boundary follows the type of the block that *reads* it: a hub block feeding softmax sits near 10-36, and a softmax block feeding a hub sits near 110-130. It is not caused by a few outlier tokens; the low erank is spread over every token.
|
| 44 |
2. At this scale, next-byte capacity follows how many blocks are softmax, not the early erank. The two mixed arms land within .02 of each other although their early eranks differ about 2x.
|
| 45 |
3. Caption-geometry alignment with T5-XXL runs the other way: the all-hub arms align best.
|
| 46 |
|
| 47 |
+
## Follow-up: raising the erank with the mixers fixed (`rkb-*`)
|
| 48 |
|
| 49 |
+
This follow-up keeps the all-softmax arm exactly as above and adds one term to the training loss. The term rewards a higher participation ratio (a smooth effective rank) for the outputs of blocks 1-4:
|
| 50 |
+
|
| 51 |
+
```
|
| 52 |
+
loss = CE - beta * sum_{b=1..4} log PR_b, PR_b = (tr C_b)^2 / ||C_b||_F^2, C_b = covariance of block b's outputs over the step's tokens
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
| arm | early erank (blocks 1-4) | fineweb, steps 2,600-3,000 | after the switch, steps 3,600-4,000 | caption CKA vs T5-XXL | caption ridge R2 into captionbert |
|
| 56 |
+
|---|---|---|---|---|---|
|
| 57 |
+
| all softmax, no bonus | 38 / 42 | 1.995 / 1.999 | 2.273 / 2.323 | .302 / .291 | .683 / .679 |
|
| 58 |
+
| bonus, beta 0.01 (`RB01`) | 253 / 260 | 2.016 / 2.000 | 2.290 / 2.300 | .384 / .385 | .681 / .677 |
|
| 59 |
+
| bonus, beta 0.1 (`RB10`) | 392 / 391 | 2.167 / 2.139 | 2.445 / 2.449 | .395 / .402 | .642 / .655 |
|
| 60 |
+
|
| 61 |
+
- A 6x rise in early erank made no measurable difference to next-byte loss. Three of the four comparisons sit inside the noise bar; the fourth is +.021 against a .016 bar.
|
| 62 |
+
- A 10x rise cost .13-.17 bits per byte. That cost grew with the strength of the added term while the erank rose only another 1.5x, so it is most likely the added objective competing with next-byte prediction.
|
| 63 |
+
- On the caption side, more erank raised CKA with T5-XXL to the all-hub level. The linear readout stayed flat or fell slightly, so the geometry became more spread out without carrying more recoverable caption content.
|
| 64 |
+
|
| 65 |
+
**Answer at this scale:** effective rank behaves like a floor, not a dial. Too little is damage, and above the floor more did not buy capacity.
|
| 66 |
+
|
| 67 |
+
Files: `rkb-<arm>.final_bf16@step4000.pt` and `rkb_ledger.json`.
|
rank_causality/rkb-RB01-s0.final_bf16@step4000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:487dca11242992ec049e5d96f6ccc6eca6c82e0350c7e9c0b82364cea95c4599
|
| 3 |
+
size 77892117
|
rank_causality/rkb-RB01-s1.final_bf16@step4000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c56e7d4e05f275b22df9c60b4816776d7f74af70d6a7c42b9d9f128221b14b4f
|
| 3 |
+
size 77892117
|
rank_causality/rkb-RB10-s0.final_bf16@step4000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e2f27d7bfe68993ee448b8d693c9a688b029c324f17329acb701057ea54702ec
|
| 3 |
+
size 77892117
|
rank_causality/rkb-RB10-s1.final_bf16@step4000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:129c943d45306b08307f149d6a52f3f5d89f3c9c965a699bce3a6dc5e02d900e
|
| 3 |
+
size 77892117
|
rank_causality/rkb_ledger.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|