metadata
license: mit
After establishing the weakness of attention, I ran a series of constellation experiments to preserve the relayed constellation behavior from cycle 1 to cycle nth.
The attention mechanism for standard attention degrades throughout the layers, roughly 8 for nearly 30% of the cosine similarity to the original.
Constellation cosine similarity remains a steadfast 99.5% cosine similarity diminishing only to 99.4% at cycle 16.
Geometric preservation is reliably contained and passed downstream through the constellation relay.
Standard attention collapses.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TEST 2: Depth stability — relay vs attention vs interleaved
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
A) Relay-only:
cycle CV CV_n eff_d cos_orig gate
1 0.0587 0.0590 119.3 0.995716 0.047
2 0.0580 0.0579 119.2 0.995005 0.047
4 0.0582 0.0591 119.2 0.994683 0.047
8 0.0595 0.0573 119.1 0.994514 0.047
12 0.0603 0.0590 119.1 0.994425 0.047
16 0.0583 0.0584 119.1 0.994436 0.047
B) Attention-only:
cycle CV CV_n eff_d cos_orig
1 0.0607 0.0597 119.3 0.990663
2 0.0724 0.0712 118.6 0.955266
4 0.0802 0.0799 117.8 0.301391
8 0.0713 0.0705 117.8 0.132908
12 0.0711 0.0671 117.8 0.093587
16 0.0699 0.0683 117.8 0.073860
C) Interleaved (attn → relay → ...):
step type CV_n eff_d cos_orig
1 attn 0.0598 119.2 0.990823
2 relay 0.0609 118.4 0.986670
4 relay 0.0605 118.4 0.985883
8 relay 0.0598 118.3 0.985115
12 relay 0.0603 118.3 0.984901
16 relay 0.0626 118.3 0.984822
The relay of constellation holds steadfast as a conduit that transfers information from A to B, but it requires speedups.
I ran it against ropes as well, similar result.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TEST 2: Depth sweep — 16 layers, all architectures
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
vanilla:
depth CV CV_n eff_d cos_orig
1 0.0590 0.0613 119.2 0.988720
2 0.0812 0.0835 118.6 0.906975
4 0.0782 0.0772 118.0 0.258551
8 0.0681 0.0678 118.0 0.124103
12 0.0672 0.0670 118.0 0.084707
16 0.0656 0.0673 118.0 0.070113
rope_std:
depth CV CV_n eff_d cos_orig
1 0.0599 0.0604 119.8 0.994533
2 0.0772 0.0784 119.6 0.937518
4 0.0715 0.0726 119.5 0.270651
8 0.0627 0.0633 119.5 0.124870
12 0.0637 0.0644 119.5 0.086343
16 0.0641 0.0635 119.5 0.072248
rope_ntk:
depth CV CV_n eff_d cos_orig
1 0.0607 0.0605 119.6 0.991296
2 0.0852 0.0831 119.3 0.909574
4 0.0745 0.0759 119.1 0.255422
8 0.0662 0.0664 119.1 0.123437
12 0.0633 0.0614 119.1 0.092321
16 0.0623 0.0634 119.1 0.076500
relay:
depth CV CV_n eff_d cos_orig
1 0.0581 0.0578 119.2 0.995456
2 0.0586 0.0577 119.2 0.994940
4 0.0592 0.0575 119.1 0.994511
8 0.0605 0.0578 119.1 0.994254
12 0.0587 0.0580 119.1 0.994219
16 0.0583 0.0591 119.1 0.994187