File size: 5,150 Bytes
33b7380 f7021a9 33b7380 f7021a9 33b7380 f7021a9 33b7380 f7021a9 33b7380 f7021a9 33b7380 f7021a9 33b7380 c1f3b44 33b7380 c1f3b44 33b7380 f7021a9 33b7380 f7021a9 33b7380 f7021a9 33b7380 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | ---
license: mit
tags:
- pytorch
- long-range-arena
- path-x
- state-space-model
- linear-recurrence
- complex-valued-neural-network
- sequence-classification
datasets:
- long-range-arena
metrics:
- accuracy
---
# PCR + Complex Screening Hybrid — Path-X (Long Range Arena)
A **phase-coherent linear recurrence (PCR)** model — a complex-diagonal
LRU/S4D-style recurrence — combined with a **non-competing complex
screening attention** module, trained on **Path-X** (Long Range Arena),
the 16,384-token binary sequence-connectivity task.
**➡️ Code, mathematical documentation, and the paper section:
[github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx](https://github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx)**
- **Task**: raw 1D token sequence in, single binary label out. No 2D
structure, no auxiliary supervision, no handcrafted features — the same
rule-compliant setting as S4/S5/LRU/MEGA on the LRA leaderboard.
- **Test accuracy**: **92.71 ± 0.89** over 3 seeds (n=20,000, full
deterministic sweep); the checkpoint in this repo is the best seed, **93.50**
- **PCR-only ablation** (no screening attention): 92.54 ± 0.28 (N=2 seeds)
| Model | Path-X (test) |
|---|---|
| Transformer / Reformer / Performer / Linformer / BigBird / Luna-256 | chance (≈50) |
| S4D-Real (θ=0, no phase) | chance |
| S4-v1 | 88.10 |
| DSS | 89.72 |
| S4D-LegS | 91.95 |
| **PCR (this repo, screening ablated)** | **92.54 ± 0.28** |
| S4D-Inv | 92.80 |
| **PCR + screening hybrid (this repo)** | **92.71 ± 0.89** (best 93.50) |
| MEGA-chunk | 93.81 |
| LRU | 94.20 |
| S4 (S4-LegS) | 96.35 |
| MEGA | 97.98 |
| S5 | 98.58 |
## Architecture
```
tokens (B, 16384)
-> linear encoder (scalar pixel -> d_model)
-> 6 x PCRBlock:
[BatchNorm -> PCRLayer (complex diagonal LTI, bidirectional, FFT-conv)
-> half-GLU -> residual]
with ComplexScreenBlock inserted after layers 2 and 4:
[chunked (1024) non-competing complex screening attention:
L2-normalized complex q,k -> trim-and-square gate
(no softmax, no row-normalization) -> TanhNorm -> modReLU gate
-> complex Hadamard -> residual]
-> LayerNorm -> mean-pool -> linear head -> 2-class logits
```
**Design principle (Phase-Coherent Transformer / PCT)** — see
[*Complex-Valued Phase-Coherent Transformer*, Hioki,
arXiv:2605.10123](https://arxiv.org/abs/2605.10123): complex
eigenvalues implement input-independent phase rotation as coherent
long-range transport (a continuous analogue of RoPE); all input-dependent
gating, normalization, and readout stay real-valued. The screening
attention used here is the PCT paper's non-competing gate — a real-valued,
element-independent, smooth gate on L2-normalized complex query-key
similarities in place of softmax — applied on top of the PCR transport
backbone. ~94% of parameters
are complex-valued (100% within the recurrence and attention score/value
paths; the ~6% real-valued mass is the input-dependent gates, norms, and
readout — kept real by design, not by omission).
Full experimental record, ablations (phase-necessity via a real-eigenvalue
control, phase-bandwidth-vs-generalization sweep), the derivations, and the
training/eval harness are in the companion repository:
[phase-coherent-transformer-r-d/pathx](https://github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx).
## Files
- `pytorch_model.pt` — `state_dict` only (2,013,716 tensor elements across
116 parameter tensors)
- `config.json` — architecture + optimizer config used for this run
## Usage
This repo ships raw weights, not a packaged Python module. The model code
(`PCRClassifier` / `PCRBlock` / `ComplexScreenBlock`, self-contained, torch
only) and a ready-made loading example are here:
**https://github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx**
```python
import json, torch
from pcr_screening import build_pcr_classifier # pathx/code/pcr_screening.py
cfg = json.load(open("config.json"))["pcr_config"]
model = build_pcr_classifier(seq_len=16384, vocab=256, **cfg)
model.load_state_dict(torch.load("pytorch_model.pt", weights_only=True), strict=True)
model.eval()
```
## Training details
- Optimizer: AdamW, base lr 4.5e-4, recurrence/B/C params at 1/3 lr with no
weight decay, cosine-hold-then-linear-decay schedule (decay starts at
step 200,000), 250,000 steps total, batch size 32.
- Eigenvalue init: ring `|λ| ∈ [0.999, 0.9999]`, phase restricted to
`θ ∈ [0, π/10]` — the phase bandwidth was found necessary for
generalization (a narrower `[0, π/50]` band memorizes train perfectly
but fails to generalize; a real-only ablation, θ=0, fails to learn at
all).
- No dropout, weight decay 0.05, gradient clip 1.0.
## Caveats
- The hybrid result is 92.71 ± 0.89 over 3 seeds (93.50 / 92.89 / 91.75);
the checkpoint released here is the best of the three. The PCR-only
ablation number (92.54 ± 0.28) is averaged over 2 seeds.
- Not benchmarked beyond Path-X, LRA Text, and LRA Image; no
task-specific hyperparameter tuning was performed for those two
auxiliary benchmarks.
|