README.md

---
license: mit
tags:
  - pytorch
  - long-range-arena
  - path-x
  - state-space-model
  - linear-recurrence
  - complex-valued-neural-network
  - sequence-classification
datasets:
  - long-range-arena
metrics:
  - accuracy
---

# PCR + Complex Screening Hybrid — Path-X (Long Range Arena)

A **phase-coherent linear recurrence (PCR)** model — a complex-diagonal
LRU/S4D-style recurrence — combined with a **non-competing complex
screening attention** module, trained on **Path-X** (Long Range Arena),
the 16,384-token binary sequence-connectivity task.

**➡️ Code, mathematical documentation, and the paper section:
[github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx](https://github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx)**

- **Task**: raw 1D token sequence in, single binary label out. No 2D
  structure, no auxiliary supervision, no handcrafted features — the same
  rule-compliant setting as S4/S5/LRU/MEGA on the LRA leaderboard.
- **Test accuracy**: **92.71 ± 0.89** over 3 seeds (n=20,000, full
  deterministic sweep); the checkpoint in this repo is the best seed, **93.50**
- **PCR-only ablation** (no screening attention): 92.54 ± 0.28 (N=2 seeds)

| Model | Path-X (test) |
|---|---|
| Transformer / Reformer / Performer / Linformer / BigBird / Luna-256 | chance (≈50) |
| S4D-Real (θ=0, no phase) | chance |
| S4-v1 | 88.10 |
| DSS | 89.72 |
| S4D-LegS | 91.95 |
| **PCR (this repo, screening ablated)** | **92.54 ± 0.28** |
| S4D-Inv | 92.80 |
| **PCR + screening hybrid (this repo)** | **92.71 ± 0.89** (best 93.50) |
| MEGA-chunk | 93.81 |
| LRU | 94.20 |
| S4 (S4-LegS) | 96.35 |
| MEGA | 97.98 |
| S5 | 98.58 |

## Architecture

```
tokens (B, 16384)
  -> linear encoder (scalar pixel -> d_model)
  -> 6 x PCRBlock:
       [BatchNorm -> PCRLayer (complex diagonal LTI, bidirectional, FFT-conv)
        -> half-GLU -> residual]
     with ComplexScreenBlock inserted after layers 2 and 4:
       [chunked (1024) non-competing complex screening attention:
        L2-normalized complex q,k -> trim-and-square gate
        (no softmax, no row-normalization) -> TanhNorm -> modReLU gate
        -> complex Hadamard -> residual]
  -> LayerNorm -> mean-pool -> linear head -> 2-class logits
```

**Design principle (Phase-Coherent Transformer / PCT)** — see
[*Complex-Valued Phase-Coherent Transformer*, Hioki,
arXiv:2605.10123](https://arxiv.org/abs/2605.10123): complex
eigenvalues implement input-independent phase rotation as coherent
long-range transport (a continuous analogue of RoPE); all input-dependent
gating, normalization, and readout stay real-valued. The screening
attention used here is the PCT paper's non-competing gate — a real-valued,
element-independent, smooth gate on L2-normalized complex query-key
similarities in place of softmax — applied on top of the PCR transport
backbone. ~94% of parameters
are complex-valued (100% within the recurrence and attention score/value
paths; the ~6% real-valued mass is the input-dependent gates, norms, and
readout — kept real by design, not by omission).

Full experimental record, ablations (phase-necessity via a real-eigenvalue
control, phase-bandwidth-vs-generalization sweep), the derivations, and the
training/eval harness are in the companion repository:
[phase-coherent-transformer-r-d/pathx](https://github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx).

## Files

- `pytorch_model.pt` — `state_dict` only (2,013,716 tensor elements across
  116 parameter tensors)
- `config.json` — architecture + optimizer config used for this run

## Usage

This repo ships raw weights, not a packaged Python module. The model code
(`PCRClassifier` / `PCRBlock` / `ComplexScreenBlock`, self-contained, torch
only) and a ready-made loading example are here:

**https://github.com/leohio/phase-coherent-transformer-r-d/tree/main/pathx**

```python
import json, torch
from pcr_screening import build_pcr_classifier   # pathx/code/pcr_screening.py

cfg = json.load(open("config.json"))["pcr_config"]
model = build_pcr_classifier(seq_len=16384, vocab=256, **cfg)
model.load_state_dict(torch.load("pytorch_model.pt", weights_only=True), strict=True)
model.eval()
```

## Training details

- Optimizer: AdamW, base lr 4.5e-4, recurrence/B/C params at 1/3 lr with no
  weight decay, cosine-hold-then-linear-decay schedule (decay starts at
  step 200,000), 250,000 steps total, batch size 32.
- Eigenvalue init: ring `|λ| ∈ [0.999, 0.9999]`, phase restricted to
  `θ ∈ [0, π/10]` — the phase bandwidth was found necessary for
  generalization (a narrower `[0, π/50]` band memorizes train perfectly
  but fails to generalize; a real-only ablation, θ=0, fails to learn at
  all).
- No dropout, weight decay 0.05, gradient clip 1.0.

## Caveats

- The hybrid result is 92.71 ± 0.89 over 3 seeds (93.50 / 92.89 / 91.75);
  the checkpoint released here is the best of the three. The PCR-only
  ablation number (92.54 ± 0.28) is averaged over 2 seeds.
- Not benchmarked beyond Path-X, LRA Text, and LRA Image; no
  task-specific hyperparameter tuning was performed for those two
  auxiliary benchmarks.