Instructions to use karimsandroid/jordan-spectral-attention with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use karimsandroid/jordan-spectral-attention with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm
# if on a CUDA device, also pip install mlx[cuda]

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("karimsandroid/jordan-spectral-attention")

prompt = "Once upon a time in"
text = generate(model, tokenizer, prompt=prompt, verbose=True)

Transformers

How to use karimsandroid/jordan-spectral-attention with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="karimsandroid/jordan-spectral-attention")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("karimsandroid/jordan-spectral-attention", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps
LM Studio

vLLM

How to use karimsandroid/jordan-spectral-attention with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "karimsandroid/jordan-spectral-attention"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "karimsandroid/jordan-spectral-attention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/karimsandroid/jordan-spectral-attention

SGLang

How to use karimsandroid/jordan-spectral-attention with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "karimsandroid/jordan-spectral-attention" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "karimsandroid/jordan-spectral-attention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "karimsandroid/jordan-spectral-attention" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "karimsandroid/jordan-spectral-attention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

MLX LM

How to use karimsandroid/jordan-spectral-attention with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Generate some text
mlx_lm.generate --model "karimsandroid/jordan-spectral-attention" --prompt "Once upon a time"

Docker Model Runner
How to use karimsandroid/jordan-spectral-attention with Docker Model Runner:
```
docker model run hf.co/karimsandroid/jordan-spectral-attention
```

Jordan-Spectral Attention (JSA)

This repository is a timestamped public research artifact for Jordan-Spectral Attention (JSA), a spectral-shift replacement for Transformer self-attention in autoregressive language modeling experiments.

JSA replaces token-to-token softmax attention with two structured branches:

Spectral global mixing over a small cosine basis of rank R.
Causal local shift mixing over the previous k tokens.

The current experimental implementation targets the OpenAI Parameter Golf MLX training path.

Core operator

For an input sequence x ∈ R^{B×T×D}:

project the token axis into R spectral modes,
gate those modes from a pooled sequence representation,
reconstruct a global sequence signal,
add small causal local shifts,
apply a learned channel scale.

The implementation lives in:

jsa/mixer.py

The Parameter Golf integration lives in:

train/train_jsa_mlx.py

Current strongest local result

These are local MLX experiments. Official Parameter Golf reproduction is pending.

Setup	Params	Artifact	Full-val BPB	Notes
SP8192 baseline	20.73M	13.62 MB	1.9096	Local 500-step baseline
JSA full replacement, rank 32, k=2	14.11M	~10.55 MB	0.91–1.11	Seed-sensitive but strong
JSA full replacement, rank 64, k=2	14.55M	~11.74–11.78 MB	0.58–0.60	Best current local result

Key caveat: these runs used 10 downloaded train shards for local iteration and full validation over the SP8192 validation split. They should not be presented as official leaderboard results until reproduced through the official track path. Also, note: The above results are based on OpenAI parameter-golf experiments run locally.

Setup and Example command

python3 -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Get the data:

chmod +x scripts/setup_sp8192.sh
bash scripts/setup_sp8192.sh

Example Run

A short 50-step sanity check is included only to verify the standalone repo wiring; headline results use the full 500-step / full-validation runs from the original experiment logs.

RUN_ID=sanity_check \
DATA_PATH=./data/datasets/fineweb10B_sp8192 \
TOKENIZER_PATH=./data/tokenizers/fineweb_8192_bpe.model \
VOCAB_SIZE=8192 \
SEED=42 \
USE_JSA=1 \
JSA_RANK=64 \
JSA_LOCAL_K=2 \
JSA_LAST_N_LAYERS=9 \
ITERATIONS=50 \
TRAIN_BATCH_TOKENS=8192 \
VAL_BATCH_SIZE=8192 \
VAL_LOSS_EVERY=0 \
VAL_MAX_SEQS=128 \
python3 train/train_jsa_mlx.py

Test

Tested on Apple Silicon + MLX 0.31.1.

Dataset note

The SP8192 dataset used by the Parameter Golf records is downloaded from the alternate manifest:

rm -f datasets/manifest.json
MATCHED_FINEWEB_REPO_ID=kevclark/parameter-golf python3 data/cached_challenge_fineweb.py --variant sp8192 --train-shards 10

Jordan-Spectral Attention Block

Repository layout

jordan-spectral-attention/
├── jsa/                     # core JSA mixer
├── train/                   # MLX training scripts
├── configs/                 # copyable run configs
├── experiments/             # result summaries
├── figures/                 # block diagram
└── logs/                    # selected logs can be added here

Attribution statement

This repository establishes a public timestamped release of Jordan-Spectral Attention (JSA), proposed and implemented by Karimulla Saheb Naik.

License

MIT. See LICENSE.

Downloads last month: -; Downloads are not tracked for this model. How to track