LWM-TF: A Wireless Foundation Model on 5G NR Time-Frequency Resource Grids

📄 Paper: arXiv:2610.21013 · 🚀 Interactive demo: wi-lab/lwm-tf-lab

LWM-TF is a 9.58 M-parameter factorized masked autoencoder pre-trained on complex 5G NR uplink (PUSCH) receive grids, taken after the FFT and before channel estimation. Each block attends first across OFDM symbols and then across resource blocks and receive antennas, and the mask follows the same two axes. Because the payload phase is unpredictable, the model reconstructs the normalized log-power of each masked token and predicts its scale separately. With the encoder frozen, a linear probe on the 256-d embedding classifies modulation, predicts CRC success and detects UE mobility, and it does so most clearly when labels are scarce.

This repository is a drop-in package: the model code (PyTorch only), the weights, an inference example, the demo data used by the Space, and the data generator that produced the pre-training corpus.

File Content
lwm_tf/model.py FactorizedMAE, patchify, factorized_mask, loss (the training code's model file)
lwm_tf/hub.py load_model() from a local folder or the Hub
model.safetensors full MAE weights (encoder + decoder), fp32, 38 MB
config.json architecture, input specification, pre-training recipe
example_inference.py embeddings, a 16-shot probe, masked reconstruction
demo_data.npz test set 2: embeddings, receiver statistics and labels for 24,576 slots, plus 576 raw grids (64 MB)
dataset/ the data generator: 5G NR PUSCH link-level chain on Sionna 2.1 (TR 38.901 and DeepMIMO channels)
requirements-data.txt versions used to generate the corpora

Quickstart

pip install torch numpy safetensors huggingface_hub
git clone https://huggingface.co/wi-lab/lwm-tf && cd lwm-tf
import torch
from lwm_tf import load_model

model = load_model("wi-lab/lwm-tf")          # or load_model(".") inside the clone
grid = torch.randn(2, 8, 14, 288)             # replace with real slots, see "Input" below
grid = grid / grid.square().mean((1, 2, 3), keepdim=True).sqrt()   # unit RMS per slot
z = model.embedding(grid)                     # [2, 256]

python example_inference.py runs the full example on the bundled demo data.

Input

Signal 5G NR PUSCH receive grid, one slot, after the FFT and before channel estimation
PHY SIMO rank-1 CP-OFDM; 30 kHz SCS; 24 PRB (288 subcarriers, 10 MHz); 14 OFDM symbols; 4 Rx antennas; 3.5 GHz
Tensor [B, 8, 14, 288] float32; channel order a0.re, a0.im, a1.re, a1.im, …
Normalization each slot divided by its RMS over antennas, symbols and subcarriers
Tokens one PRB (12 subcarriers × Re/Im) of one antenna in one OFDM symbol → 14 × 96 = 1,344 tokens
Embedding model.embedding(grid): encoder on the unmasked grid, mean over all tokens → 256-d

Model

  • Encoder: dim 256, 8 factorized blocks (temporal attention over 14 symbols, then spectro-spatial attention over 96 antenna-PRB positions), 4 heads. Decoder: dim 128, 4 blocks.
  • Objective: patch-normalized log-power reconstruction of masked tokens, plus per-patch scale heads (λ = 0.1 each). The mask keeps 7 of 14 symbols and the same 48 of 96 positions, so 75% of tokens are masked. The relative-phase and complex heads exist in the module but were not trained (weight 0).

This checkpoint

P0, seed 0, 200,000 steps on a 1,000,000-slot corpus: TR 38.901 UMa / UMi / RMa (250k each, Sionna) and DeepMIMO city_0–19 ray-traced links (250k). 837,288 slots were used for pre-training, and two DeepMIMO cities are held out for validation and two for test. The schedule follows the paper's recipe to 40k steps and then runs one warm-restart cosine cycle from 40k to 200k (batch 512).

The results in the paper are for the 40k checkpoint pre-trained on the 3GPP-only corpus. This 200k checkpoint was trained afterwards on the 4-class corpus. It has not been evaluated with the paper's protocol, and the paper's numbers do not apply to it.

Demo-protocol results (test set 2)

Frozen 256-d embedding with a logistic probe, against seven engineered receiver statistics (power moments, null/pilot/data power fractions, SNR estimate). Test set 2 holds 24,576 separately generated 3GPP slots; its 192 generation batches are split in half. K labeled slots per class are drawn from one half, and the score is on the other. Each value is the mean ± sd over 10 paired draws.

Task (metric) K / class LWM-TF (this checkpoint) Receiver statistics
Modulation, 3 classes (accuracy) 16 0.871 ± 0.013 0.432 ± 0.015
1000 0.941 ± 0.001 0.484 ± 0.002
CRC success (AUROC) 16 0.888 ± 0.024 0.782 ± 0.012
1000 0.962 ± 0.001 0.836 ± 0.003
Mobility ≥ 20 km/h (AUROC) 16 0.847 ± 0.035 0.486 ± 0.012
1000 0.936 ± 0.002 0.484 ± 0.005

This is a demo protocol: both the labeled pool and the scored half come from test set 2. The paper fits its probes on a separate development bank.

Demo data (demo_data.npz, no pickle)

Key Shape Content
embedding [24576, 256] float32 model.embedding, bf16 autocast on GPU
receiver_stats [24576, 7] float32 the seven receiver statistics
label_* [24576] scenario (0 UMa, 1 UMi, 2 RMa), modulation_order (2/4/6), crc, speed_kmh, target_snr_db, dmrs_position, los, indoor, mcs_index, distance_m, batch_id
grids, grid_index [576, 8, 14, 288] float16, [576] raw grids, 32 per scenario × modulation × mobility, and their slot indices
meta JSON string provenance (checkpoint and bank hashes)

Data generation

dataset/ is the generator of the pre-training corpus. Each sample is one PUSCH slot of a 3GPP-compliant uplink, simulated stage by stage. The Space ("Data generation" tab) follows 24 example slots through these stages.

Stage Code
Link configuration: 24 PRB at 30 kHz, DMRS additional position 0–3 config.py · build_pusch_config, transmitter.py · build_resource_grid
Axis sampling: scenario, DMRS position, speed (25% static, else U(0, 120) km/h), target SNR U(−5, 30) dB, modulation corpus.py · plan_mix, channel.py · draw_target_snr_db, link_adaptation.py · draw_modulation_orders
Channel: TR 38.901 UMa / UMi / RMa (Sionna) or DeepMIMO US 20 ray tracing channel.py · build_channel, set_topology, deepmimo.py · LinkPool, synthesize
Noise calibration: σ² from the drawn target SNR channel.py · noise_variance_for_snr
Link adaptation: LMMSE post-equalization SINR, ILLA, constrained to the drawn modulation link_adaptation.py · select_mcs, snap_mcs_to_modulation
Transmit chain: TB CRC, LDPC, rate matching, scrambling, QAM, layer and RE mapping with DMRS transmitter.py · transmit_per_mcs
Receiver, for labels only: LS estimate, LMMSE, LDPC decoding, CRC and BER receiver.py · build_channel_estimator, decode_per_mcs
Storage: per-slot RMS normalization, float16 shards plus a label file corpus.py · receive, write_mix_corpus
pip install -r requirements-data.txt
python -m dataset.generate                 # one 16-slot UMa batch, printing every stage
# 4-class corpus (250k slots per class). Download the 20 DeepMIMO city_*_3p5 scenarios first.
export DEEPMIMO_SCENARIOS=/path/to/deepmimo_scenarios
python -m dataset.corpus OUT_DIR --per-class 250000

dataset.corpus.write_corpus writes a 3GPP-only corpus with the same chain. The code in dataset/ matches the version that generated the corpora, with two exceptions: the DeepMIMO folder is read from $DEEPMIMO_SCENARIOS, and a docstring no longer cites an internal analysis file.

Limitations

  • All training and test data are simulated (Sionna TR 38.901 and DeepMIMO ray tracing). The model has not been validated on measured signals.
  • The input format is fixed: 4 Rx antennas, 24 PRB at 30 kHz, single-layer PUSCH, QPSK to 64QAM in the test data. Other numerologies or antenna counts need re-training.
  • The embedding is meant for frozen probes and few-shot tasks. It is not a receiver and does not replace channel estimation or decoding.

Citation

@article{kim2026lwmtf,
  title   = {{LWM-TF}: A Wireless Foundation Model on {5G NR} Time-Frequency Resource Grids},
  author  = {Kim, Namhyun and Alikhani, Sadjad and Alkhateeb, Ahmed},
  journal = {arXiv preprint arXiv:2610.21013},
  year    = {2026}
}

License: MIT.

Downloads last month
16
Safetensors
Model size
9.58M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using wi-lab/lwm-tf 1

Paper for wi-lab/lwm-tf