LWM-TF: A Wireless Foundation Model on 5G NR Time-Frequency Resource Grids
📄 Paper: arXiv:2610.21013 · 🚀 Interactive demo: wi-lab/lwm-tf-lab
LWM-TF is a 9.58 M-parameter factorized masked autoencoder pre-trained on complex 5G NR uplink (PUSCH) receive grids, taken after the FFT and before channel estimation. Each block attends first across OFDM symbols and then across resource blocks and receive antennas, and the mask follows the same two axes. Because the payload phase is unpredictable, the model reconstructs the normalized log-power of each masked token and predicts its scale separately. With the encoder frozen, a linear probe on the 256-d embedding classifies modulation, predicts CRC success and detects UE mobility, and it does so most clearly when labels are scarce.
This repository is a drop-in package: the model code (PyTorch only), the weights, an inference example, the demo data used by the Space, and the data generator that produced the pre-training corpus.
| File | Content |
|---|---|
lwm_tf/model.py |
FactorizedMAE, patchify, factorized_mask, loss (the training code's model file) |
lwm_tf/hub.py |
load_model() from a local folder or the Hub |
model.safetensors |
full MAE weights (encoder + decoder), fp32, 38 MB |
config.json |
architecture, input specification, pre-training recipe |
example_inference.py |
embeddings, a 16-shot probe, masked reconstruction |
demo_data.npz |
test set 2: embeddings, receiver statistics and labels for 24,576 slots, plus 576 raw grids (64 MB) |
dataset/ |
the data generator: 5G NR PUSCH link-level chain on Sionna 2.1 (TR 38.901 and DeepMIMO channels) |
requirements-data.txt |
versions used to generate the corpora |
Quickstart
pip install torch numpy safetensors huggingface_hub
git clone https://huggingface.co/wi-lab/lwm-tf && cd lwm-tf
import torch
from lwm_tf import load_model
model = load_model("wi-lab/lwm-tf") # or load_model(".") inside the clone
grid = torch.randn(2, 8, 14, 288) # replace with real slots, see "Input" below
grid = grid / grid.square().mean((1, 2, 3), keepdim=True).sqrt() # unit RMS per slot
z = model.embedding(grid) # [2, 256]
python example_inference.py runs the full example on the bundled demo data.
Input
| Signal | 5G NR PUSCH receive grid, one slot, after the FFT and before channel estimation |
| PHY | SIMO rank-1 CP-OFDM; 30 kHz SCS; 24 PRB (288 subcarriers, 10 MHz); 14 OFDM symbols; 4 Rx antennas; 3.5 GHz |
| Tensor | [B, 8, 14, 288] float32; channel order a0.re, a0.im, a1.re, a1.im, … |
| Normalization | each slot divided by its RMS over antennas, symbols and subcarriers |
| Tokens | one PRB (12 subcarriers × Re/Im) of one antenna in one OFDM symbol → 14 × 96 = 1,344 tokens |
| Embedding | model.embedding(grid): encoder on the unmasked grid, mean over all tokens → 256-d |
Model
- Encoder: dim 256, 8 factorized blocks (temporal attention over 14 symbols, then spectro-spatial attention over 96 antenna-PRB positions), 4 heads. Decoder: dim 128, 4 blocks.
- Objective: patch-normalized log-power reconstruction of masked tokens, plus per-patch scale heads (λ = 0.1 each). The mask keeps 7 of 14 symbols and the same 48 of 96 positions, so 75% of tokens are masked. The relative-phase and complex heads exist in the module but were not trained (weight 0).
This checkpoint
P0, seed 0, 200,000 steps on a 1,000,000-slot corpus: TR 38.901 UMa / UMi / RMa (250k each, Sionna) and DeepMIMO city_0–19 ray-traced links (250k). 837,288 slots were used for pre-training, and two DeepMIMO cities are held out for validation and two for test. The schedule follows the paper's recipe to 40k steps and then runs one warm-restart cosine cycle from 40k to 200k (batch 512).
The results in the paper are for the 40k checkpoint pre-trained on the 3GPP-only corpus. This 200k checkpoint was trained afterwards on the 4-class corpus. It has not been evaluated with the paper's protocol, and the paper's numbers do not apply to it.
Demo-protocol results (test set 2)
Frozen 256-d embedding with a logistic probe, against seven engineered receiver statistics (power moments, null/pilot/data power fractions, SNR estimate). Test set 2 holds 24,576 separately generated 3GPP slots; its 192 generation batches are split in half. K labeled slots per class are drawn from one half, and the score is on the other. Each value is the mean ± sd over 10 paired draws.
| Task (metric) | K / class | LWM-TF (this checkpoint) | Receiver statistics |
|---|---|---|---|
| Modulation, 3 classes (accuracy) | 16 | 0.871 ± 0.013 | 0.432 ± 0.015 |
| 1000 | 0.941 ± 0.001 | 0.484 ± 0.002 | |
| CRC success (AUROC) | 16 | 0.888 ± 0.024 | 0.782 ± 0.012 |
| 1000 | 0.962 ± 0.001 | 0.836 ± 0.003 | |
| Mobility ≥ 20 km/h (AUROC) | 16 | 0.847 ± 0.035 | 0.486 ± 0.012 |
| 1000 | 0.936 ± 0.002 | 0.484 ± 0.005 |
This is a demo protocol: both the labeled pool and the scored half come from test set 2. The paper fits its probes on a separate development bank.
Demo data (demo_data.npz, no pickle)
| Key | Shape | Content |
|---|---|---|
embedding |
[24576, 256] float32 | model.embedding, bf16 autocast on GPU |
receiver_stats |
[24576, 7] float32 | the seven receiver statistics |
label_* |
[24576] | scenario (0 UMa, 1 UMi, 2 RMa), modulation_order (2/4/6), crc, speed_kmh, target_snr_db, dmrs_position, los, indoor, mcs_index, distance_m, batch_id |
grids, grid_index |
[576, 8, 14, 288] float16, [576] | raw grids, 32 per scenario × modulation × mobility, and their slot indices |
meta |
JSON string | provenance (checkpoint and bank hashes) |
Data generation
dataset/ is the generator of the pre-training corpus. Each sample is one PUSCH slot of a 3GPP-compliant uplink,
simulated stage by stage. The Space ("Data generation" tab) follows
24 example slots through these stages.
| Stage | Code |
|---|---|
| Link configuration: 24 PRB at 30 kHz, DMRS additional position 0–3 | config.py · build_pusch_config, transmitter.py · build_resource_grid |
| Axis sampling: scenario, DMRS position, speed (25% static, else U(0, 120) km/h), target SNR U(−5, 30) dB, modulation | corpus.py · plan_mix, channel.py · draw_target_snr_db, link_adaptation.py · draw_modulation_orders |
| Channel: TR 38.901 UMa / UMi / RMa (Sionna) or DeepMIMO US 20 ray tracing | channel.py · build_channel, set_topology, deepmimo.py · LinkPool, synthesize |
| Noise calibration: σ² from the drawn target SNR | channel.py · noise_variance_for_snr |
| Link adaptation: LMMSE post-equalization SINR, ILLA, constrained to the drawn modulation | link_adaptation.py · select_mcs, snap_mcs_to_modulation |
| Transmit chain: TB CRC, LDPC, rate matching, scrambling, QAM, layer and RE mapping with DMRS | transmitter.py · transmit_per_mcs |
| Receiver, for labels only: LS estimate, LMMSE, LDPC decoding, CRC and BER | receiver.py · build_channel_estimator, decode_per_mcs |
| Storage: per-slot RMS normalization, float16 shards plus a label file | corpus.py · receive, write_mix_corpus |
pip install -r requirements-data.txt
python -m dataset.generate # one 16-slot UMa batch, printing every stage
# 4-class corpus (250k slots per class). Download the 20 DeepMIMO city_*_3p5 scenarios first.
export DEEPMIMO_SCENARIOS=/path/to/deepmimo_scenarios
python -m dataset.corpus OUT_DIR --per-class 250000
dataset.corpus.write_corpus writes a 3GPP-only corpus with the same chain. The code in dataset/ matches the
version that generated the corpora, with two exceptions: the DeepMIMO folder is read from $DEEPMIMO_SCENARIOS, and a
docstring no longer cites an internal analysis file.
Limitations
- All training and test data are simulated (Sionna TR 38.901 and DeepMIMO ray tracing). The model has not been validated on measured signals.
- The input format is fixed: 4 Rx antennas, 24 PRB at 30 kHz, single-layer PUSCH, QPSK to 64QAM in the test data. Other numerologies or antenna counts need re-training.
- The embedding is meant for frozen probes and few-shot tasks. It is not a receiver and does not replace channel estimation or decoding.
Citation
@article{kim2026lwmtf,
title = {{LWM-TF}: A Wireless Foundation Model on {5G NR} Time-Frequency Resource Grids},
author = {Kim, Namhyun and Alikhani, Sadjad and Alkhateeb, Ahmed},
journal = {arXiv preprint arXiv:2610.21013},
year = {2026}
}
License: MIT.
- Downloads last month
- 16