QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy

Weights for QueryOcc (Lilja, Lan, Fu, Hammarstrand), CVPR 2026.

Project page · Paper (arXiv) · Code

QueryOcc learns continuous 3D semantic occupancy from multi-view camera images by supervising directly in 4D space-time: positive and negative queries are sampled along rays from point clouds observed in adjacent frames — no rendering losses, no voxelized lidar aggregation. No human annotation is used at any stage. Two arms are published here, differing only in which point clouds provide the supervision: the camera-only arm (pseudo point clouds from a vision foundation model — the paper's headline model) and the lidar-supervised arm (real lidar sweeps with per-point pseudo-semantics).

Files

File	Preset	Supervision	Sem. RayIoU	Dyn. RayIoU	Occ. RayIoU	Sem. IoU	Occ. IoU
queryocc-nusc.pth	queryocc-nusc	camera-only (pseudo)	24.25	22.45	45.98	21.51	55.06
queryocc-lidar-nusc.pth	queryocc-lidar-nusc	real lidar	23.2	19.5	48.8	20.4	56.9

Occ3D-nuScenes validation split, self-supervised protocol. 112.1 M parameters, 449 MB each.

sha256  queryocc-nusc.pth        4a54ffb927fff5cb94086a14dfc02caaa99511ab2a97170fd6a2b1ffd0601604
sha256  queryocc-lidar-nusc.pth  561112071f8c1417b466798353dadaf5c54175b2107d25f8434839145c9d9dcc

Lidar supervision trades semantic accuracy for better occupancy, which is what its accurate depth would predict.

On the camera-only numbers

The paper reports 23.6 / 21.7 / 45.2 / 21.3 / 55.0 for queryocc-nusc. This checkpoint is above the paper on all five, from a post-publication learning-rate sweep rather than any change to the method: the peak LR is 7.5e-5 instead of the paper's 5e-5, and nothing else differs. Architecture, input resolution (256×704), supervision heads and step budget are unchanged, so the comparison to the paper's baselines is unaffected. The repository's queryocc-nusc preset now defaults to 7.5e-5, and reproduces this checkpoint; pass --model.max_lr=5e-5 for the exact paper configuration.

Two caveats worth stating plainly. Occ3D-nuScenes has no held-out test split — these numbers are the validation split, which is what the benchmark and the paper report on, and which the LR sweep also selected on. And the metric carries real seed noise: three seeds of this configuration scored 23.91 / 24.24 / 24.25 semantic RayIoU (mean 24.13). The published file is the best of those three, so 24.13 is the honest expectation when retraining, and 24.25 is what this particular file scores. Every seed beat the paper's 23.6.

Usage

git clone https://github.com/LiljaAdam/queryocc && cd queryocc
./setup_uv_env.sh && source .venv/bin/activate

python queryocc/train.py queryocc-nusc --no-train --test \
    --load_ckpt_from=hf://QueryOcc/queryocc

The hf://<owner>/<repo> spec resolves <preset>.pth inside the repo, so selecting queryocc-lidar-nusc instead fetches the other file from the same URL. The checkpoint is downloaded and cached automatically, and records the preset it belongs to — loading it into a different preset is an error rather than a silent partial load. To use a manually downloaded file, pass its path instead.

Evaluation additionally requires nuScenes, the Occ3D-nuScenes labels and a CUDA toolkit (the RayIoU raycaster is JIT-compiled on first use); the repository README covers the setup. Nothing else is needed — in particular, evaluating these weights does not require the gated DINOv3 backbone checkpoint that training uses.

Add --allow_visualization --visualizer.f_viz_occ_3d_bev --visualizer.f_viz_bev_features to log BEV predictions and a PCA projection of the BEV features to Weights & Biases.

Loading the weights directly

The file is a plain torch.save payload containing only tensors and primitives, so it needs no trust in the publisher:

import torch

payload = torch.load("queryocc-nusc.pth", map_location="cpu", weights_only=True)
payload["preset"]  # 'queryocc-nusc'
payload["metrics"]  # the numbers in the table above
payload["state_dict"]  # 'net.'-prefixed network weights

Training data and intended use

Trained on nuScenes trainval: multi-view camera images, supervised either by pseudo point clouds derived from monocular metric depth and vision-foundation-model features (camera-only arm) or by lidar sweeps carrying offline per-point pseudo-semantics (lidar arm). Research artifact: a demonstration of self-supervised occupancy learning, not a validated driving component. Its outputs reflect the geometry, sensor suite and geographies of nuScenes (Boston and Singapore) and should not be relied on for safety-critical decisions. Use is subject to the nuScenes terms as well as the license below.

License

Apache 2.0, matching the code.

Citation

@inproceedings{lilja2026queryocc,
  title={QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy},
  author={Lilja, Adam and Lan, Ji and Fu, Junsheng and Hammarstrand, Lars},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={21399--21408},
  year={2026}
}