Concerto-Base — transformers format

Unofficial transformers port of Concerto concerto_base.pth — Base (pre-release of Utonia; inputs without color / normal allowed as zeros); Point Transformer V3 encoder, channels [48, 96, 192, 384, 512]. Loads with trust_remote_code; the concerto package is not needed. model.safetensors holds the checkpoint's state_dict under model.<original name>, converted from .pth (every tensor checked equal).

Not an official release. Concerto was created by Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, Zhuotao Tian, Naiyan Wang and Hengshuang Zhao — https://github.com/Pointcept/Concerto, paper "Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations" (NeurIPS 2025, arXiv 2510.23607).

What it is

ConcertoModel.model is the package's PointTransformerV3, built from the config stored in the checkpoint (config.json → ptv3_config) as concerto.load builds it; forward runs PointTransformerV3.forward and returns a ConcertoModelOutput (last_hidden_state: the output Point's features; point: the whole output Point). The code is the official inference code (Pointcept/Concerto @ 5b411171) vendored into flat files (concerto_model.py, concerto_transform.py, concerto_z_order.py, concerto_hilbert.py); the only edits are imports. ConcertoProcessor (processing_concerto.py, processor_config.json) wraps ConcertoFeatureExtractor, which runs concerto.transform.default() on a copy of the input arrays.

Verified against the official package (same process and seeds, V100, enable_flash=False): preprocessing, every tensor of the output Point and the features mapped back to every input point are bit-identical, in fp32 and under fp16 autocast, on an indoor and an outdoor demo scene.

Usage

import numpy as np, torch
from transformers import AutoModel, AutoProcessor

# enable_flash=False without FlashAttention (pre-Ampere GPUs), as the Concerto README suggests
model = AutoModel.from_pretrained("Concerto-Base", trust_remote_code=True, enable_flash=False).cuda().eval()
processor = AutoProcessor.from_pretrained("Concerto-Base", trust_remote_code=True)
point = processor({"coord": coord, "color": color, "normal": normal})  # (N, 3) numpy arrays
point = {k: v.cuda() if torch.is_tensor(v) else v for k, v in point.items()}
with torch.inference_mode():
    outputs = model(**point)  # outputs.last_hidden_state: features of the encoder's output points; outputs.point: the Point

• return_dict=False returns a tuple.
• Per-input-point features: walk outputs.point's pooling_parent / pooling_inverse back up and index with inverse, as in the Concerto README.
• The checkpoint config has shuffle_orders=True and GridSample keeps a random point per voxel: seed numpy and torch for reproducible outputs.
• Runtime packages beyond torch / transformers: spconv, torch_scatter, timm, addict, scipy (as the original); flash_attn when enable_flash is on.

Licensing

• Weights (Pointcept/Concerto): CC-BY-NC-4.0 (non-commercial) — LICENSE.
• Code (vendored files and the port): Apache-2.0 — LICENSE.code.
• Attribution, sources and the changes made here: NOTICE.

Citation

If you use this model, please cite the original work, as its authors ask:

@inproceedings{zhang2025concerto,
  title={Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations},
  author={Zhang, Yujia and Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Tian, Zhuotao and Wang, Naiyan and Zhao, Hengshuang},
  booktitle={NeurIPS},
  year={2025}
}

@inproceedings{wu2025sonata,
    title={Sonata: Self-Supervised Learning of Reliable Point Representations},
    author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian},
    booktitle={CVPR},
    year={2025}
}

@inproceedings{wu2024ptv3,
    title={Point Transformer V3: Simpler, Faster, Stronger},
    author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2024ppt,
    title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training},
    author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2023masked,
  title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning},
  author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang},
  journal={CVPR},
  year={2023}
}

@inproceedings{wu2022ptv2,
    title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling},
    author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang},
    booktitle={NeurIPS},
    year={2022}
}

@misc{pointcept2023,
    title={Pointcept: A Codebase for Point Cloud Perception Research},
    author={Pointcept Contributors},
    howpublished={\url{https://github.com/Pointcept/Pointcept}},
    year={2023}
}