Do you have a plan to release pretrained models by Point cloud and Cross-Modality?
Hi Kulibinai,
I've been looking into the currently available models and noticed that they seem to be trained exclusively on multi-view images.
I was wondering if you have any plans on your roadmap to release pretrained models that directly accept point clouds for training and inference? Additionally, I would love to know if there are any ongoing efforts regarding cross-modality models.
Thank you for your great work, and I look forward to your response!
Hi Geonwoo,
Thanks for the interest, and apologies for the slow reply.
We released only the image-conditioned model because it gives the best performance. We had not planned to release the point-cloud version, but since we have received several requests for it, we are preparing it now — the code and weights should be out in the coming weeks.
As for ongoing work: we are not currently developing the point-cloud modality further, since it turns out to be unstable during RL fine-tuning. Our effort stays on the image-conditioned setting for now.
I'll post in this thread once the point-cloud release is up.
Best,
Dmitrii Zhemchuzhnikov
Hi Geonwoo,
Following up as promised — the release is up, and sooner than "coming weeks" suggested.
Weights
Point-cloud conditioned: https://huggingface.co/kulibinai/cadreasoner-pc
Cross-modality (point cloud + multi-view renders): https://huggingface.co/kulibinai/cadreasoner-cm
Code: https://github.com/zhemdi/CADReasoner, in the pc_cm/ directory.
So to correct my earlier answer on your second question: there is a cross-modality checkpoint, and it is now public. What I meant is that we are not actively developing that direction further.
A few notes on using them:
Both checkpoints use a different architecture from the image-conditioned model — a Qwen2-VL backbone plus a Fourier point-cloud encoder (Cadrille, in pc_cm/cadrille.py). The point embeddings are scattered into a run of pad tokens prepended to the prompt, so they will not load with a plain Qwen2VLForConditionalGeneration.from_pretrained. Use the scripts in pc_cm/, not the ones in the repository root.
python3 pc_cm/test.py --dataset --checkpoint kulibinai/cadreasoner-pc
--use_pc true --use_img false --n_points 128 --n_iters 3 --outdir
For the cross-modality variant, use kulibinai/cadreasoner-cm with --use_pc true --use_img true. Keep --n_points 128 to match training. Beyond the root Dockerfile you will also need opencv-python and rtree.
The image-conditioned model still gives the best numbers in our implementation, so please keep that in mind when comparing. If you hit problems with the released code, please let me know: zhemchuzhnikovds@my.msu.ru
Best,
Dmitrii Zhemchuzhnikov
Thanks for your help. I'll try it again!
Best,
Geonwoo Lee