README.md

metadata

license: other
tags:
  - qwen3-vl
  - spatial-reasoning
  - 3d
extra_gated_prompt: Research checkpoints from INSAIT. Request access for research use.

Qwen3-VL-3D spatial-alignment checkpoints (binned / special-token variant)

Qwen3VL3D = Qwen/Qwen3-VL-2B-Instruct + a MoGe depth encoder and a depth projector. Spatial values use binned special tokens: <dist0000..1023> (distances) and <loc0000..1023> (pixel locations), added on top of the 151669-token base vocab (padded to 153728). Distances use normal->uniform bin edges over [-3, 5], power 1.05.

NOTE: these are barrel training checkpoints (FSDP trainer state_dict), NOT AutoModel.from_pretrained-loadable. Load them through the barrel Qwen3VL3D pipeline via pretrain_config.state_dict.checkpoint_path. Each stage folder ships its full_config.yaml, model_config.yaml, and the git hash/diff/branch needed to reproduce.

Stages

folder	run	steps	num_rel_err_median	num_acc_10pct
stage1_align_special_tokens/	depth-only alignment (trains depth projector + new token rows)	2000	~0.068 (2nd-half)	~0.557
stage2_oxe_finetune/	OXE-mix finetune from stage 1 (llava/egoexo4d/bridge)	10000	~0.047 (2nd-half), 0.040 final	~0.611, 0.639 final

Judge spatial accuracy by num_rel_err_median / num_acc_10pct, not by 3d_loss.

Branch: anna_qwen3vl_3d_basic. See each folder's git.hash / git.diff for the exact code.