Instructions to use Chenfy0407/SparseTrajTokens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Chenfy0407/SparseTrajTokens with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Sparse Trajectory Tokens β released checkpoints
Weights for Sparse Trajectory Tokens for Metric Motion Queries in Multimodal Language Models (ICLR 2027
submission). Code and instructions: see the accompanying repository (README.md there); the dataset is in the
companion dataset repo TrajQA-B.
Every directory is a PEFT / transformers checkpoint directory as written by src/training/g1/train.py
(tokenizer and processor files included) and is loaded by src/training/g1/eval.py --ckpt <dir> on top of the
base model Qwen/Qwen3-VL-8B-Instruct (not included).
| Directory | Paper | Files | Steps |
|---|---|---|---|
checkpoints/stage1_projector/ |
Stage 1: trajectory-token projector warmed up on rule-generated motion captions | traj_encoder.pt (fp32, 85.93M params) |
26,000 |
checkpoints/stage2_token_only/ |
Stage 2: token-only QA (LoRA r128 on the LLM + projector) | adapter_model.safetensors (349.18M params), traj_encoder.pt |
8,400 |
checkpoints/stage3_final/ |
Stage 3: final model (frames + trajectory tokens) β all reported numbers | same as above | 3,507 |
checkpoints/rgb_only_arm/ |
Frames-only arm fine-tuned on the same questions (ablation) | adapter_model.safetensors |
9,002 |
checkpoints/SHA256SUMS lists every file; verify with sha256sum -c SHA256SUMS inside checkpoints/.
Results (TrajQA-B, numeric families, accuracy %)
| Model | Pooled | Held-out | Test |
|---|---|---|---|
stage3_final β frames + trajectory tokens |
82.2 | 81.7 | 82.5 |
stage3_final β trajectory tokens only (no frames) |
82.2 | 81.5 | 82.8 |
stage3_final β frames + donor tokens (control) |
29.6 | 30.6 | 28.8 |
rgb_only_arm β frames only |
48.3 | 47.8 | 48.8 |
| Qwen3-VL-8B zero-shot | 28.1 | 28.5 | 27.7 |
Usage
# from the code repository root, with the dataset placed under data/
QWEN3VL_PATH=<path to Qwen3-VL-8B-Instruct> PY=python bash scripts/eval/eval_final.sh heldout checkpoints/stage3_final
Training recipes (three stages, global batch 16, AdamW, seed 0) are in scripts/train/ of the code repository;
hyper-parameters of every run are in the dataset repo under results/training_logs/<run>/meta.json.
License
Apache License 2.0 (see LICENSE in the code repository). The weights are LoRA adapters and a projector on top of
Qwen/Qwen3-VL-8B-Instruct, which keeps its own license.
- Downloads last month
- -
Model tree for Chenfy0407/SparseTrajTokens
Base model
Qwen/Qwen3-VL-8B-Instruct