ddz16 commited on
Commit
ec11adf
·
verified ·
1 Parent(s): 510d17f

Fix usage instructions: correct model class / plugin guidance, English-only

Browse files
Files changed (1) hide show
  1. README.md +21 -13
README.md CHANGED
@@ -7,26 +7,34 @@ tags:
7
  - camera-movement
8
  - video-understanding
9
  - qwen3-vl
10
- - sft
11
  - vggt-injection
12
  ---
13
 
14
  # CamInject-4B
15
 
16
- This repository contains the **CamInject-4B** model from the paper [Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation](https://huggingface.co/papers/2608.10932).
 
 
17
 
18
- **Project page**: https://ddz16.github.io/cammotion.github.io
19
- **GitHub**: https://github.com/ddz16/CamDistill
 
20
 
21
- Camera-movement VGGT-Direct 注入 SFT 微调模型,基于 `Qwen/Qwen3-VL-4B-Instruct`。
 
 
22
 
23
- - Checkpoint: `checkpoint-1326`
24
- - 训练框架: [ms-swift](https://github.com/modelscope/ms-swift)
25
 
26
- ## 使用
 
27
 
28
- ```python
29
- from transformers import AutoModelForCausalLM, AutoProcessor
30
- model = AutoModelForCausalLM.from_pretrained("ddz16/CamInject-4B", torch_dtype="bfloat16", device_map="auto")
31
- processor = AutoProcessor.from_pretrained("ddz16/CamInject-4B")
32
- ```
 
 
 
 
 
7
  - camera-movement
8
  - video-understanding
9
  - qwen3-vl
 
10
  - vggt-injection
11
  ---
12
 
13
  # CamInject-4B
14
 
15
+ Camera-movement understanding model that **injects frozen VGGT camera tokens** into
16
+ `Qwen/Qwen3-VL-4B-Instruct`. Given a video, it outputs structured JSON describing every
17
+ camera-movement segment.
18
 
19
+ - **Paper**: [Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation](https://huggingface.co/papers/2608.10932)
20
+ - **Project page**: https://ddz16.github.io/cammotion.github.io
21
+ - **Code**: https://github.com/ddz16/CamDistill
22
 
23
+ > ⚠️ **This model cannot be loaded with plain 🤗 Transformers.** It requires a custom model type
24
+ > (registered via a plugin) **and** runs VGGT online to produce camera tokens. Loading it as a
25
+ > standard `Qwen3VLForConditionalGeneration` would not work correctly. Use the CamDistill repo.
26
 
27
+ ## Usage
 
28
 
29
+ Clone the [CamDistill repo](https://github.com/ddz16/CamDistill) and clone VGGT-Omega (set `VGGT_OMEGA_REPO`, see the repo's
30
+ setup). CamInject runs VGGT online during inference:
31
 
32
+ ```bash
33
+ VGGT_TEACHER_TYPE=vggt_omega \
34
+ python camera_movement_sft/infer_single.py \
35
+ --model ddz16/CamInject-4B \
36
+ --video /path/to/video.mp4 \
37
+ --variant caminject
38
+ ```
39
+
40
+ See the repo's README for environment setup and batch evaluation.