Video-Text-to-Text
Transformers
Safetensors
qwen3_vl
image-text-to-text
camera-movement
video-understanding
qwen3-vl
vggt-injection
Instructions to use ddz16/CamInject-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ddz16/CamInject-4B with Transformers:
# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ddz16/CamInject-4B") model = AutoModelForMultimodalLM.from_pretrained("ddz16/CamInject-4B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Fix usage instructions: correct model class / plugin guidance, English-only
Browse files
README.md
CHANGED
|
@@ -7,26 +7,34 @@ tags:
|
|
| 7 |
- camera-movement
|
| 8 |
- video-understanding
|
| 9 |
- qwen3-vl
|
| 10 |
-
- sft
|
| 11 |
- vggt-injection
|
| 12 |
---
|
| 13 |
|
| 14 |
# CamInject-4B
|
| 15 |
|
| 16 |
-
|
|
|
|
|
|
|
| 17 |
|
| 18 |
-
**
|
| 19 |
-
**
|
|
|
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
| 22 |
|
| 23 |
-
|
| 24 |
-
- 训练框架: [ms-swift](https://github.com/modelscope/ms-swift)
|
| 25 |
|
| 26 |
-
|
|
|
|
| 27 |
|
| 28 |
-
```
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
- camera-movement
|
| 8 |
- video-understanding
|
| 9 |
- qwen3-vl
|
|
|
|
| 10 |
- vggt-injection
|
| 11 |
---
|
| 12 |
|
| 13 |
# CamInject-4B
|
| 14 |
|
| 15 |
+
Camera-movement understanding model that **injects frozen VGGT camera tokens** into
|
| 16 |
+
`Qwen/Qwen3-VL-4B-Instruct`. Given a video, it outputs structured JSON describing every
|
| 17 |
+
camera-movement segment.
|
| 18 |
|
| 19 |
+
- **Paper**: [Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation](https://huggingface.co/papers/2608.10932)
|
| 20 |
+
- **Project page**: https://ddz16.github.io/cammotion.github.io
|
| 21 |
+
- **Code**: https://github.com/ddz16/CamDistill
|
| 22 |
|
| 23 |
+
> ⚠️ **This model cannot be loaded with plain 🤗 Transformers.** It requires a custom model type
|
| 24 |
+
> (registered via a plugin) **and** runs VGGT online to produce camera tokens. Loading it as a
|
| 25 |
+
> standard `Qwen3VLForConditionalGeneration` would not work correctly. Use the CamDistill repo.
|
| 26 |
|
| 27 |
+
## Usage
|
|
|
|
| 28 |
|
| 29 |
+
Clone the [CamDistill repo](https://github.com/ddz16/CamDistill) and clone VGGT-Omega (set `VGGT_OMEGA_REPO`, see the repo's
|
| 30 |
+
setup). CamInject runs VGGT online during inference:
|
| 31 |
|
| 32 |
+
```bash
|
| 33 |
+
VGGT_TEACHER_TYPE=vggt_omega \
|
| 34 |
+
python camera_movement_sft/infer_single.py \
|
| 35 |
+
--model ddz16/CamInject-4B \
|
| 36 |
+
--video /path/to/video.mp4 \
|
| 37 |
+
--variant caminject
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
See the repo's README for environment setup and batch evaluation.
|