Hugging Face's logo

nkkbr
/
ViCA2-init

Video-Text-to-Text
Transformers
Safetensors
sam2
English
vica_qwen
text-generation
multimodal
vision-language
video understanding
visuospatial cognition
spatial reasoning
vlm
llava
qwen
siglip
hiera
dual-encoder
Model card Files
xet
 Community
1
How to use from the
Use from the
sam2 library
# Use SAM2 with images
import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained(nkkbr/ViCA2-init)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)
# Use SAM2 with videos
import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained(nkkbr/ViCA2-init)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>):

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Usage and Full Documentation

For detailed model description, training setup, datasets, evaluation results, and inference code, please refer to the following links:

GitHub

Hugging Face Models

Downloads last month
8
Safetensors
Model size
8B params
Tensor type
F32
·
Inference Providers NEW
Video-Text-to-Text
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including nkkbr/ViCA2-init