transformers/pr_33962/en/model_doc/sam2_video.md

SAM2 Video

Overview

SAM2 (Segment Anything Model 2) was proposed in Segment Anything in Images and Videos by Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer.

The model can be used to predict segmentation masks of any object of interest given an input image or video, and input points or bounding boxes.

The abstract from the paper is the following:

We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing a version of our model, the dataset and an interactive demo.

Tips:

• Batch & Video Support: SAM2 natively supports batch processing and seamless video segmentation, while original SAM is designed for static images and simpler one-image-at-a-time workflows.
• Accuracy & Generalization: SAM2 shows improved segmentation quality, robustness, and zero-shot generalization to new domains compared to the original SAM, especially with mixed prompts.

This model was contributed by sangbumchoi and yonigozlan. The original code can be found here.

Usage example

Video Segmentation and Tracking

SAM2's key strength is its ability to track objects across video frames. Here's how to use it for video segmentation:

Basic Video Tracking

>>> from transformers import Sam2VideoModel, Sam2VideoProcessor
from accelerate import Accelerator
>>> import torch

>>> device = Accelerator().device
>>> model = Sam2VideoModel.from_pretrained("facebook/sam2.1-hiera-tiny").to(device, dtype=torch.bfloat16)
>>> processor = Sam2VideoProcessor.from_pretrained("facebook/sam2.1-hiera-tiny")

>>> # Load video frames (example assumes you have a list of PIL Images)
>>> # video_frames = [Image.open(f"frame_{i:05d}.jpg") for i in range(num_frames)]

>>> # For this example, we'll use the video loading utility
>>> from transformers.video_utils import load_video
>>> video_url = "https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/bedroom.mp4"
>>> video_frames, _ = load_video(video_url)

>>> # Initialize video inference session
>>> inference_session = processor.init_video_session(
...     video=video_frames,
...     inference_device=device,
...     dtype=torch.bfloat16,
... )

>>> # Add click on first frame to select object
>>> ann_frame_idx = 0
>>> ann_obj_id = 1
>>> points = [[[[210, 350]]]]
>>> labels = [[[1]]]

>>> processor.add_inputs_to_inference_session(
...     inference_session=inference_session,
...     frame_idx=ann_frame_idx,
...     obj_ids=ann_obj_id,
...     input_points=points,
...     input_labels=labels,
... )

>>> # Segment the object on the first frame
>>> outputs = model(
...     inference_session=inference_session,
...     frame_idx=ann_frame_idx,
... )
>>> video_res_masks = processor.post_process_masks(
...     [outputs.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
... )[0]
>>> print(f"Segmentation shape: {video_res_masks.shape}")
Segmentation shape: torch.Size([1, 1, 480, 854])

>>> # Propagate through the entire video
>>> video_segments = {}
>>> for sam2_video_output in model.propagate_in_video_iterator(inference_session):
...     video_res_masks = processor.post_process_masks(
...         [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
...     )[0]
...     video_segments[sam2_video_output.frame_idx] = video_res_masks

>>> print(f"Tracked object through {len(video_segments)} frames")
Tracked object through 180 frames

Multi-Object Video Tracking

Track multiple objects simultaneously across video frames:

>>> # Reset for new tracking session
>>> inference_session.reset_inference_session()

>>> # Add multiple objects on the first frame
>>> ann_frame_idx = 0
>>> obj_ids = [2, 3]
>>> input_points = [[[[200, 300]], [[400, 150]]]]  # Points for two objects (batched)
>>> input_labels = [[[1], [1]]]

>>> processor.add_inputs_to_inference_session(
...     inference_session=inference_session,
...     frame_idx=ann_frame_idx,
...     obj_ids=obj_ids,
...     input_points=input_points,
...     input_labels=input_labels,
... )

>>> # Get masks for both objects on first frame
>>> outputs = model(
...     inference_session=inference_session,
...     frame_idx=ann_frame_idx,
... )

>>> # Propagate both objects through video
>>> video_segments = {}
>>> for sam2_video_output in model.propagate_in_video_iterator(inference_session):
...     video_res_masks = processor.post_process_masks(
...         [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
...     )[0]
...     video_segments[sam2_video_output.frame_idx] = {
...         obj_id: video_res_masks[i]
...         for i, obj_id in enumerate(inference_session.obj_ids)
...     }

>>> print(f"Tracked {len(inference_session.obj_ids)} objects through {len(video_segments)} frames")
Tracked 2 objects through 180 frames

Refining Video Segmentation

You can add additional clicks on any frame to refine the tracking:

>>> # Add refinement click on a later frame
>>> refine_frame_idx = 50
>>> ann_obj_id = 2  # Refining first object
>>> points = [[[[220, 280]]]]  # Additional point
>>> labels = [[[1]]]  # Positive click

>>> processor.add_inputs_to_inference_session(
...     inference_session=inference_session,
...     frame_idx=refine_frame_idx,
...     obj_ids=ann_obj_id,
...     input_points=points,
...     input_labels=labels,
... )

>>> # Re-propagate with the additional information
>>> video_segments = {}
>>> for sam2_video_output in model.propagate_in_video_iterator(inference_session):
...     video_res_masks = processor.post_process_masks(
...         [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
...     )[0]
...     video_segments[sam2_video_output.frame_idx] = video_res_masks

Streaming Video Inference

For real-time applications, SAM2 supports processing video frames as they arrive:

>>> # Initialize session for streaming
>>> inference_session = processor.init_video_session(
...     inference_device=device,
...     dtype=torch.bfloat16,
... )

>>> # Process frames one by one
>>> for frame_idx, frame in enumerate(video_frames[:10]):  # Process first 10 frames
...     inputs = processor(images=frame, device=device, return_tensors="pt")
...
...     if frame_idx == 0:
...         # Add point input on first frame
...         processor.add_inputs_to_inference_session(
...             inference_session=inference_session,
...             frame_idx=0,
...             obj_ids=1,
...             input_points=[[[[210, 350], [250, 220]]]],
...             input_labels=[[[1, 1]]],
...             original_size=inputs.original_sizes[0], # need to be provided when using streaming video inference
...         )
...
...     # Process current frame
...     sam2_video_output = model(inference_session=inference_session, frame=inputs.pixel_values[0])
...
...     video_res_masks = processor.post_process_masks(
...         [sam2_video_output.pred_masks], original_sizes=inputs.original_sizes, binarize=False
...     )[0]
...     print(f"Frame {frame_idx}: mask shape {video_res_masks.shape}")

Video Batch Processing for Multiple Objects

Track multiple objects simultaneously in video by adding them all at once:

>>> # Initialize video session
>>> inference_session = processor.init_video_session(
...     video=video_frames,
...     inference_device=device,
...     dtype=torch.bfloat16,
... )

>>> # Add multiple objects on the first frame using batch processing
>>> ann_frame_idx = 0
>>> obj_ids = [2, 3]  # Track two different objects
>>> input_points = [
...     [[[200, 300], [230, 250], [275, 175]], [[400, 150]]]
... ]  # Object 2: 3 points (2 positive, 1 negative); Object 3: 1 point
>>> input_labels = [
...     [[1, 1, 0], [1]]
... ]  # Object 2: positive, positive, negative; Object 3: positive

>>> processor.add_inputs_to_inference_session(
...     inference_session=inference_session,
...     frame_idx=ann_frame_idx,
...     obj_ids=obj_ids,
...     input_points=input_points,
...     input_labels=input_labels,
... )

>>> # Get masks for all objects on the first frame
>>> outputs = model(
...     inference_session=inference_session,
...     frame_idx=ann_frame_idx,
... )
>>> video_res_masks = processor.post_process_masks(
...     [outputs.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
... )[0]
>>> print(f"Generated masks for {video_res_masks.shape[0]} objects")
Generated masks for 2 objects

>>> # Propagate all objects through the video
>>> video_segments = {}
>>> for sam2_video_output in model.propagate_in_video_iterator(inference_session):
...     video_res_masks = processor.post_process_masks(
...         [sam2_video_output.pred_masks], original_sizes=[[inference_session.video_height, inference_session.video_width]], binarize=False
...     )[0]
...     video_segments[sam2_video_output.frame_idx] = {
...         obj_id: video_res_masks[i]
...         for i, obj_id in enumerate(inference_session.obj_ids)
...     }

>>> print(f"Tracked {len(inference_session.obj_ids)} objects through {len(video_segments)} frames")
Tracked 2 objects through 180 frames

Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SAM.

• Demo notebook 🌎 for using the model, contributed by Sangbum Choi.

Sam2VideoConfig[[transformers.Sam2VideoConfig]]

class transformers.Sam2VideoConfigtransformers.Sam2VideoConfighttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/configuration_sam2_video.py#L150[{"name": "vision_config", "val": " = None"}, {"name": "prompt_encoder_config", "val": " = None"}, {"name": "mask_decoder_config", "val": " = None"}, {"name": "initializer_range", "val": " = 0.02"}, {"name": "num_maskmem", "val": " = 7"}, {"name": "image_size", "val": " = 1024"}, {"name": "sigmoid_scale_for_mem_enc", "val": " = 20.0"}, {"name": "sigmoid_bias_for_mem_enc", "val": " = -10.0"}, {"name": "enable_occlusion_spatial_embedding", "val": " = True"}, {"name": "multimask_output_in_sam", "val": " = True"}, {"name": "multimask_min_pt_num", "val": " = 0"}, {"name": "multimask_max_pt_num", "val": " = 1"}, {"name": "multimask_output_for_tracking", "val": " = True"}, {"name": "max_object_pointers_in_encoder", "val": " = 16"}, {"name": "enable_temporal_pos_encoding_for_object_pointers", "val": " = True"}, {"name": "memory_attention_hidden_size", "val": " = 256"}, {"name": "memory_attention_num_layers", "val": " = 4"}, {"name": "memory_attention_num_attention_heads", "val": " = 1"}, {"name": "memory_attention_downsample_rate", "val": " = 1"}, {"name": "memory_attention_feed_forward_hidden_size", "val": " = 2048"}, {"name": "memory_attention_feed_forward_hidden_act", "val": " = 'relu'"}, {"name": "memory_attention_dropout", "val": " = 0.1"}, {"name": "memory_attention_rope_theta", "val": " = 10000"}, {"name": "memory_attention_rope_feat_sizes", "val": " = None"}, {"name": "memory_attention_rope_dropout", "val": " = 0.1"}, {"name": "memory_encoder_hidden_size", "val": " = 256"}, {"name": "memory_encoder_output_channels", "val": " = 64"}, {"name": "mask_downsampler_embed_dim", "val": " = 256"}, {"name": "mask_downsampler_kernel_size", "val": " = 3"}, {"name": "mask_downsampler_stride", "val": " = 2"}, {"name": "mask_downsampler_padding", "val": " = 1"}, {"name": "mask_downsampler_total_stride", "val": " = 16"}, {"name": "mask_downsampler_hidden_act", "val": " = 'gelu'"}, {"name": "memory_fuser_num_layers", "val": " = 2"}, {"name": "memory_fuser_embed_dim", "val": " = 256"}, {"name": "memory_fuser_intermediate_dim", "val": " = 1024"}, {"name": "memory_fuser_kernel_size", "val": " = 7"}, {"name": "memory_fuser_padding", "val": " = 3"}, {"name": "memory_fuser_layer_scale_init_value", "val": " = 1e-06"}, {"name": "memory_fuser_hidden_act", "val": " = 'gelu'"}, {"name": "**kwargs", "val": ""}]- vision_config (Union[dict, Sam2VisionConfig], optional) -- Dictionary of configuration options used to initialize Sam2VisionConfig.

• prompt_encoder_config (Union[dict, Sam2PromptEncoderConfig], optional) -- Dictionary of configuration options used to initialize Sam2PromptEncoderConfig.
• mask_decoder_config (Union[dict, Sam2MaskDecoderConfig], optional) -- Dictionary of configuration options used to initialize Sam2MaskDecoderConfig.
• initializer_range (float, optional, defaults to 0.02) -- Standard deviation for parameter initialization.
• num_maskmem (int, optional, defaults to 7) -- The number of memory slots for the mask memory.
• image_size (int, optional, defaults to 1024) -- The size of the input images.
• sigmoid_scale_for_mem_enc (float, optional, defaults to 20.0) -- Scale factor for the sigmoid function in the memory encoder.
• sigmoid_bias_for_mem_enc (float, optional, defaults to -10.0) -- Bias for the sigmoid function in the memory encoder.
• enable_occlusion_spatial_embedding (bool, optional, defaults to True) -- Whether to enable spatial embedding for occlusions.
• multimask_output_in_sam (bool, optional, defaults to True) -- Whether to output multiple masks from the SAM head.
• multimask_min_pt_num (int, optional, defaults to 0) -- The minimum number of points to trigger multimask output.
• multimask_max_pt_num (int, optional, defaults to 1) -- The maximum number of points to trigger multimask output.
• multimask_output_for_tracking (bool, optional, defaults to True) -- Whether to use multimask output for tracking.
• max_object_pointers_in_encoder (int, optional, defaults to 16) -- The maximum number of object pointers in the encoder.
• enable_temporal_pos_encoding_for_object_pointers (bool, optional, defaults to True) -- Whether to enable temporal positional encoding for object pointers.
• memory_attention_hidden_size (int, optional, defaults to 256) -- Dimensionality of the memory attention hidden states.
• memory_attention_num_layers (int, optional, defaults to 4) -- The number of layers in the memory attention module.
• memory_attention_num_attention_heads (int, optional, defaults to 1) -- Number of attention heads for each attention layer in the memory attention.
• memory_attention_downsample_rate (int, optional, defaults to 1) -- The downsample rate for the attention layers.
• memory_attention_feed_forward_hidden_size (int, optional, defaults to 2048) -- The dimension of the feedforward network in the memory attention module.
• memory_attention_feed_forward_hidden_act (str, optional, defaults to "relu") -- The non-linear activation function in the feedforward network in the memory attention module.
• memory_attention_dropout (float, optional, defaults to 0.1) -- The dropout rate for the memory attention module.
• memory_attention_rope_theta (float, optional, defaults to 10000) -- The Rope theta parameter.
• memory_attention_rope_feat_sizes (list[int], optional, defaults to [64, 64]) -- The feature sizes for the Rope positional encoding.
• memory_attention_rope_dropout (float, optional, defaults to 0.1) -- The dropout rate for the Rope positional encoding.
• memory_encoder_hidden_size (int, optional, defaults to 256) -- Dimensionality of the memory encoder hidden states.
• memory_encoder_output_channels (int, optional, defaults to 64) -- The number of output channels for the memory encoder.
• mask_downsampler_embed_dim (int, optional, defaults to 256) -- The dimension of the mask downsampler embedding.
• mask_downsampler_kernel_size (int, optional, defaults to 3) -- The kernel size for the mask downsampler.
• mask_downsampler_stride (int, optional, defaults to 2) -- The stride for the mask downsampler.
• mask_downsampler_padding (int, optional, defaults to 1) -- The padding for the mask downsampler.
• mask_downsampler_total_stride (int, optional, defaults to 16) -- The total stride for the mask downsampler.
• mask_downsampler_hidden_act (str, optional, defaults to "gelu") -- The non-linear activation function in the mask downsampler.
• memory_fuser_num_layers (int, optional, defaults to 2) -- The number of layers in the memory fuser.
• memory_fuser_embed_dim (int, optional, defaults to 256) -- The dimension of the embedding layer in the memory fuser.
• memory_fuser_intermediate_dim (int, optional, defaults to 1024) -- The dimension of the intermediate layer in the memory fuser.
• memory_fuser_kernel_size (int, optional, defaults to 7) -- The kernel size for the memory fuser.
• memory_fuser_padding (int, optional, defaults to 3) -- The padding for the memory fuser.
• memory_fuser_layer_scale_init_value (float, optional, defaults to 1e-06) -- The initial value for the layer scale in the memory fuser.
• memory_fuser_hidden_act (str, optional, defaults to "gelu") -- The non-linear activation function in the memory fuser.
• kwargs (optional) -- Dictionary of keyword arguments.0

Sam2Config is the configuration class to store the configuration of a Sam2Model. It is used to instantiate a SAM2 model according to the specified arguments, defining the memory attention, memory encoder, and image encoder configs. Instantiating a configuration defaults will yield a similar configuration to that of the SAM 2.1 Hiera-tiny facebook/sam2.1-hiera-tiny architecture.

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import (
...     Sam2VisionConfig,
...     Sam2PromptEncoderConfig,
...     Sam2MaskDecoderConfig,
...     Sam2Model,
... )

>>> # Initializing a Sam2Config with `"facebook/sam2.1_hiera_tiny"` style configuration
>>> configuration = Sam2config()

>>> # Initializing a Sam2Model (with random weights) from the `"facebook/sam2.1_hiera_tiny"` style configuration
>>> model = Sam2Model(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

>>> # We can also initialize a Sam2Config from a Sam2VisionConfig, Sam2PromptEncoderConfig, and Sam2MaskDecoderConfig

>>> # Initializing SAM2 vision encoder, memory attention, and memory encoder configurations
>>> vision_config = Sam2VisionConfig()
>>> prompt_encoder_config = Sam2PromptEncoderConfig()
>>> mask_decoder_config = Sam2MaskDecoderConfig()

>>> config = Sam2Config(vision_config, prompt_encoder_config, mask_decoder_config)

Sam2VideoMaskDecoderConfig[[transformers.Sam2VideoMaskDecoderConfig]]

class transformers.Sam2VideoMaskDecoderConfigtransformers.Sam2VideoMaskDecoderConfighttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/configuration_sam2_video.py#L77[{"name": "hidden_size", "val": " = 256"}, {"name": "hidden_act", "val": " = 'gelu'"}, {"name": "mlp_dim", "val": " = 2048"}, {"name": "num_hidden_layers", "val": " = 2"}, {"name": "num_attention_heads", "val": " = 8"}, {"name": "attention_downsample_rate", "val": " = 2"}, {"name": "num_multimask_outputs", "val": " = 3"}, {"name": "iou_head_depth", "val": " = 3"}, {"name": "iou_head_hidden_dim", "val": " = 256"}, {"name": "dynamic_multimask_via_stability", "val": " = True"}, {"name": "dynamic_multimask_stability_delta", "val": " = 0.05"}, {"name": "dynamic_multimask_stability_thresh", "val": " = 0.98"}, {"name": "**kwargs", "val": ""}]- hidden_size (int, optional, defaults to 256) -- Dimensionality of the hidden states.

• hidden_act (str, optional, defaults to "gelu") -- The non-linear activation function in the SAM2_VIDEO mask decoder.
• mlp_dim (int, optional, defaults to 2048) -- The dimension of the MLP in the two-way transformer.
• num_hidden_layers (int, optional, defaults to 2) -- The number of hidden layers in the two-way transformer.
• num_attention_heads (int, optional, defaults to 8) -- The number of attention heads in the two-way transformer.
• attention_downsample_rate (int, optional, defaults to 2) -- The downsample rate for the attention layers.
• num_multimask_outputs (int, optional, defaults to 3) -- The number of multimask outputs.
• iou_head_depth (int, optional, defaults to 3) -- The depth of the IoU head.
• iou_head_hidden_dim (int, optional, defaults to 256) -- The hidden dimension of the IoU head.
• dynamic_multimask_via_stability (bool, optional, defaults to True) -- Whether to use dynamic multimask via stability.
• dynamic_multimask_stability_delta (float, optional, defaults to 0.05) -- The stability delta for the dynamic multimask.
• dynamic_multimask_stability_thresh (float, optional, defaults to 0.98) -- The stability threshold for the dynamic multimask.0

This is the configuration class to store the configuration of a Sam2VideoMaskDecoder. It is used to instantiate a SAM2_VIDEO memory encoder according to the specified arguments, defining the model architecture.

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Sam2VideoPromptEncoderConfig[[transformers.Sam2VideoPromptEncoderConfig]]

class transformers.Sam2VideoPromptEncoderConfigtransformers.Sam2VideoPromptEncoderConfighttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/configuration_sam2_video.py#L25[{"name": "hidden_size", "val": " = 256"}, {"name": "image_size", "val": " = 1024"}, {"name": "patch_size", "val": " = 16"}, {"name": "mask_input_channels", "val": " = 16"}, {"name": "num_point_embeddings", "val": " = 4"}, {"name": "hidden_act", "val": " = 'gelu'"}, {"name": "layer_norm_eps", "val": " = 1e-06"}, {"name": "scale", "val": " = 1"}, {"name": "**kwargs", "val": ""}]- hidden_size (int, optional, defaults to 256) -- Dimensionality of the hidden states.

• image_size (int, optional, defaults to 1024) -- The expected output resolution of the image.
• patch_size (int, optional, defaults to 16) -- The size (resolution) of each patch.
• mask_input_channels (int, optional, defaults to 16) -- The number of channels to be fed to the MaskDecoder module.
• num_point_embeddings (int, optional, defaults to 4) -- The number of point embeddings to be used.
• hidden_act (str, optional, defaults to "gelu") -- The non-linear activation function in the encoder and pooler.
• layer_norm_eps (float, optional, defaults to 1e-06) -- The epsilon used by the layer normalization layers.
• scale (float, optional, defaults to 1) -- The scale factor for the prompt encoder.0

This is the configuration class to store the configuration of a Sam2VideoPromptEncoder. The Sam2VideoPromptEncoder module is used to encode the input 2D points and bounding boxes.

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Sam2VideoProcessor[[transformers.Sam2VideoProcessor]]

class transformers.Sam2VideoProcessortransformers.Sam2VideoProcessorhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/processing_sam2_video.py#L37[{"name": "image_processor", "val": ""}, {"name": "video_processor", "val": ""}, {"name": "target_size", "val": ": typing.Optional[int] = None"}, {"name": "point_pad_value", "val": ": int = -10"}, {"name": "**kwargs", "val": ""}]- image_processor (Sam2ImageProcessorFast) -- An instance of Sam2ImageProcessorFast.

• video_processor (Sam2VideoVideoProcessor) -- An instance of Sam2VideoVideoProcessor.
• target_size (int, optional) -- The target size (target_size, target_size) to which the image will be resized.
• point_pad_value (int, optional, defaults to -10) -- The value used for padding input points.0

Constructs a SAM2 processor which wraps a SAM2 image processor and an 2D points & Bounding boxes processor into a single processor.

Sam2VideoProcessor offers all the functionalities of Sam2ImageProcessorFast and Sam2VideoProcessor. See the docstring of call() and call() for more information.

calltransformers.Sam2VideoProcessor.callhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/processing_sam2_video.py#L67[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "segmentation_maps", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "input_points", "val": ": typing.Union[list[list[list[list[float]]]], torch.Tensor, NoneType] = None"}, {"name": "input_labels", "val": ": typing.Union[list[list[list[int]]], torch.Tensor, NoneType] = None"}, {"name": "input_boxes", "val": ": typing.Union[list[list[list[float]]], torch.Tensor, NoneType] = None"}, {"name": "original_sizes", "val": ": typing.Union[list[list[float]], torch.Tensor, NoneType] = None"}, {"name": "return_tensors", "val": ": typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None"}, {"name": "**kwargs", "val": ""}]- images (ImageInput, optional) -- The image(s) to process.

• segmentation_maps (ImageInput, optional) -- The segmentation maps to process.
• input_points (list[list[list[list[float]]]], torch.Tensor, optional) -- The points to add to the frame.
• input_labels (list[list[list[int]]], torch.Tensor, optional) -- The labels for the points.
• input_boxes (list[list[list[float]]], torch.Tensor, optional) -- The bounding boxes to add to the frame.
• original_sizes (list[list[float]], torch.Tensor, optional) -- The original sizes of the images.
• return_tensors (str or TensorType, optional) -- The type of tensors to return.
• **kwargs -- Additional keyword arguments to pass to the image processor.0A BatchEncoding with the following fields- pixel_values (torch.Tensor): The processed image(s).
• original_sizes (list[list[float]]): The original sizes of the images.
• reshaped_input_sizes (torch.Tensor): The reshaped input sizes of the images.
• labels (torch.Tensor): The processed segmentation maps (if provided).
• input_points (torch.Tensor): The processed points.
• input_labels (torch.Tensor): The processed labels.
• input_boxes (torch.Tensor): The processed bounding boxes.

This method uses Sam2VideoImageProcessorFast.__call__ method to prepare image(s) for the model. It also prepares 2D points and bounding boxes for the model if they are provided.

post_process_maskstransformers.Sam2VideoProcessor.post_process_maskshttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/processing_sam2_video.py#L483[{"name": "masks", "val": ""}, {"name": "original_sizes", "val": ""}, {"name": "mask_threshold", "val": " = 0.0"}, {"name": "binarize", "val": " = True"}, {"name": "max_hole_area", "val": " = 0.0"}, {"name": "max_sprinkle_area", "val": " = 0.0"}, {"name": "apply_non_overlapping_constraints", "val": " = False"}, {"name": "**kwargs", "val": ""}]- masks (Union[List[torch.Tensor], List[np.ndarray]]) -- Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format.

• original_sizes (Union[torch.Tensor, List[Tuple[int,int]]]) -- The original sizes of each image before it was resized to the model's expected input shape, in (height, width) format.
• mask_threshold (float, optional, defaults to 0.0) -- Threshold for binarization and post-processing operations.
• binarize (bool, optional, defaults to True) -- Whether to binarize the masks.
• max_hole_area (float, optional, defaults to 0.0) -- The maximum area of a hole to fill.
• max_sprinkle_area (float, optional, defaults to 0.0) -- The maximum area of a sprinkle to fill.
• apply_non_overlapping_constraints (bool, optional, defaults to False) -- Whether to apply non-overlapping constraints to the masks.0(torch.Tensor)Batched masks in batch_size, num_channels, height, width) format, where (height, width) is given by original_size.

Remove padding and upscale masks to the original image size.

init_video_sessiontransformers.Sam2VideoProcessor.init_video_sessionhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/processing_sam2_video.py#L529[{"name": "video", "val": ": typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "inference_device", "val": ": typing.Union[str, ForwardRef('torch.device')] = 'cpu'"}, {"name": "inference_state_device", "val": ": typing.Union[str, ForwardRef('torch.device')] = None"}, {"name": "processing_device", "val": ": typing.Union[str, ForwardRef('torch.device')] = None"}, {"name": "video_storage_device", "val": ": typing.Union[str, ForwardRef('torch.device')] = None"}, {"name": "max_vision_features_cache_size", "val": ": int = 1"}, {"name": "dtype", "val": ": dtype = torch.float32"}]- video (VideoInput, optional) -- The video to process. No need to provide when streaming.

• inference_device (str or torch.device, optional, defaults to "cpu") -- The device to use for inference.
• inference_state_device (str or torch.device, optional) -- The device to store the inference state on.
• processing_device (str or torch.device, optional) -- The device to use for video processing.
• video_storage_device (str or torch.device, optional) -- The device to store the processed video frames on.
• max_vision_features_cache_size (int, optional, defaults to 1) -- The maximum number of vision features to cache.
• dtype (torch.dtype, optional, defaults to torch.float32) -- The torch dtype to use for the whole session.0

Initializes a video session for inference. If a video is provided (async inference), the video will be processed and stored on the video_storage_device.

add_inputs_to_inference_sessiontransformers.Sam2VideoProcessor.add_inputs_to_inference_sessionhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/processing_sam2_video.py#L582[{"name": "inference_session", "val": ": Sam2VideoInferenceSession"}, {"name": "frame_idx", "val": ": int"}, {"name": "obj_ids", "val": ": typing.Union[list[int], int]"}, {"name": "input_points", "val": ": typing.Union[list[list[list[list[float]]]], torch.Tensor, NoneType] = None"}, {"name": "input_labels", "val": ": typing.Union[list[list[list[int]]], torch.Tensor, NoneType] = None"}, {"name": "input_boxes", "val": ": typing.Union[list[list[list[float]]], torch.Tensor, NoneType] = None"}, {"name": "input_masks", "val": ": typing.Union[numpy.ndarray, torch.Tensor, list[numpy.ndarray], list[torch.Tensor], NoneType] = None"}, {"name": "original_size", "val": ": typing.Optional[tuple[int, int]] = None"}, {"name": "clear_old_inputs", "val": ": bool = True"}]- inference_session (Sam2VideoInferenceSession) -- The inference session for the video.

• frame_idx (int) -- The index of the frame to process.
• obj_ids (list[int] or int) -- The object ID(s) to associate with the points or box. These can be any integers and can be reused later on to specify an object.
• input_points (list[list[list[list[float]]]], torch.Tensor, optional) -- The points to add to the frame.
• input_labels (list[list[list[int]]], torch.Tensor, optional) -- The labels for the points.
• input_boxes (list[list[list[float]]], torch.Tensor, optional) -- The bounding boxes to add to the frame.
• input_masks (np.ndarray, torch.Tensor, list[np.ndarray], or list[torch.Tensor], optional) -- The mask(s) to add to the frame.
• original_size (tuple[int, int], optional) -- The original size of the video. Provide when streaming.
• clear_old_inputs (bool, optional, defaults to True) -- Whether to clear old inputs for the object.0

Process new points, boxes, or masks for a video frame and add them to the inference session.

Sam2VideoVideoProcessor[[transformers.Sam2VideoVideoProcessor]]

class transformers.Sam2VideoVideoProcessortransformers.Sam2VideoVideoProcessorhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/video_processing_sam2_video.py#L29[{"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.VideosKwargs]"}]

post_process_maskstransformers.Sam2VideoVideoProcessor.post_process_maskshttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/video_processing_sam2_video.py#L59[{"name": "masks", "val": ""}, {"name": "original_sizes", "val": ""}, {"name": "reshaped_input_sizes", "val": ""}, {"name": "mask_threshold", "val": " = 0.0"}, {"name": "binarize", "val": " = True"}, {"name": "pad_size", "val": " = None"}]- masks (Union[List[torch.Tensor], List[np.ndarray]]) -- Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format.

• original_sizes (Union[torch.Tensor, List[Tuple[int,int]]]) -- The original sizes of each image before it was resized to the model's expected input shape, in (height, width) format.
• reshaped_input_sizes (Union[torch.Tensor, List[Tuple[int,int]]]) -- The size of each image as it is fed to the model, in (height, width) format. Used to remove padding.
• mask_threshold (float, optional, defaults to 0.0) -- The threshold to use for binarizing the masks.
• binarize (bool, optional, defaults to True) -- Whether to binarize the masks.
• pad_size (int, optional, defaults to self.pad_size) -- The target size the images were padded to before being passed to the model. If None, the target size is assumed to be the processor's pad_size.0(torch.Tensor)Batched masks in batch_size, num_channels, height, width) format, where (height, width) is given by original_size.

Remove padding and upscale masks to the original image size.

Sam2VideoInferenceSession[[transformers.Sam2VideoInferenceSession]]

class transformers.Sam2VideoInferenceSessiontransformers.Sam2VideoInferenceSessionhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L100[{"name": "video", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "video_height", "val": ": typing.Optional[int] = None"}, {"name": "video_width", "val": ": typing.Optional[int] = None"}, {"name": "inference_device", "val": ": typing.Union[torch.device, str] = 'cpu'"}, {"name": "inference_state_device", "val": ": typing.Union[torch.device, str] = 'cpu'"}, {"name": "video_storage_device", "val": ": typing.Union[torch.device, str] = 'cpu'"}, {"name": "dtype", "val": ": typing.Union[torch.dtype, str] = 'float32'"}, {"name": "max_vision_features_cache_size", "val": ": int = 1"}]- video (torch.FloatTensor, optional) -- The video to process. No need to provide when streaming.

• video_height (int, optional) -- The height of the video.
• video_width (int, optional) -- The width of the video.
• inference_device (torch.device, optional, defaults to "cpu") -- The device to use for inference.
• inference_state_device (torch.device, optional, defaults to "cpu") -- The device to store the inference state on.
• video_storage_device (torch.device, optional, defaults to "cpu") -- The device to store the video on.
• dtype (torch.dtype, optional, defaults to "float32") -- The dtype to use for the video.
• max_vision_features_cache_size (int, optional, defaults to 1) -- The maximum number of vision features to cache.0

Manages video inference session parameters, state and cache.

add_mask_inputstransformers.Sam2VideoInferenceSession.add_mask_inputshttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L220[{"name": "obj_idx", "val": ": int"}, {"name": "frame_idx", "val": ": int"}, {"name": "inputs", "val": ": Tensor"}] Add mask inputs with automatic device placement.

add_new_frametransformers.Sam2VideoInferenceSession.add_new_framehttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L295[{"name": "pixel_values", "val": ": Tensor"}, {"name": "frame_idx", "val": ": typing.Optional[int] = None"}] Add new frame with automatic device placement.

add_point_inputstransformers.Sam2VideoInferenceSession.add_point_inputshttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L206[{"name": "obj_idx", "val": ": int"}, {"name": "frame_idx", "val": ": int"}, {"name": "inputs", "val": ": dict"}] Add point inputs with automatic device placement.

get_frametransformers.Sam2VideoInferenceSession.get_framehttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L311[{"name": "frame_idx", "val": ": int"}] Get frame from video.

get_obj_numtransformers.Sam2VideoInferenceSession.get_obj_numhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L201[] Get the total number of unique object ids received so far in this session.

get_outputtransformers.Sam2VideoInferenceSession.get_outputhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L268[{"name": "obj_idx", "val": ": int"}, {"name": "frame_idx", "val": ": int"}, {"name": "output_key", "val": ": str"}, {"name": "is_conditioning_frame", "val": ": bool = True"}]- obj_idx (int) -- The index of the object.

• frame_idx (int) -- The index of the frame.
• output_key (str) -- The key of the output.
• is_conditioning_frame (bool) -- Whether the output is for a conditioning frame.0

Get output with smart device management.

obj_id_to_idxtransformers.Sam2VideoInferenceSession.obj_id_to_idxhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L175[{"name": "obj_id", "val": ": int"}] Map object ID to index, creating new entry if needed.

obj_idx_to_idtransformers.Sam2VideoInferenceSession.obj_idx_to_idhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L197[{"name": "obj_idx", "val": ": int"}] Map model-side object index to client-side object id.

remove_mask_inputstransformers.Sam2VideoInferenceSession.remove_mask_inputshttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L226[{"name": "obj_idx", "val": ": int"}, {"name": "frame_idx", "val": ": int"}] Remove mask inputs.

remove_point_inputstransformers.Sam2VideoInferenceSession.remove_point_inputshttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L216[{"name": "obj_idx", "val": ": int"}, {"name": "frame_idx", "val": ": int"}] Remove point inputs.

reset_inference_sessiontransformers.Sam2VideoInferenceSession.reset_inference_sessionhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L327[] Reset tracking data and cache.

reset_tracking_datatransformers.Sam2VideoInferenceSession.reset_tracking_datahttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L315[] Reset tracking data but keep cache.

store_outputtransformers.Sam2VideoInferenceSession.store_outputhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L231[{"name": "obj_idx", "val": ": int"}, {"name": "frame_idx", "val": ": int"}, {"name": "output_key", "val": ": typing.Optional[str] = None"}, {"name": "output_value", "val": ": typing.Union[torch.Tensor, dict, NoneType] = None"}, {"name": "is_conditioning_frame", "val": ": bool = True"}]- obj_idx (int) -- The index of the object.

• frame_idx (int) -- The index of the frame.
• output_key (Optional[str]) -- The key of the output. If None, the output is stored as a dictionary.
• output_value (Optional[Union[torch.Tensor, dict]]) -- The value of the output.
• is_conditioning_frame (bool) -- Whether the output is for a conditioning frame.0

Store output with smart device management. If output_key is None, the output is stored as a dictionary.

Sam2VideoModel[[transformers.Sam2VideoModel]]

class transformers.Sam2VideoModeltransformers.Sam2VideoModelhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L1560[{"name": "config", "val": ": Sam2VideoConfig"}]- config (Sam2VideoConfig) -- Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.0

The bare Sam2 Video Model outputting raw hidden-states without any specific head on top.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forwardtransformers.Sam2VideoModel.forwardhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L1698[{"name": "inference_session", "val": ": Sam2VideoInferenceSession"}, {"name": "frame_idx", "val": ": typing.Optional[int] = None"}, {"name": "frame", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "reverse", "val": ": bool = False"}]- inference_session (~models.sam2_video.modeling_sam2_video.Sam2VideoInferenceSession) -- The video inference session object.

• frame_idx (int, optional) -- The index of the frame on which to run inference. No need to provide when inferring on a new streamed frame.

• frame (torch.Tensor, optional) -- The frame to process. Provide when streaming.

• reverse (bool, optional, defaults to False) -- Whether to propagate in reverse.0transformers.models.sam2_video.modeling_sam2_video.Sam2VideoSegmentationOutput or tuple(torch.FloatTensor)A transformers.models.sam2_video.modeling_sam2_video.Sam2VideoSegmentationOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (Sam2VideoConfig) and inputs.

• pred_masks (torch.FloatTensor of shape (batch_size, num_masks, height, width)) -- The predicted masks stored at the model's resolution.

• frame_idx (int, optional, defaults to None) -- The frame index of the video. Propagate the objects through a streamed video frame.

propagate_in_video_iteratortransformers.Sam2VideoModel.propagate_in_video_iteratorhttps://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/sam2_video/modeling_sam2_video.py#L2581[{"name": "inference_session", "val": ": Sam2VideoInferenceSession"}, {"name": "start_frame_idx", "val": ": typing.Optional[int] = None"}, {"name": "max_frame_num_to_track", "val": ": typing.Optional[int] = None"}, {"name": "reverse", "val": ": bool = False"}]- inference_session (~models.sam2_video.modeling_sam2_video.Sam2VideoInferenceSession) -- The video inference session object.

• start_frame_idx (int, optional) -- The starting frame index for propagation. Need to be provided if forward hasn't been called on new inputs yet. If not provided, the starting frame index will be the earliest frame with input points.

• max_frame_num_to_track (int, optional) -- The maximum number of frames to track.

• reverse (bool, optional, defaults to False) -- Whether to propagate in reverse.0transformers.models.sam2_video.modeling_sam2_video.Sam2VideoSegmentationOutput or tuple(torch.FloatTensor)A transformers.models.sam2_video.modeling_sam2_video.Sam2VideoSegmentationOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (Sam2VideoConfig) and inputs.

• pred_masks (torch.FloatTensor of shape (batch_size, num_masks, height, width)) -- The predicted masks stored at the model's resolution.

• frame_idx (int, optional, defaults to None) -- The frame index of the video.

Propagate the objects through the video frames. Used when initializing an inference session with a whole video. Yields Sam2VideoSegmentationOutput for each frame.