diffs/39600.patch

diff --git a/docs/source/en/main_classes/image_processor.md b/docs/source/en/main_classes/image_processor.md
index 10e78b34a4a6..7dc9de60571f 100644
--- a/docs/source/en/main_classes/image_processor.md
+++ b/docs/source/en/main_classes/image_processor.md
@@ -16,8 +16,7 @@ rendered properly in your Markdown viewer.
 
 # Image Processor
 
-An image processor is in charge of preparing input features for vision models and post processing their outputs. This includes transformations such as resizing, normalization, and conversion to Numpy and PyTorch tensors. It may also include model specific post-processing such as converting logits to segmentation masks.
-
+An image processor is in charge of loading images (optionally), preparing input features for vision models and post processing their outputs. This includes transformations such as resizing, normalization, and conversion to PyTorch and Numpy tensors. It may also include model specific post-processing such as converting logits to segmentation masks.
 Fast image processors are available for a few models and more will be added in the future. They are based on the [torchvision](https://pytorch.org/vision/stable/index.html) library and provide a significant speed-up, especially when processing on GPU.
 They have the same API as the base image processors and can be used as drop-in replacements.
 To use a fast image processor, you need to install the `torchvision` library, and set the `use_fast` argument to `True` when instantiating the image processor:
diff --git a/docs/source/en/main_classes/video_processor.md b/docs/source/en/main_classes/video_processor.md
index 4ff973d2ed29..ee69030ab1a1 100644
--- a/docs/source/en/main_classes/video_processor.md
+++ b/docs/source/en/main_classes/video_processor.md
@@ -14,10 +14,9 @@ rendered properly in your Markdown viewer.
 
 -->
 
-
 # Video Processor
 
-A **Video Processor** is a utility responsible for preparing input features for video models, as well as handling the post-processing of their outputs. It provides transformations such as resizing, normalization, and conversion into PyTorch. 
+A **Video Processor** is a utility responsible for preparing input features for video models, as well as handling the post-processing of their outputs. It provides transformations such as resizing, normalization, and conversion into PyTorch. Along ith transformations the `VideoProcessor` class handles video decoding from local paths or URLs (requires [`torchcodec`](https://pypi.org/project/torchcodec/)) and frame sampling according to model-specific strategies.
 
 The video processor extends the functionality of image processors by allowing Vision Large Language Models (VLMs) to handle videos with a distinct set of arguments compared to images. It serves as the bridge between raw video data and the model, ensuring that input features are optimized for the VLM.
 
@@ -48,6 +47,47 @@ processor = torch.compile(processor)
 processed_video = processor(video, return_tensors="pt")
 ```
 
+#### Sampling behavior
+
+The video processor can also sample video frames using the technique best suited for the given model. Sampling behavior is controlled with the `do_sample_frames` argument and can be configured through model-specific parameters such as `num_frames` or `fps` (the rate at which the video will be sampled). If the input video is given as a local path or URL (`str`), the processor will decode it automatically. To obtain metadata about the decoded video, such as sampled frame indices, original dimensions, duration, and fps, pass `return_metadata=True` to the processor.
+
+<Tip warning={false}>
+
+- Specifying `num_frames` does not guarantee the output will contain exactly that number of frames. Depending on the model, the sampler may enforce minimum or maximum frame limits.
+
+- The default decoder is [`torchcodec`](https://pypi.org/project/torchcodec/), which must be installed.
+
+</Tip>
+
+
+```python
+from transformers import AutoVideoProcessor
+
+processor = AutoVideoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-0.5b-ov-hf", device="cuda")
+processed_video_inputs = processor(videos=["video_path.mp4"], return_metadata=True, do_sample_frames=True, return_tensors="pt")
+video_metadata = processed_video_inputs["video_metadata"]
+
+# See how many frames the original video had and what was the original FPS
+print(video_metadata.total_num_frames, video_metadata.fps)
+```
+
+If you pass an already decoded video array but still want to enable model-specific frame sampling, it is strongly recommended to provide video_metadata. This allows the sampler to know the original video’s duration and FPS. You can pass metadata as a `VideoMetadata` object or as a plain dict.
+
+```python
+from transformers import AutoVideoProcessor
+from transformers.video_utils import VideoMetadata
+
+processor = AutoVideoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-0.5b-ov-hf", device="cuda")
+my_decodec_video = torch.randint(0, 255, size=(100, 3, 1280, 1280)) # short video of 100 frames
+video_metadata = VideoMetadata(
+    total_num_frames=100,
+    fps=24,
+    duration=4.1, # in seconds
+)
+processed_video_inputs = processor(videos=["video_path.mp4"], video_metadata=video_metadata, do_sample_frames=True, num_frames=10, return_tensors="pt")
+print(processed_video_inputs.pixel_values_videos.shape)
+>>> [10, 3, 384, 384]
+```
 
 ## BaseVideoProcessor
 
diff --git a/src/transformers/audio_utils.py b/src/transformers/audio_utils.py
index cb607e3fc94f..0564fdd1b25f 100644
--- a/src/transformers/audio_utils.py
+++ b/src/transformers/audio_utils.py
@@ -21,7 +21,7 @@
 import os
 import warnings
 from io import BytesIO
-from typing import Any, Optional, Union
+from typing import Any, Optional, Sequence, Union
 
 import numpy as np
 import requests
@@ -31,6 +31,7 @@
     is_numpy_array,
     is_soundfile_available,
     is_torch_tensor,
+    is_torchcodec_available,
     requires_backends,
 )
 
@@ -44,6 +45,12 @@
     # TODO: @eustlb, we actually don't need librosa but soxr is installed with librosa
     import soxr
 
+if is_torchcodec_available():
+    from torchcodec.decoders import AudioDecoder
+
+
+AudioInput = Union[np.ndarray, "torch.Tensor", Sequence[np.ndarray], Sequence["torch.Tensor"]]  # noqa: F821
+
 
 def load_audio(audio: Union[str, np.ndarray], sampling_rate=16000, timeout=None) -> np.ndarray:
     """
@@ -61,14 +68,14 @@ def load_audio(audio: Union[str, np.ndarray], sampling_rate=16000, timeout=None)
     Returns:
         `np.ndarray`: A numpy array representing the audio.
     """
-    requires_backends(load_audio, ["librosa"])
-
     if isinstance(audio, str):
-        # Load audio from URL (e.g https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/translate_to_chinese.wav)
-        if audio.startswith("http://") or audio.startswith("https://"):
-            audio = librosa.load(BytesIO(requests.get(audio, timeout=timeout).content), sr=sampling_rate)[0]
-        elif os.path.isfile(audio):
-            audio = librosa.load(audio, sr=sampling_rate)[0]
+        # Try to load with `torchcodec` but do not enforce users to install it. If not found
+        # fallback to `librosa`. If using an audio-only model, most probably `torchcodec` won't be
+        # needed.
+        if is_torchcodec_available():
+            audio = load_audio_torchcodec(audio, sampling_rate=sampling_rate)
+        else:
+            audio = load_audio_librosa(audio, sampling_rate=sampling_rate, timeout=timeout)
     elif isinstance(audio, np.ndarray):
         audio = audio
     else:
@@ -78,6 +85,54 @@ def load_audio(audio: Union[str, np.ndarray], sampling_rate=16000, timeout=None)
     return audio
 
 
+def load_audio_torchcodec(audio: Union[str, np.ndarray], sampling_rate=16000) -> np.ndarray:
+    """
+    Loads `audio` to an np.ndarray object using `torchcodec`.
+
+    Args:
+        audio (`str` or `np.ndarray`):
+            The audio to be loaded to the numpy array format.
+        sampling_rate (`int`, *optional*, defaults to 16000):
+            The sampling rate to be used when loading the audio. It should be same as the
+            sampling rate the model you will be using further was trained with.
+
+    Returns:
+        `np.ndarray`: A numpy array representing the audio.
+    """
+    requires_backends(load_audio, ["torchcodec"])
+
+    # Set `num_channels` to `1` which is what most models expects and the default in librosa
+    decoder = AudioDecoder(audio, sample_rate=sampling_rate, num_channels=1)
+    audio = decoder.get_all_samples().data[0].numpy()  # NOTE: feature extractors don't accept torch tensors
+    return audio
+
+
+def load_audio_librosa(audio: Union[str, np.ndarray], sampling_rate=16000, timeout=None) -> np.ndarray:
+    """
+    Loads `audio` to an np.ndarray object using `librosa`.
+
+    Args:
+        audio (`str` or `np.ndarray`):
+            The audio to be loaded to the numpy array format.
+        sampling_rate (`int`, *optional*, defaults to 16000):
+            The sampling rate to be used when loading the audio. It should be same as the
+            sampling rate the model you will be using further was trained with.
+        timeout (`float`, *optional*):
+            The timeout value in seconds for the URL request.
+
+    Returns:
+        `np.ndarray`: A numpy array representing the audio.
+    """
+    requires_backends(load_audio, ["librosa"])
+
+    # Load audio from URL (e.g https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/translate_to_chinese.wav)
+    if audio.startswith("http://") or audio.startswith("https://"):
+        audio = librosa.load(BytesIO(requests.get(audio, timeout=timeout).content), sr=sampling_rate)[0]
+    elif os.path.isfile(audio):
+        audio = librosa.load(audio, sr=sampling_rate)[0]
+    return audio
+
+
 def load_audio_as(
     audio: str,
     return_format: str,
@@ -157,11 +212,6 @@ def load_audio_as(
         raise ValueError(f"Error loading audio: {e}")
 
 
-AudioInput = Union[
-    np.ndarray, "torch.Tensor", list[np.ndarray], tuple[np.ndarray], list["torch.Tensor"], tuple["torch.Tensor"]  # noqa: F821
-]
-
-
 def is_valid_audio(audio):
     return is_numpy_array(audio) or is_torch_tensor(audio)
 
diff --git a/src/transformers/image_processing_base.py b/src/transformers/image_processing_base.py
index 4d708efb7c2a..899f4ea746b6 100644
--- a/src/transformers/image_processing_base.py
+++ b/src/transformers/image_processing_base.py
@@ -17,14 +17,13 @@
 import json
 import os
 import warnings
-from io import BytesIO
 from typing import Any, Optional, TypeVar, Union
 
 import numpy as np
-import requests
 
 from .dynamic_module_utils import custom_object_save
 from .feature_extraction_utils import BatchFeature as BaseBatchFeature
+from .image_utils import is_valid_image, load_image
 from .utils import (
     IMAGE_PROCESSOR_NAME,
     PushToHubMixin,
@@ -33,15 +32,10 @@
     download_url,
     is_offline_mode,
     is_remote_url,
-    is_vision_available,
     logging,
 )
 
 
-if is_vision_available():
-    from PIL import Image
-
-
 ImageProcessorType = TypeVar("ImageProcessorType", bound="ImageProcessingMixin")
 
 
@@ -514,25 +508,19 @@ def register_for_auto_class(cls, auto_class="AutoImageProcessor"):
 
         cls._auto_class = auto_class
 
-    def fetch_images(self, image_url_or_urls: Union[str, list[str]]):
+    def fetch_images(self, image_url_or_urls: Union[str, list[str], list[list[str]]]):
         """
         Convert a single or a list of urls into the corresponding `PIL.Image` objects.
 
         If a single url is passed, the return value will be a single object. If a list is passed a list of objects is
         returned.
         """
-        headers = {
-            "User-Agent": (
-                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0"
-                " Safari/537.36"
-            )
-        }
         if isinstance(image_url_or_urls, list):
             return [self.fetch_images(x) for x in image_url_or_urls]
         elif isinstance(image_url_or_urls, str):
-            response = requests.get(image_url_or_urls, stream=True, headers=headers)
-            response.raise_for_status()
-            return Image.open(BytesIO(response.content))
+            return load_image(image_url_or_urls)
+        elif is_valid_image(image_url_or_urls):
+            return image_url_or_urls
         else:
             raise TypeError(f"only a single or a list of entries is supported but got type={type(image_url_or_urls)}")
 
diff --git a/src/transformers/image_processing_utils_fast.py b/src/transformers/image_processing_utils_fast.py
index 9998ad60fd5b..670fb5ece4cd 100644
--- a/src/transformers/image_processing_utils_fast.py
+++ b/src/transformers/image_processing_utils_fast.py
@@ -85,7 +85,7 @@ def validate_fast_preprocess_arguments(
     crop_size: Optional[SizeDict] = None,
     do_resize: Optional[bool] = None,
     size: Optional[SizeDict] = None,
-    resample: Optional["PILImageResampling"] = None,
+    interpolation: Optional["F.InterpolationMode"] = None,
     return_tensors: Optional[Union[str, TensorType]] = None,
     data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
 ):
@@ -105,7 +105,7 @@ def validate_fast_preprocess_arguments(
         crop_size=crop_size,
         do_resize=do_resize,
         size=size,
-        resample=resample,
+        interpolation=interpolation,
     )
     # Extra checks for ImageProcessorFast
     if return_tensors is not None and return_tensors != "pt":
@@ -469,6 +469,8 @@ def _prepare_images_structure(
         Returns:
             `ImageInput`: The images with a valid nesting.
         """
+        # Checks for `str` in case of URL/local path and optionally loads images
+        images = self.fetch_images(images)
         return make_flat_list_of_images(images, expected_ndims=expected_ndims)
 
     def _process_image(
@@ -582,11 +584,19 @@ def _further_process_kwargs(
 
         kwargs["size"] = size
         kwargs["crop_size"] = crop_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["data_format"] = data_format
 
+        # torch resize uses interpolation instead of resample
+        # Check if resample is an int before checking if it's an instance of PILImageResampling
+        # because if pillow < 9.1.0, resample is an int and PILImageResampling is a module.
+        # Checking PILImageResampling will fail with error `TypeError: isinstance() arg 2 must be a type or tuple of types`.
+        resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[resample] if isinstance(resample, (PILImageResampling, int)) else resample
+        )
+
         return kwargs
 
     def _validate_preprocess_kwargs(
@@ -600,7 +610,7 @@ def _validate_preprocess_kwargs(
         size: Optional[SizeDict] = None,
         do_center_crop: Optional[bool] = None,
         crop_size: Optional[SizeDict] = None,
-        resample: Optional[Union["PILImageResampling", "F.InterpolationMode"]] = None,
+        interpolation: Optional["F.InterpolationMode"] = None,
         return_tensors: Optional[Union[str, TensorType]] = None,
         data_format: Optional[ChannelDimension] = None,
         **kwargs,
@@ -618,7 +628,7 @@ def _validate_preprocess_kwargs(
             size=size,
             do_center_crop=do_center_crop,
             crop_size=crop_size,
-            resample=resample,
+            interpolation=interpolation,
             return_tensors=return_tensors,
             data_format=data_format,
         )
@@ -646,18 +656,7 @@ def preprocess(self, images: ImageInput, *args, **kwargs: Unpack[DefaultFastImag
         # Validate kwargs
         self._validate_preprocess_kwargs(**kwargs)
 
-        # torch resize uses interpolation instead of resample
-        resample = kwargs.pop("resample")
-
-        # Check if resample is an int before checking if it's an instance of PILImageResampling
-        # because if pillow < 9.1.0, resample is an int and PILImageResampling is a module.
-        # Checking PILImageResampling will fail with error `TypeError: isinstance() arg 2 must be a type or tuple of types`.
-        kwargs["interpolation"] = (
-            pil_torch_interpolation_mapping[resample] if isinstance(resample, (int, PILImageResampling)) else resample
-        )
-
         # Pop kwargs that are not needed in _preprocess
-        kwargs.pop("default_to_square")
         kwargs.pop("data_format")
 
         return self._preprocess_image_like_inputs(
diff --git a/src/transformers/image_utils.py b/src/transformers/image_utils.py
index d58e589c4601..d55bf710c06c 100644
--- a/src/transformers/image_utils.py
+++ b/src/transformers/image_utils.py
@@ -535,6 +535,7 @@ def validate_preprocess_arguments(
     do_resize: Optional[bool] = None,
     size: Optional[dict[str, int]] = None,
     resample: Optional["PILImageResampling"] = None,
+    interpolation: Optional["InterpolationMode"] = None,
 ):
     """
     Checks validity of typically used arguments in an `ImageProcessor` `preprocess` method.
@@ -559,8 +560,13 @@ def validate_preprocess_arguments(
     if do_center_crop and crop_size is None:
         raise ValueError("`crop_size` must be specified if `do_center_crop` is `True`.")
 
-    if do_resize and (size is None or resample is None):
-        raise ValueError("`size` and `resample` must be specified if `do_resize` is `True`.")
+    if interpolation is not None and resample is not None:
+        raise ValueError(
+            "Only one of `interpolation` and `resample` should be specified, depending on image processor type."
+        )
+
+    if do_resize and not (size is not None and (resample is not None or interpolation is not None)):
+        raise ValueError("`size` and `resample/interpolation` must be specified if `do_resize` is `True`.")
 
 
 # In the future we can add a TF implementation here when we have TF models.
diff --git a/src/transformers/models/aria/image_processing_aria.py b/src/transformers/models/aria/image_processing_aria.py
index 8db238d66f15..6146f9b32bd2 100644
--- a/src/transformers/models/aria/image_processing_aria.py
+++ b/src/transformers/models/aria/image_processing_aria.py
@@ -228,6 +228,7 @@ def preprocess(
         if max_image_size not in [490, 980]:
             raise ValueError("max_image_size must be either 490 or 980")
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/aria/modular_aria.py b/src/transformers/models/aria/modular_aria.py
index d4c36734846b..d91807fcbaf1 100644
--- a/src/transformers/models/aria/modular_aria.py
+++ b/src/transformers/models/aria/modular_aria.py
@@ -614,6 +614,7 @@ def preprocess(
         if max_image_size not in [490, 980]:
             raise ValueError("max_image_size must be either 490 or 980")
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/aya_vision/processing_aya_vision.py b/src/transformers/models/aya_vision/processing_aya_vision.py
index 1878d2c0b5d6..7045c967046d 100644
--- a/src/transformers/models/aya_vision/processing_aya_vision.py
+++ b/src/transformers/models/aya_vision/processing_aya_vision.py
@@ -189,6 +189,7 @@ def __call__(
         # Process images
         image_inputs = {}
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             images = make_flat_list_of_images(images)
             image_inputs = self.image_processor(images=images, **output_kwargs["images_kwargs"])
             num_patches = image_inputs.pop("num_patches")
diff --git a/src/transformers/models/blip/image_processing_blip.py b/src/transformers/models/blip/image_processing_blip.py
index 4c3ec00d26fd..b932cb1453f2 100644
--- a/src/transformers/models/blip/image_processing_blip.py
+++ b/src/transformers/models/blip/image_processing_blip.py
@@ -231,6 +231,7 @@ def preprocess(
 
         size = size if size is not None else self.size
         size = get_size_dict(size, default_to_square=False)
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/bridgetower/image_processing_bridgetower.py b/src/transformers/models/bridgetower/image_processing_bridgetower.py
index c3151f9c1386..7e047284aa2f 100644
--- a/src/transformers/models/bridgetower/image_processing_bridgetower.py
+++ b/src/transformers/models/bridgetower/image_processing_bridgetower.py
@@ -465,6 +465,7 @@ def preprocess(
 
         size = size if size is not None else self.size
         size = get_size_dict(size, default_to_square=False)
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/chameleon/image_processing_chameleon.py b/src/transformers/models/chameleon/image_processing_chameleon.py
index 96df32630f3c..651fd63b7e44 100644
--- a/src/transformers/models/chameleon/image_processing_chameleon.py
+++ b/src/transformers/models/chameleon/image_processing_chameleon.py
@@ -248,6 +248,7 @@ def preprocess(
         image_std = image_std if image_std is not None else self.image_std
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/clip/image_processing_clip.py b/src/transformers/models/clip/image_processing_clip.py
index df96f0f64b89..25709a3d5462 100644
--- a/src/transformers/models/clip/image_processing_clip.py
+++ b/src/transformers/models/clip/image_processing_clip.py
@@ -285,6 +285,7 @@ def preprocess(
 
         validate_kwargs(captured_kwargs=kwargs.keys(), valid_processor_keys=self._valid_processor_keys)
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/colpali/modular_colpali.py b/src/transformers/models/colpali/modular_colpali.py
index d36d59d44f88..3b86a0ee1116 100644
--- a/src/transformers/models/colpali/modular_colpali.py
+++ b/src/transformers/models/colpali/modular_colpali.py
@@ -19,7 +19,7 @@
 from transformers.models.paligemma.processing_paligemma import IMAGE_TOKEN, PaliGemmaProcessor, build_string_from_input
 
 from ...feature_extraction_utils import BatchFeature
-from ...image_utils import ImageInput, is_valid_image, make_flat_list_of_images
+from ...image_utils import ImageInput, make_flat_list_of_images
 from ...processing_utils import ProcessingKwargs, Unpack
 from ...tokenization_utils_base import PreTokenizedInput, TextInput
 from ...utils import is_torch_available, logging
@@ -147,13 +147,8 @@ def __call__(
             raise ValueError("Only one of text or images can be processed at a time")
 
         if images is not None:
-            if is_valid_image(images):
-                images = [images]
-            elif isinstance(images, list) and is_valid_image(images[0]):
-                pass
-            elif not (isinstance(images, list) and isinstance(images[0], list) and is_valid_image(images[0][0])):
-                raise ValueError("images must be an image, list of images or list of list of images")
-
+            images = self.image_processor.fetch_images(images)
+            images = make_flat_list_of_images(images)
             texts_doc = [self.visual_prompt_prefix] * len(images)
             images = [image.convert("RGB") for image in images]
 
@@ -167,7 +162,6 @@ def __call__(
                 )
                 for prompt, image_list in zip(texts_doc, images)
             ]
-            images = make_flat_list_of_images(images)
             pixel_values = self.image_processor(images, **output_kwargs["images_kwargs"])["pixel_values"]
 
             # max_length has to account for the image tokens
diff --git a/src/transformers/models/colpali/processing_colpali.py b/src/transformers/models/colpali/processing_colpali.py
index ec6055489f70..429856ec30cb 100644
--- a/src/transformers/models/colpali/processing_colpali.py
+++ b/src/transformers/models/colpali/processing_colpali.py
@@ -23,7 +23,7 @@
 from typing import Optional, Union
 
 from ...feature_extraction_utils import BatchFeature
-from ...image_utils import ImageInput, is_valid_image, make_flat_list_of_images
+from ...image_utils import ImageInput, make_flat_list_of_images
 from ...processing_utils import MultiModalData, ProcessingKwargs, ProcessorMixin, Unpack
 from ...tokenization_utils_base import AddedToken, PreTokenizedInput, TextInput
 from ...utils import is_torch_available
@@ -191,13 +191,8 @@ def __call__(
             raise ValueError("Only one of text or images can be processed at a time")
 
         if images is not None:
-            if is_valid_image(images):
-                images = [images]
-            elif isinstance(images, list) and is_valid_image(images[0]):
-                pass
-            elif not (isinstance(images, list) and isinstance(images[0], list) and is_valid_image(images[0][0])):
-                raise ValueError("images must be an image, list of images or list of list of images")
-
+            images = self.image_processor.fetch_images(images)
+            images = make_flat_list_of_images(images)
             texts_doc = [self.visual_prompt_prefix] * len(images)
             images = [image.convert("RGB") for image in images]
 
@@ -211,7 +206,6 @@ def __call__(
                 )
                 for prompt, image_list in zip(texts_doc, images)
             ]
-            images = make_flat_list_of_images(images)
             pixel_values = self.image_processor(images, **output_kwargs["images_kwargs"])["pixel_values"]
 
             # max_length has to account for the image tokens
diff --git a/src/transformers/models/deepseek_vl/image_processing_deepseek_vl.py b/src/transformers/models/deepseek_vl/image_processing_deepseek_vl.py
index 24b162fa9f92..8a68d434837f 100644
--- a/src/transformers/models/deepseek_vl/image_processing_deepseek_vl.py
+++ b/src/transformers/models/deepseek_vl/image_processing_deepseek_vl.py
@@ -277,6 +277,7 @@ def preprocess(
 
         size = size if size is not None else self.size
         size = get_size_dict(size, default_to_square=False)
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/deepseek_vl/modular_deepseek_vl.py b/src/transformers/models/deepseek_vl/modular_deepseek_vl.py
index b9f3fc37ba7a..6d9b7709eae6 100644
--- a/src/transformers/models/deepseek_vl/modular_deepseek_vl.py
+++ b/src/transformers/models/deepseek_vl/modular_deepseek_vl.py
@@ -16,10 +16,7 @@
 
 from ...configuration_utils import PretrainedConfig
 from ...image_processing_utils import BatchFeature
-from ...image_utils import (
-    ImageInput,
-    make_flat_list_of_images,
-)
+from ...image_utils import ImageInput
 from ...processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
 from ...tokenization_utils_base import (
     PreTokenizedInput,
@@ -302,7 +299,6 @@ def __call__(
 
         # process images if pixel_values are provided
         if images is not None:
-            images = make_flat_list_of_images(images)
             data["pixel_values"] = self.image_processor(images, **output_kwargs["images_kwargs"])["pixel_values"]
 
         return BatchFeature(data=data)
diff --git a/src/transformers/models/deepseek_vl/processing_deepseek_vl.py b/src/transformers/models/deepseek_vl/processing_deepseek_vl.py
index 244e642d7c36..ada14ab87b90 100644
--- a/src/transformers/models/deepseek_vl/processing_deepseek_vl.py
+++ b/src/transformers/models/deepseek_vl/processing_deepseek_vl.py
@@ -21,7 +21,7 @@
 from typing import Union
 
 from ...image_processing_utils import BatchFeature
-from ...image_utils import ImageInput, make_flat_list_of_images
+from ...image_utils import ImageInput
 from ...processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
 from ...tokenization_utils_base import PreTokenizedInput, TextInput
 
@@ -128,7 +128,6 @@ def __call__(
 
         # process images if pixel_values are provided
         if images is not None:
-            images = make_flat_list_of_images(images)
             data["pixel_values"] = self.image_processor(images, **output_kwargs["images_kwargs"])["pixel_values"]
 
         return BatchFeature(data=data)
diff --git a/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid.py b/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid.py
index 9cf30a476bce..4589a0deeca5 100644
--- a/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid.py
+++ b/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid.py
@@ -34,7 +34,7 @@
     get_image_size,
     infer_channel_dimension_format,
     is_scaled_image,
-    make_list_of_images,
+    make_flat_list_of_images,
     to_numpy_array,
     valid_images,
     validate_preprocess_arguments,
@@ -327,7 +327,8 @@ def preprocess(
         high_res_size = high_res_size if high_res_size is not None else self.high_res_size
         high_res_size_dict = get_size_dict(high_res_size)
 
-        images = make_list_of_images(images)
+        images = self.fetch_images(images)
+        images = make_flat_list_of_images(images)
 
         if not valid_images(images):
             raise ValueError(
diff --git a/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid_fast.py b/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid_fast.py
index d720c48e4124..14a99c56a049 100644
--- a/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid_fast.py
+++ b/src/transformers/models/deepseek_vl_hybrid/image_processing_deepseek_vl_hybrid_fast.py
@@ -312,9 +312,15 @@ def _further_process_kwargs(
             else high_res_resample
         )
 
+        low_res_resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[low_res_resample]
+            if isinstance(low_res_resample, (int, PILImageResampling))
+            else low_res_resample
+        )
+
         kwargs["size"] = size
         kwargs["high_res_size"] = high_res_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["high_res_image_mean"] = high_res_image_mean
diff --git a/src/transformers/models/deepseek_vl_hybrid/modular_deepseek_vl_hybrid.py b/src/transformers/models/deepseek_vl_hybrid/modular_deepseek_vl_hybrid.py
index 19865daed94f..b6ada0c8f1e5 100644
--- a/src/transformers/models/deepseek_vl_hybrid/modular_deepseek_vl_hybrid.py
+++ b/src/transformers/models/deepseek_vl_hybrid/modular_deepseek_vl_hybrid.py
@@ -36,7 +36,6 @@
     infer_channel_dimension_format,
     is_scaled_image,
     make_flat_list_of_images,
-    make_list_of_images,
     to_numpy_array,
     valid_images,
     validate_preprocess_arguments,
@@ -633,7 +632,8 @@ def preprocess(
         high_res_size = high_res_size if high_res_size is not None else self.high_res_size
         high_res_size_dict = get_size_dict(high_res_size)
 
-        images = make_list_of_images(images)
+        images = self.fetch_images(images)
+        images = make_flat_list_of_images(images)
 
         if not valid_images(images):
             raise ValueError(
@@ -804,9 +804,15 @@ def _further_process_kwargs(
             else high_res_resample
         )
 
+        low_res_resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[low_res_resample]
+            if isinstance(low_res_resample, (int, PILImageResampling))
+            else low_res_resample
+        )
+
         kwargs["size"] = size
         kwargs["high_res_size"] = high_res_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["high_res_image_mean"] = high_res_image_mean
@@ -969,7 +975,6 @@ def __call__(
 
         # process images if pixel_values are provided
         if images is not None:
-            images = make_flat_list_of_images(images)
             inputs = self.image_processor(images, **output_kwargs["images_kwargs"])
             data["pixel_values"] = inputs["pixel_values"]
             data["high_res_pixel_values"] = inputs["high_res_pixel_values"]
diff --git a/src/transformers/models/deepseek_vl_hybrid/processing_deepseek_vl_hybrid.py b/src/transformers/models/deepseek_vl_hybrid/processing_deepseek_vl_hybrid.py
index 4fb765c79764..914c59ad205c 100644
--- a/src/transformers/models/deepseek_vl_hybrid/processing_deepseek_vl_hybrid.py
+++ b/src/transformers/models/deepseek_vl_hybrid/processing_deepseek_vl_hybrid.py
@@ -21,7 +21,7 @@
 from typing import Union
 
 from ...image_processing_utils_fast import BatchFeature
-from ...image_utils import ImageInput, make_flat_list_of_images
+from ...image_utils import ImageInput
 from ...processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
 from ...tokenization_utils_base import PreTokenizedInput, TextInput
 
@@ -128,7 +128,6 @@ def __call__(
 
         # process images if pixel_values are provided
         if images is not None:
-            images = make_flat_list_of_images(images)
             inputs = self.image_processor(images, **output_kwargs["images_kwargs"])
             data["pixel_values"] = inputs["pixel_values"]
             data["high_res_pixel_values"] = inputs["high_res_pixel_values"]
diff --git a/src/transformers/models/emu3/image_processing_emu3.py b/src/transformers/models/emu3/image_processing_emu3.py
index b96983b0e858..5a480351307a 100644
--- a/src/transformers/models/emu3/image_processing_emu3.py
+++ b/src/transformers/models/emu3/image_processing_emu3.py
@@ -381,6 +381,7 @@ def preprocess(
         do_pad = do_pad if do_pad is not None else self.do_pad
 
         if images is not None:
+            images = self.fetch_images(images)
             images = make_batched_images(images)
 
         if images is not None and not valid_images(images):
diff --git a/src/transformers/models/eomt/image_processing_eomt.py b/src/transformers/models/eomt/image_processing_eomt.py
index 37b9b11103ad..05131c543d83 100644
--- a/src/transformers/models/eomt/image_processing_eomt.py
+++ b/src/transformers/models/eomt/image_processing_eomt.py
@@ -578,6 +578,7 @@ def preprocess(
         image_std = image_std if image_std is not None else self.image_std
         ignore_index = ignore_index if ignore_index is not None else self.ignore_index
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/flava/image_processing_flava_fast.py b/src/transformers/models/flava/image_processing_flava_fast.py
index 854a8780e28f..5dcc5326d968 100644
--- a/src/transformers/models/flava/image_processing_flava_fast.py
+++ b/src/transformers/models/flava/image_processing_flava_fast.py
@@ -337,7 +337,6 @@ def _further_process_kwargs(
 
         kwargs["size"] = size
         kwargs["crop_size"] = crop_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["codebook_size"] = codebook_size
@@ -351,6 +350,15 @@ def _further_process_kwargs(
             else codebook_resample
         )
 
+        # torch resize uses interpolation instead of resample
+        # Check if resample is an int before checking if it's an instance of PILImageResampling
+        # because if pillow < 9.1.0, resample is an int and PILImageResampling is a module.
+        # Checking PILImageResampling will fail with error `TypeError: isinstance() arg 2 must be a type or tuple of types`.
+        resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[resample] if isinstance(resample, (PILImageResampling, int)) else resample
+        )
+
         return kwargs
 
     def _preprocess_image(
diff --git a/src/transformers/models/gemma3/image_processing_gemma3.py b/src/transformers/models/gemma3/image_processing_gemma3.py
index c74410c43818..f7bd414dbb91 100644
--- a/src/transformers/models/gemma3/image_processing_gemma3.py
+++ b/src/transformers/models/gemma3/image_processing_gemma3.py
@@ -334,6 +334,7 @@ def preprocess(
             else self.pan_and_scan_min_ratio_to_activate
         )
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/gemma3/processing_gemma3.py b/src/transformers/models/gemma3/processing_gemma3.py
index 683a8ffdfe8b..4c27053e1a6f 100644
--- a/src/transformers/models/gemma3/processing_gemma3.py
+++ b/src/transformers/models/gemma3/processing_gemma3.py
@@ -101,8 +101,9 @@ def __call__(
 
         image_inputs = {}
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             batched_images = make_nested_list_of_images(images)
-            image_inputs = self.image_processor(batched_images, **output_kwargs["images_kwargs"])
+            image_inputs = self.image_processor(images, **output_kwargs["images_kwargs"])
 
             # Create empty text to be replaced with placeholders
             if not text:
diff --git a/src/transformers/models/gemma3n/processing_gemma3n.py b/src/transformers/models/gemma3n/processing_gemma3n.py
index 62c60395d051..19274fece4c1 100644
--- a/src/transformers/models/gemma3n/processing_gemma3n.py
+++ b/src/transformers/models/gemma3n/processing_gemma3n.py
@@ -134,6 +134,7 @@ def __call__(
             audio_inputs = {}
 
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             batched_images = make_nested_list_of_images(images)
             image_inputs = self.image_processor(batched_images, **output_kwargs["images_kwargs"])
 
diff --git a/src/transformers/models/glm4v/image_processing_glm4v.py b/src/transformers/models/glm4v/image_processing_glm4v.py
index 678eb10406a9..db2691c38dc7 100644
--- a/src/transformers/models/glm4v/image_processing_glm4v.py
+++ b/src/transformers/models/glm4v/image_processing_glm4v.py
@@ -370,7 +370,6 @@ def preprocess(
                 - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
 
         """
-
         if size is not None and ("shortest_edge" not in size or "longest_edge" not in size):
             raise ValueError("size must contain 'shortest_edge' and 'longest_edge' keys.")
         elif size is None:
@@ -390,6 +389,7 @@ def preprocess(
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
         if images is not None:
+            images = self.fetch_images(images)
             images = make_flat_list_of_images(images)
 
         if images is not None and not valid_images(images):
@@ -461,9 +461,9 @@ def get_number_of_image_patches(self, height: int, width: int, images_kwargs=Non
             height=height,
             width=width,
             factor=factor,
-            temporal_factor=self.temporal_patch_size,
             min_pixels=size["shortest_edge"],
             max_pixels=size["longest_edge"],
+            temporal_factor=self.temporal_patch_size,
         )
         grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
         return grid_h * grid_w
diff --git a/src/transformers/models/glm4v/modular_glm4v.py b/src/transformers/models/glm4v/modular_glm4v.py
index f76e04fc89c2..f39ba8e3824a 100644
--- a/src/transformers/models/glm4v/modular_glm4v.py
+++ b/src/transformers/models/glm4v/modular_glm4v.py
@@ -1510,6 +1510,7 @@ class Glm4vProcessorKwargs(Qwen2_VLProcessorKwargs):
             "padding": False,
             "return_mm_token_type_ids": False,
         },
+        "videos_kwargs": {"return_metadata": True},
     }
 
 
@@ -1591,11 +1592,14 @@ def __call__(
 
         if videos is not None:
             videos_inputs = self.video_processor(videos=videos, **output_kwargs["videos_kwargs"])
-            timestamps = videos_inputs.pop("timestamps")
+            # If user has not requested video metadata, pop it
+            if "return_metadata" not in kwargs:
+                video_metadata = videos_inputs.pop("video_metadata")
+            else:
+                video_metadata = videos_inputs["video_metadata"]
             video_grid_thw = videos_inputs["video_grid_thw"]
         else:
             videos_inputs = {}
-            timestamps = []
             video_grid_thw = None
 
         if not isinstance(text, list):
@@ -1620,14 +1624,19 @@ def __call__(
                     num_frames = video_grid_thw[video_index][0]
                     video_structure = ""
 
-                    if hasattr(timestamps, "tolist"):
-                        timestamps_list = timestamps.tolist()[0]
-                    else:
-                        timestamps_list = timestamps[0] if isinstance(timestamps[0], list) else timestamps
+                    metadata = video_metadata[i]
+                    if metadata.fps is None:
+                        logger.warning_once(
+                            "SmolVLM requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
+                            "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
+                            "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+                        )
+                    metadata.fps = 24 if metadata.fps is None else metadata.fps
+                    timestamps = metadata.timestamps[::2]  # mrope
 
                     unique_timestamps = []
-                    for idx in range(0, len(timestamps_list)):
-                        unique_timestamps.append(timestamps_list[idx])
+                    for idx in range(0, len(timestamps)):
+                        unique_timestamps.append(timestamps[idx])
 
                     selected_timestamps = unique_timestamps[:num_frames]
                     while len(selected_timestamps) < num_frames:
@@ -1635,7 +1644,7 @@ def __call__(
 
                     for frame_idx in range(num_frames):
                         timestamp_sec = selected_timestamps[frame_idx]
-                        frame_structure = f"<|begin_of_image|>{self.image_token}<|end_of_image|>{timestamp_sec}"
+                        frame_structure = f"<|begin_of_image|>{self.image_token}<|end_of_image|>{int(timestamp_sec)}"
                         video_structure += frame_structure
 
                     text[i] = text[i].replace(self.video_token, video_structure, 1)
diff --git a/src/transformers/models/glm4v/processing_glm4v.py b/src/transformers/models/glm4v/processing_glm4v.py
index e4c5f926cfff..3cec2c897258 100644
--- a/src/transformers/models/glm4v/processing_glm4v.py
+++ b/src/transformers/models/glm4v/processing_glm4v.py
@@ -26,9 +26,13 @@
 from ...image_utils import ImageInput
 from ...processing_utils import ImagesKwargs, MultiModalData, ProcessingKwargs, ProcessorMixin, Unpack, VideosKwargs
 from ...tokenization_utils_base import PreTokenizedInput, TextInput
+from ...utils import logging
 from ...video_utils import VideoInput
 
 
+logger = logging.get_logger(__name__)
+
+
 class Glm4vVideosProcessorKwargs(VideosKwargs, total=False):
     fps: Union[list[float], float]
 
@@ -46,6 +50,7 @@ class Glm4vProcessorKwargs(ProcessingKwargs, total=False):
             "padding": False,
             "return_mm_token_type_ids": False,
         },
+        "videos_kwargs": {"return_metadata": True},
     }
     videos_kwargs: Glm4vVideosProcessorKwargs
 
@@ -142,11 +147,14 @@ def __call__(
 
         if videos is not None:
             videos_inputs = self.video_processor(videos=videos, **output_kwargs["videos_kwargs"])
-            timestamps = videos_inputs.pop("timestamps")
+            # If user has not requested video metadata, pop it
+            if "return_metadata" not in kwargs:
+                video_metadata = videos_inputs.pop("video_metadata")
+            else:
+                video_metadata = videos_inputs["video_metadata"]
             video_grid_thw = videos_inputs["video_grid_thw"]
         else:
             videos_inputs = {}
-            timestamps = []
             video_grid_thw = None
 
         if not isinstance(text, list):
@@ -171,14 +179,19 @@ def __call__(
                     num_frames = video_grid_thw[video_index][0]
                     video_structure = ""
 
-                    if hasattr(timestamps, "tolist"):
-                        timestamps_list = timestamps.tolist()[0]
-                    else:
-                        timestamps_list = timestamps[0] if isinstance(timestamps[0], list) else timestamps
+                    metadata = video_metadata[i]
+                    if metadata.fps is None:
+                        logger.warning_once(
+                            "SmolVLM requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
+                            "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
+                            "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+                        )
+                    metadata.fps = 24 if metadata.fps is None else metadata.fps
+                    timestamps = metadata.timestamps[::2]  # mrope
 
                     unique_timestamps = []
-                    for idx in range(0, len(timestamps_list)):
-                        unique_timestamps.append(timestamps_list[idx])
+                    for idx in range(0, len(timestamps)):
+                        unique_timestamps.append(timestamps[idx])
 
                     selected_timestamps = unique_timestamps[:num_frames]
                     while len(selected_timestamps) < num_frames:
@@ -186,7 +199,7 @@ def __call__(
 
                     for frame_idx in range(num_frames):
                         timestamp_sec = selected_timestamps[frame_idx]
-                        frame_structure = f"<|begin_of_image|>{self.image_token}<|end_of_image|>{timestamp_sec}"
+                        frame_structure = f"<|begin_of_image|>{self.image_token}<|end_of_image|>{int(timestamp_sec)}"
                         video_structure += frame_structure
 
                     text[i] = text[i].replace(self.video_token, video_structure, 1)
diff --git a/src/transformers/models/glm4v/video_processing_glm4v.py b/src/transformers/models/glm4v/video_processing_glm4v.py
index ddee09a8e876..a327ac200507 100644
--- a/src/transformers/models/glm4v/video_processing_glm4v.py
+++ b/src/transformers/models/glm4v/video_processing_glm4v.py
@@ -120,27 +120,42 @@ def _further_process_kwargs(
 
     def sample_frames(
         self,
-        video: torch.Tensor,
-        metadata: Union[VideoMetadata, dict],
+        metadata: VideoMetadata,
+        fps: Optional[Union[int, float]] = None,
+        **kwargs,
     ):
-        total_frames = video.shape[0]
-        video_fps = getattr(metadata, "fps", 2.0)
-        meta_frames = getattr(metadata, "total_num_frames", total_frames)
-        max_frame_idx = meta_frames - 1
-        duration = getattr(metadata, "duration", None)
-        if duration is None:
-            duration = round(max_frame_idx / video_fps) + 1
+        """
+        Args:
+            metadata (`VideoMetadata`):
+                Metadata of the video containing information about total duration, fps and total number of frames.
+            fps (`int` or `float`, *optional*):
+                Target frames to sample per second. Defaults to `self.fps`.
+        Returns:
+            np.ndarray:
+                Indices to sample video frames.
+        """
+        if metadata is None or getattr(metadata, "fps", None) is None:
+            raise ValueError(
+                "Asked to sample frames per second but no video metadata was provided which is required when sampling in GLM4V. "
+                "Please pass in `VideoMetadata` object or set `do_sample_frames=False`"
+            )
+
+        total_frames = metadata.total_num_frames
+        requested_fps = fps if fps is not None else self.fps
+
+        max_frame_idx = total_frames - 1
+        duration = metadata.duration or round(max_frame_idx / metadata.fps) + 1
 
         if duration <= self.max_duration:
-            n = int(math.floor(duration * self.fps))
-            frame_indices = [min(max_frame_idx, int(math.ceil(i * video_fps / self.fps))) for i in range(n)]
+            n = int(math.floor(duration * requested_fps))
+            frame_indices = [min(max_frame_idx, int(math.ceil(i * metadata.fps / requested_fps))) for i in range(n)]
         else:
-            num_samples = int(self.max_duration * self.fps)
-            if num_samples >= meta_frames:
-                frame_indices = list(range(meta_frames))
+            num_samples = int(self.max_duration * requested_fps)
+            if num_samples >= total_frames:
+                frame_indices = list(range(total_frames))
             else:
                 target_seconds = np.linspace(0, duration, num_samples, endpoint=True)
-                frame_indices = [min(max_frame_idx, int(math.ceil(t * video_fps))) for t in target_seconds]
+                frame_indices = [min(max_frame_idx, int(math.ceil(t * metadata.fps))) for t in target_seconds]
 
         seen, uniq = set(), []
         for idx in frame_indices:
@@ -151,23 +166,18 @@ def sample_frames(
         if len(uniq) & 1:
             uniq.append(uniq[-1])
 
-        frame_indices = uniq
-        sampled_video = video[frame_indices]
-        full_second_idxs = [int(idx / video_fps) for idx in frame_indices]
-        second_idxs = full_second_idxs[::2]  # mrope
-        return sampled_video, second_idxs
+        return np.array(uniq)
 
     def _preprocess(
         self,
         videos: list[torch.Tensor],
-        video_metadata: Optional[Union[list[VideoMetadata], list[dict]]] = None,
+        do_convert_rgb: bool = True,
         do_resize: bool = True,
-        size: bool = SizeDict,
+        size: Optional[SizeDict] = None,
         interpolation: PILImageResampling = PILImageResampling.BICUBIC,
         do_rescale: bool = True,
         rescale_factor: float = 1 / 255.0,
         do_normalize: bool = True,
-        do_sample_frames: bool = True,
         image_mean: Optional[Union[float, list[float]]] = None,
         image_std: Optional[Union[float, list[float]]] = None,
         patch_size: Optional[int] = None,
@@ -176,25 +186,7 @@ def _preprocess(
         return_tensors: Optional[Union[str, TensorType]] = None,
         **kwargs,
     ):
-        timestamps_list = []
-        if do_sample_frames:
-            if video_metadata is None or (isinstance(video_metadata, list) and video_metadata[0] is None):
-                raise ValueError(
-                    "Frame sampling is enabled but no video metadata was found. "
-                    "Please pass in `VideoMetadata` object per each input video or set `do_sample_frames=False`"
-                )
-            processed_videos = []
-            for video, metadata in zip(videos, video_metadata):
-                video, timestamps = self.sample_frames(video, metadata)
-                timestamps_list.append(timestamps)
-                processed_videos.append(video)
-        else:
-            # Assume 24 fps by default and prepare timestamps for the whole video when all frames are sampled
-            processed_videos = videos
-            timestamps_list = [[idx // 24 for idx in range(len(video))] for video in videos]
-            timestamps_list = timestamps_list[::2]  # mrope
-
-        grouped_videos, grouped_videos_index = group_videos_by_shape(processed_videos)
+        grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
         resized_videos_grouped = {}
 
         for shape, stacked_videos in grouped_videos.items():
@@ -271,7 +263,6 @@ def _preprocess(
         data = {
             "pixel_values_videos": pixel_values_videos,
             "video_grid_thw": video_grid_thw,
-            "timestamps": timestamps_list,
         }
 
         return BatchFeature(data=data, tensor_type=return_tensors)
diff --git a/src/transformers/models/got_ocr2/image_processing_got_ocr2.py b/src/transformers/models/got_ocr2/image_processing_got_ocr2.py
index 6a0dca873558..a1a48fa6cf7b 100644
--- a/src/transformers/models/got_ocr2/image_processing_got_ocr2.py
+++ b/src/transformers/models/got_ocr2/image_processing_got_ocr2.py
@@ -339,6 +339,8 @@ def preprocess(
 
         size = size if size is not None else self.size
         size = get_size_dict(size, default_to_square=False)
+
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/idefics/image_processing_idefics.py b/src/transformers/models/idefics/image_processing_idefics.py
index 74a13ff5d11e..190e1d31dc78 100644
--- a/src/transformers/models/idefics/image_processing_idefics.py
+++ b/src/transformers/models/idefics/image_processing_idefics.py
@@ -151,6 +151,7 @@ def preprocess(
         if isinstance(images, list) and len(images) == 0:
             return []
 
+        images = self.fetch_images(images)
         images = make_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/idefics2/image_processing_idefics2.py b/src/transformers/models/idefics2/image_processing_idefics2.py
index 8ad3fe1e142d..2e564708a078 100644
--- a/src/transformers/models/idefics2/image_processing_idefics2.py
+++ b/src/transformers/models/idefics2/image_processing_idefics2.py
@@ -472,6 +472,7 @@ def preprocess(
         do_pad = do_pad if do_pad is not None else self.do_pad
         do_image_splitting = do_image_splitting if do_image_splitting is not None else self.do_image_splitting
 
+        images = self.fetch_images(images)
         images_list = make_nested_list_of_images(images)
 
         if not valid_images(images_list[0]):
diff --git a/src/transformers/models/idefics3/image_processing_idefics3.py b/src/transformers/models/idefics3/image_processing_idefics3.py
index f98413d13350..feae13ab2e45 100644
--- a/src/transformers/models/idefics3/image_processing_idefics3.py
+++ b/src/transformers/models/idefics3/image_processing_idefics3.py
@@ -687,6 +687,7 @@ def preprocess(
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
         do_pad = do_pad if do_pad is not None else self.do_pad
 
+        images = self.fetch_images(images)
         images_list = make_nested_list_of_images(images)
 
         if not valid_images(images_list[0]):
diff --git a/src/transformers/models/instructblipvideo/video_processing_instructblipvideo.py b/src/transformers/models/instructblipvideo/video_processing_instructblipvideo.py
index 55f3a7494c40..805ecda06497 100644
--- a/src/transformers/models/instructblipvideo/video_processing_instructblipvideo.py
+++ b/src/transformers/models/instructblipvideo/video_processing_instructblipvideo.py
@@ -35,7 +35,7 @@
 )
 from ...utils.import_utils import requires
 from ...video_processing_utils import BaseVideoProcessor
-from ...video_utils import VideoMetadata, group_videos_by_shape, reorder_videos
+from ...video_utils import group_videos_by_shape, reorder_videos
 
 
 if is_vision_available():
@@ -76,7 +76,6 @@ def __init__(self, **kwargs: Unpack[InstructBlipVideoVideoProcessorInitKwargs]):
     def _preprocess(
         self,
         videos: list["torch.Tensor"],
-        video_metadata: Union[list[VideoMetadata], list[dict]],
         do_convert_rgb: bool,
         do_resize: bool,
         size: SizeDict,
@@ -88,24 +87,11 @@ def _preprocess(
         do_pad: bool,
         rescale_factor: float,
         do_normalize: bool,
-        do_sample_frames: bool,
         image_mean: Optional[Union[float, list[float]]],
         image_std: Optional[Union[float, list[float]]],
-        fps: Optional[Union[int, float]] = None,
-        num_frames: Optional[int] = None,
         return_tensors: Optional[Union[str, TensorType]] = None,
-        device: Optional["torch.Tensor"] = None,
+        **kwargs,
     ) -> BatchFeature:
-        if do_sample_frames:
-            videos = [
-                self.sample_frames(video, metadata, num_frames, fps) for video, metadata in zip(videos, video_metadata)
-            ]
-
-        # We need to sample frames first before moving to device, if `do_sample_frames=True`. Otherwise
-        # moving the whole video incurs high GPU mem usage for long videos
-        if device is not None:
-            videos = [video.to(device) for video in videos]
-
         # Group videos by size for batched resizing
         grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
         resized_videos_grouped = {}
diff --git a/src/transformers/models/internvl/processing_internvl.py b/src/transformers/models/internvl/processing_internvl.py
index c33b55eea3b2..179dccb63eb3 100644
--- a/src/transformers/models/internvl/processing_internvl.py
+++ b/src/transformers/models/internvl/processing_internvl.py
@@ -21,7 +21,7 @@
 from ...image_utils import ImageInput, concatenate_list, make_flat_list_of_images
 from ...processing_utils import ImagesKwargs, MultiModalData, ProcessingKwargs, ProcessorMixin, Unpack
 from ...tokenization_utils_base import PreTokenizedInput, TextInput
-from ...video_utils import VideoInput, make_batched_videos
+from ...video_utils import VideoInput
 
 
 class InternVLImagesKwargs(ImagesKwargs, total=False):
@@ -216,13 +216,13 @@ def __call__(
         video_patch_indices = np.array([0])
         video_num_patches_indices = np.array([0])
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             images = make_flat_list_of_images(images)
             image_inputs = self.image_processor(images=images, **output_kwargs["images_kwargs"])
             image_num_patches = image_inputs.pop("num_patches")
             image_pixel_values = image_inputs.pop("pixel_values")
             image_num_patches_indices = np.cumsum(image_num_patches)
         if videos is not None:
-            videos = make_batched_videos(videos)
             video_inputs = self.video_processor(videos=videos, **output_kwargs["videos_kwargs"])
             video_pixel_values = video_inputs.pop("pixel_values_videos")
 
@@ -246,7 +246,7 @@ def __call__(
             )
             if images is not None and image_index != len(images):
                 raise ValueError("Number of image placeholders in the prompt does not match the number of images.")
-            if videos is not None and video_index != len(videos):
+            if videos is not None and video_index != len(num_frames_per_video):
                 raise ValueError("Number of video placeholders in the prompt does not match the number of videos.")
 
             # Concatenate the interleaved image and video patches (function agnostic to the patches type (list, numpy array, torch tensor))
diff --git a/src/transformers/models/internvl/video_processing_internvl.py b/src/transformers/models/internvl/video_processing_internvl.py
index 7817eddddbb1..2fc5729119e9 100644
--- a/src/transformers/models/internvl/video_processing_internvl.py
+++ b/src/transformers/models/internvl/video_processing_internvl.py
@@ -73,11 +73,11 @@ def __init__(self, **kwargs: Unpack[InternVLVideoProcessorInitKwargs]):
 
     def sample_frames(
         self,
-        video: "torch.Tensor",
-        metadata: Optional[Union[VideoMetadata, dict]] = None,
+        metadata: VideoMetadata,
         num_frames: Optional[int] = None,
         fps: Optional[Union[int, float]] = None,
         initial_shift: Optional[Union[bool, float, int]] = None,
+        **kwargs,
     ):
         """
         Default sampling function which uniformly samples the desired number of frames between 0 and total number of frames.
@@ -85,9 +85,7 @@ def sample_frames(
         and `fps` are mutually exclusive.
 
         Args:
-            video (`torch.Tensor`):
-                Video that need to be sampled.
-            metadata (`VideoMetadata`, *optional*):
+            metadata (`VideoMetadata`):
                 Metadata of the video containing information about total duration, fps and total number of frames.
             num_frames (`int`, *optional*):
                 Maximum number of frames to sample. Defaults to `self.num_frames`.
@@ -97,21 +95,21 @@ def sample_frames(
                 The initial shift to apply when sampling frames. If `True`, the shift is set so that frames are sampled from the middle of the video.
 
         Returns:
-            torch.Tensor:
-                Sampled video frames.
+            np.ndarray:
+                Indices to sample video frames.
         """
         num_frames = num_frames if num_frames is not None else self.num_frames
         initial_shift = initial_shift if initial_shift is not None else self.initial_shift
-        total_num_frames = video.shape[0]
+        total_num_frames = metadata.total_num_frames
 
         # If num_frames is not given but fps is, calculate num_frames from fps
         if num_frames is None and fps is not None:
-            if metadata is None:
+            if metadata is None or metadata.fps is None:
                 raise ValueError(
                     "Asked to sample `fps` frames per second but no video metadata was provided which is required when sampling with `fps`. "
                     "Please pass in `VideoMetadata` object or use a fixed `num_frames` per input video"
                 )
-            num_frames = int(total_num_frames / metadata["fps"] * fps)
+            num_frames = int(total_num_frames / metadata.fps * fps)
 
         if initial_shift is True:
             initial_shift = total_num_frames / num_frames / 2
@@ -122,13 +120,11 @@ def sample_frames(
             )
 
         indices = torch.arange(initial_shift, total_num_frames, total_num_frames / num_frames).int()
-        video = video[indices].contiguous()
-        return video
+        return indices
 
     def _preprocess(
         self,
         videos: list["torch.Tensor"],
-        video_metadata: Union[list[VideoMetadata], list[dict]],
         do_convert_rgb: bool,
         do_resize: bool,
         size: SizeDict,
@@ -142,25 +138,9 @@ def _preprocess(
         do_normalize: bool,
         image_mean: Optional[Union[float, list[float]]],
         image_std: Optional[Union[float, list[float]]],
-        do_sample_frames: Optional[bool] = None,
-        fps: Optional[Union[int, float]] = None,
-        num_frames: Optional[int] = None,
-        initial_shift: Optional[Union[bool, float, int]] = None,
         return_tensors: Optional[Union[str, TensorType]] = None,
-        device: Optional["torch.Tensor"] = None,
+        **kwargs,
     ) -> BatchFeature:
-        if do_sample_frames:
-            # Sample video frames
-            videos = [
-                self.sample_frames(video, metadata, fps=fps, num_frames=num_frames, initial_shift=initial_shift)
-                for video, metadata in zip(videos, video_metadata)
-            ]
-
-        # We need to sample frames first before moving to device, if `do_sample_frames=True`. Otherwise
-        # moving the whole video incurs high GPU mem usage for long videos
-        if device is not None:
-            videos = [video.to(device) for video in videos]
-
         # Group videos by size for batched resizing
         grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
         resized_videos_grouped = {}
diff --git a/src/transformers/models/janus/image_processing_janus.py b/src/transformers/models/janus/image_processing_janus.py
index 0535f38a33c5..ac2012c62b04 100644
--- a/src/transformers/models/janus/image_processing_janus.py
+++ b/src/transformers/models/janus/image_processing_janus.py
@@ -280,6 +280,7 @@ def preprocess(
 
         size = size if size is not None else self.size
         size = get_size_dict(size, default_to_square=False)
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/llama4/processing_llama4.py b/src/transformers/models/llama4/processing_llama4.py
index 2307ceeffb5c..ce590bc6f40b 100644
--- a/src/transformers/models/llama4/processing_llama4.py
+++ b/src/transformers/models/llama4/processing_llama4.py
@@ -188,6 +188,7 @@ def __call__(
         # Process images
         image_inputs = {}
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             images = make_flat_list_of_images(images)
             image_inputs = self.image_processor(images=images, **output_kwargs["images_kwargs"])
             image_height, image_width = image_inputs["pixel_values"][0].shape[-2:]
diff --git a/src/transformers/models/llava/image_processing_llava.py b/src/transformers/models/llava/image_processing_llava.py
index fa737bd9b9c6..a18a9e66ed8a 100644
--- a/src/transformers/models/llava/image_processing_llava.py
+++ b/src/transformers/models/llava/image_processing_llava.py
@@ -367,6 +367,7 @@ def preprocess(
 
         validate_kwargs(captured_kwargs=kwargs.keys(), valid_processor_keys=self._valid_processor_keys)
 
+        images = self.fetch_images(images)
         images = make_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/llava_next/image_processing_llava_next.py b/src/transformers/models/llava_next/image_processing_llava_next.py
index 21c47701405d..87ac7f1ce61a 100644
--- a/src/transformers/models/llava_next/image_processing_llava_next.py
+++ b/src/transformers/models/llava_next/image_processing_llava_next.py
@@ -640,6 +640,7 @@ def preprocess(
         do_pad = do_pad if do_pad is not None else self.do_pad
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/llava_next_video/image_processing_llava_next_video.py b/src/transformers/models/llava_next_video/image_processing_llava_next_video.py
index 2bb7b43e677d..5508f5dcdd4b 100644
--- a/src/transformers/models/llava_next_video/image_processing_llava_next_video.py
+++ b/src/transformers/models/llava_next_video/image_processing_llava_next_video.py
@@ -356,6 +356,7 @@ def preprocess(
         image_std = image_std if image_std is not None else self.image_std
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
+        images = self.fetch_images(images)
         images = make_batched_videos(images)
         logger.warning(
             "`LlavaNextVideoImageProcessor` is deprecated and will be removed in v5.0. "
diff --git a/src/transformers/models/llava_onevision/image_processing_llava_onevision.py b/src/transformers/models/llava_onevision/image_processing_llava_onevision.py
index 2c924142cbe8..091073b1f4c0 100644
--- a/src/transformers/models/llava_onevision/image_processing_llava_onevision.py
+++ b/src/transformers/models/llava_onevision/image_processing_llava_onevision.py
@@ -692,6 +692,7 @@ def preprocess(
         # only single image patching is supported
         need_patching = [n == 1 for n in batch_num_images for _ in range(n)]
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/mllama/image_processing_mllama.py b/src/transformers/models/mllama/image_processing_mllama.py
index 01ae14c9f5cb..ba1a596aa459 100644
--- a/src/transformers/models/mllama/image_processing_mllama.py
+++ b/src/transformers/models/mllama/image_processing_mllama.py
@@ -692,6 +692,7 @@ def preprocess(
         # extra validation
         _validate_mllama_preprocess_arguments(do_resize, size, do_pad, max_image_tiles)
 
+        images = self.fetch_images(images)
         images_list = make_nested_list_of_images(images)
 
         if self.do_convert_rgb:
diff --git a/src/transformers/models/mllama/processing_mllama.py b/src/transformers/models/mllama/processing_mllama.py
index 0e989b2da7a2..0dae7c834303 100644
--- a/src/transformers/models/mllama/processing_mllama.py
+++ b/src/transformers/models/mllama/processing_mllama.py
@@ -290,6 +290,7 @@ def __call__(
 
         n_images_in_images = [0]
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             images = make_nested_list_of_images(images)
             n_images_in_images = [len(sample) for sample in images]
 
diff --git a/src/transformers/models/paligemma/processing_paligemma.py b/src/transformers/models/paligemma/processing_paligemma.py
index 831112f1de9e..7ab447049800 100644
--- a/src/transformers/models/paligemma/processing_paligemma.py
+++ b/src/transformers/models/paligemma/processing_paligemma.py
@@ -21,7 +21,7 @@
 import numpy as np
 
 from ...feature_extraction_utils import BatchFeature
-from ...image_utils import ImageInput, is_valid_image, make_flat_list_of_images
+from ...image_utils import ImageInput, is_valid_image
 from ...processing_utils import (
     ImagesKwargs,
     MultiModalData,
@@ -275,7 +275,6 @@ def __call__(
                     )
                     for prompt, image_list in zip(text, images)
                 ]
-                images = make_flat_list_of_images(images)
             else:
                 expanded_samples = []
                 for sample in text:
diff --git a/src/transformers/models/pixtral/image_processing_pixtral.py b/src/transformers/models/pixtral/image_processing_pixtral.py
index 152373b7e548..6022560ba01a 100644
--- a/src/transformers/models/pixtral/image_processing_pixtral.py
+++ b/src/transformers/models/pixtral/image_processing_pixtral.py
@@ -32,7 +32,7 @@
     get_image_size,
     infer_channel_dimension_format,
     is_scaled_image,
-    make_list_of_images,
+    make_flat_list_of_images,
     to_numpy_array,
     valid_images,
     validate_kwargs,
@@ -397,7 +397,8 @@ def preprocess(
 
         validate_kwargs(captured_kwargs=kwargs.keys(), valid_processor_keys=self._valid_processor_keys)
 
-        images = make_list_of_images(images)
+        images = self.fetch_images(images)
+        images = make_flat_list_of_images(images)
 
         if not valid_images(images[0]):
             raise ValueError(
diff --git a/src/transformers/models/pixtral/processing_pixtral.py b/src/transformers/models/pixtral/processing_pixtral.py
index b59aa840ff87..42edbe24f1f5 100644
--- a/src/transformers/models/pixtral/processing_pixtral.py
+++ b/src/transformers/models/pixtral/processing_pixtral.py
@@ -21,7 +21,7 @@
 import numpy as np
 
 from ...feature_extraction_utils import BatchFeature
-from ...image_utils import ImageInput, is_valid_image, load_image
+from ...image_utils import ImageInput, is_valid_image
 from ...processing_utils import (
     MultiModalData,
     ProcessingKwargs,
@@ -166,21 +166,6 @@ def __call__(
         patch_size = self.patch_size * self.spatial_merge_size
 
         if images is not None:
-            if is_image_or_image_url(images):
-                images = [images]
-            elif isinstance(images, (list, tuple)) and is_image_or_image_url(images[0]):
-                pass
-            elif (
-                isinstance(images, (list, tuple))
-                and isinstance(images[0], (list, tuple))
-                and is_image_or_image_url(images[0][0])
-            ):
-                images = [image for sublist in images for image in sublist]
-            else:
-                raise ValueError(
-                    "Invalid input images. Please provide a single image, a list of images, or a list of lists of images."
-                )
-            images = [load_image(im) if isinstance(im, str) else im for im in images]
             image_inputs = self.image_processor(images, patch_size=patch_size, **output_kwargs["images_kwargs"])
         else:
             image_inputs = {}
diff --git a/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py b/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py
index 3b3f764a84df..e7f2f3f5b66f 100644
--- a/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py
+++ b/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py
@@ -27,7 +27,7 @@
 from ...image_utils import ImageInput
 from ...processing_utils import ImagesKwargs, ProcessingKwargs, ProcessorMixin, Unpack, VideosKwargs
 from ...tokenization_utils_base import AudioInput, PreTokenizedInput, TextInput
-from ...video_utils import VideoInput, make_batched_videos
+from ...video_utils import VideoInput
 
 
 class Qwen2_5_OmniVideosKwargs(VideosKwargs):
@@ -154,7 +154,6 @@ def __call__(
         seconds_per_chunk = output_kwargs["videos_kwargs"].pop("seconds_per_chunk")
         position_id_per_seconds = output_kwargs["videos_kwargs"].pop("position_id_per_seconds")
         use_audio_in_video = output_kwargs["videos_kwargs"].pop("use_audio_in_video")
-        fps = output_kwargs["videos_kwargs"].get("fps", 2.0)
 
         if audio is not None:
             output_kwargs["audio_kwargs"]["padding"] = "max_length"  # Support "max_length" padding only here
@@ -179,14 +178,15 @@ def __call__(
             image_grid_thw = iter([])
 
         if videos is not None:
-            videos = make_batched_videos(videos)
             videos_inputs = self.video_processor(videos=videos, **output_kwargs["videos_kwargs"])
-            fps = [fps] * len(videos)
-            videos_inputs["video_second_per_grid"] = [
-                self.video_processor.temporal_patch_size / fps[i] for i in range(len(fps))
-            ]
-            video_grid_thw = iter(videos_inputs["video_grid_thw"])
-            video_second_per_grid = iter(videos_inputs["video_second_per_grid"])
+
+            fps = output_kwargs["videos_kwargs"].get("fps", 2.0)
+            video_grid_thw = videos_inputs["video_grid_thw"]
+            second_per_grid_ts = [self.video_processor.temporal_patch_size / fps] * len(video_grid_thw)
+            videos_inputs["video_second_per_grid"] = second_per_grid_ts
+
+            video_grid_thw = iter(video_grid_thw)
+            video_second_per_grid = iter(second_per_grid_ts)
         else:
             videos_inputs = {}
             video_grid_thw = iter([])
diff --git a/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py b/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py
index e1529594d076..dbde67be1e69 100644
--- a/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py
+++ b/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py
@@ -405,6 +405,7 @@ def preprocess(
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
         if images is not None:
+            images = self.fetch_images(images)
             images = make_flat_list_of_images(images)
 
         if images is not None and not valid_images(images):
diff --git a/src/transformers/models/qwen2_vl/video_processing_qwen2_vl.py b/src/transformers/models/qwen2_vl/video_processing_qwen2_vl.py
index 6c62a568f6cf..f73a65484219 100644
--- a/src/transformers/models/qwen2_vl/video_processing_qwen2_vl.py
+++ b/src/transformers/models/qwen2_vl/video_processing_qwen2_vl.py
@@ -134,13 +134,13 @@ def __init__(self, **kwargs: Unpack[Qwen2VLVideoProcessorInitKwargs]):
 
     def sample_frames(
         self,
-        video: "torch.Tensor",
-        frame_factor: int,
-        min_frames: int,
-        max_frames: int,
-        metadata: Optional[Union[VideoMetadata, dict]] = None,
+        metadata: VideoMetadata,
+        temporal_patch_size: Optional[int] = None,
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
         num_frames: Optional[int] = None,
         fps: Optional[Union[int, float]] = None,
+        **kwargs,
     ):
         """
         Default sampling function which uniformly samples the desired number of frames between 0 and total number of frames.
@@ -148,45 +148,46 @@ def sample_frames(
         and `fps` are mutually exclusive.
 
         Args:
-            video (`torch.Tensor`):
-                Video that need to be sampled.
-            frame_factor (`int`):
+            metadata (`VideoMetadata`):
+                Metadata of the video containing information about total duration, fps and total number of frames.
+            temporal_patch_size (`int`, *optional*):
                 The temporal patch size of the vision encoder. Number of sampled frames will be rounded to be divisible by frame factor.
-            min_frames (`int`):
+            min_frames (`int`, *optional*):
                 The minimum number of frames that can be sampled.
-            max_frames (`int`):
+            max_frames (`int`, *optional*):
                 The maximum number of frames that can be sampled.
-            metadata (`VideoMetadata`, *optional*):
-                Metadata of the video containing information about total duration, fps and total number of frames.
             num_frames (`int`, *optional*):
                 Maximum number of frames to sample. Defaults to `self.num_frames`.
             fps (`int` or `float`, *optional*):
                 Target frames to sample per second. Defaults to `self.fps`.
 
         Returns:
-            torch.Tensor:
-                Sampled video frames.
+            np.ndarray:
+                Indices to sample video frames.
         """
         if fps is not None and num_frames is not None:
             raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
 
         num_frames = num_frames if num_frames is not None else self.num_frames
         fps = fps if fps is not None else self.fps
-        total_num_frames = video.shape[0]
+        temporal_patch_size = temporal_patch_size if temporal_patch_size is not None else self.temporal_patch_size
+        min_frames = min_frames if min_frames is not None else self.min_frames
+        max_frames = max_frames if max_frames is not None else self.max_frames
+        total_num_frames = metadata.total_num_frames
 
         # If num_frames is not given but fps is, calculate num_frames from fps
         if num_frames is not None:
-            num_frames = round(num_frames / frame_factor) * frame_factor
+            num_frames = round(num_frames / temporal_patch_size) * temporal_patch_size
         elif fps is not None:
-            if metadata is None:
+            if metadata is None or metadata.fps is None:
                 raise ValueError(
                     "Asked to sample `fps` frames per second but no video metadata was provided which is required when sampling with `fps`. "
                     "Please pass in `VideoMetadata` object or use a fixed `num_frames` per input video"
                 )
-            max_frames = math.floor(min(max_frames, total_num_frames) / frame_factor) * frame_factor
-            num_frames = total_num_frames / metadata["fps"] * fps
+            max_frames = math.floor(min(max_frames, total_num_frames) / temporal_patch_size) * temporal_patch_size
+            num_frames = total_num_frames / metadata.fps * fps
             num_frames = min(min(max(num_frames, min_frames), max_frames), total_num_frames)
-            num_frames = math.floor(num_frames / frame_factor) * frame_factor
+            num_frames = math.floor(num_frames / temporal_patch_size) * temporal_patch_size
 
         if num_frames > total_num_frames:
             raise ValueError(
@@ -198,14 +199,12 @@ def sample_frames(
             indices = torch.arange(0, total_num_frames, total_num_frames / num_frames).int()
         else:
             indices = torch.arange(0, total_num_frames).int()
-        video = video[indices].contiguous()
 
-        return video
+        return indices
 
     def _preprocess(
         self,
         videos: list["torch.Tensor"],
-        video_metadata: Union[list[VideoMetadata], list[dict]],
         do_convert_rgb: bool,
         do_resize: bool,
         size: SizeDict,
@@ -213,7 +212,6 @@ def _preprocess(
         do_rescale: bool,
         rescale_factor: float,
         do_normalize: bool,
-        do_sample_frames: bool,
         image_mean: Optional[Union[float, list[float]]],
         image_std: Optional[Union[float, list[float]]],
         min_pixels: Optional[int] = None,
@@ -221,34 +219,10 @@ def _preprocess(
         patch_size: Optional[int] = None,
         temporal_patch_size: Optional[int] = None,
         merge_size: Optional[int] = None,
-        fps: Optional[Union[int, float]] = None,
-        num_frames: Optional[int] = None,
-        min_frames: Optional[int] = None,
-        max_frames: Optional[int] = None,
         return_tensors: Optional[Union[str, TensorType]] = None,
         device: Optional["torch.Tensor"] = None,
         **kwargs,
     ):
-        if do_sample_frames:
-            # Sample video frames
-            videos = [
-                self.sample_frames(
-                    video,
-                    frame_factor=temporal_patch_size,
-                    min_frames=min_frames,
-                    max_frames=max_frames,
-                    metadata=metadata,
-                    num_frames=num_frames,
-                    fps=fps,
-                )
-                for video, metadata in zip(videos, video_metadata)
-            ]
-
-        # We need to sample frames first before moving to device, if `do_sample_frames=True`. Otherwise
-        # moving the whole video incurs high GPU mem usage for long videos
-        if device is not None:
-            videos = [video.to(device) for video in videos]
-
         # Group videos by size for batched resizing
         grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
         resized_videos_grouped = {}
diff --git a/src/transformers/models/sam/image_processing_sam_fast.py b/src/transformers/models/sam/image_processing_sam_fast.py
index fa4ff5020e92..77b4b490e136 100644
--- a/src/transformers/models/sam/image_processing_sam_fast.py
+++ b/src/transformers/models/sam/image_processing_sam_fast.py
@@ -36,6 +36,7 @@
     ImageInput,
     PILImageResampling,
     SizeDict,
+    pil_torch_interpolation_mapping,
 )
 from ...processing_utils import Unpack
 from ...utils import (
@@ -185,11 +186,19 @@ def _further_process_kwargs(
         kwargs["pad_size"] = pad_size
         kwargs["mask_size"] = mask_size
         kwargs["mask_pad_size"] = mask_pad_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["data_format"] = data_format
 
+        # torch resize uses interpolation instead of resample
+        # Check if resample is an int before checking if it's an instance of PILImageResampling
+        # because if pillow < 9.1.0, resample is an int and PILImageResampling is a module.
+        # Checking PILImageResampling will fail with error `TypeError: isinstance() arg 2 must be a type or tuple of types`.
+        resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[resample] if isinstance(resample, (PILImageResampling, int)) else resample
+        )
+
         return kwargs
 
     @auto_docstring
diff --git a/src/transformers/models/sam2/image_processing_sam2_fast.py b/src/transformers/models/sam2/image_processing_sam2_fast.py
index d68e41fc6d60..4b65bec77b57 100644
--- a/src/transformers/models/sam2/image_processing_sam2_fast.py
+++ b/src/transformers/models/sam2/image_processing_sam2_fast.py
@@ -427,11 +427,19 @@ def _further_process_kwargs(
 
         kwargs["size"] = size
         kwargs["mask_size"] = mask_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["data_format"] = data_format
 
+        # torch resize uses interpolation instead of resample
+        # Check if resample is an int before checking if it's an instance of PILImageResampling
+        # because if pillow < 9.1.0, resample is an int and PILImageResampling is a module.
+        # Checking PILImageResampling will fail with error `TypeError: isinstance() arg 2 must be a type or tuple of types`.
+        resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[resample] if isinstance(resample, (PILImageResampling, int)) else resample
+        )
+
         return kwargs
 
     @auto_docstring
diff --git a/src/transformers/models/sam2/modular_sam2.py b/src/transformers/models/sam2/modular_sam2.py
index debaf3cb40cf..e9578df3cdbf 100644
--- a/src/transformers/models/sam2/modular_sam2.py
+++ b/src/transformers/models/sam2/modular_sam2.py
@@ -60,8 +60,6 @@
     TensorType,
     auto_docstring,
     is_torch_available,
-    is_torchvision_available,
-    is_torchvision_v2_available,
     logging,
 )
 from ..auto import AutoModel
@@ -78,11 +76,6 @@
     import torch
     from torch.nn import functional as F
 
-if is_torchvision_v2_available():
-    pass
-elif is_torchvision_available():
-    pass
-
 
 logger = logging.get_logger(__name__)
 
@@ -213,11 +206,19 @@ def _further_process_kwargs(
 
         kwargs["size"] = size
         kwargs["mask_size"] = mask_size
-        kwargs["default_to_square"] = default_to_square
         kwargs["image_mean"] = image_mean
         kwargs["image_std"] = image_std
         kwargs["data_format"] = data_format
 
+        # torch resize uses interpolation instead of resample
+        # Check if resample is an int before checking if it's an instance of PILImageResampling
+        # because if pillow < 9.1.0, resample is an int and PILImageResampling is a module.
+        # Checking PILImageResampling will fail with error `TypeError: isinstance() arg 2 must be a type or tuple of types`.
+        resample = kwargs.pop("resample")
+        kwargs["interpolation"] = (
+            pil_torch_interpolation_mapping[resample] if isinstance(resample, (PILImageResampling, int)) else resample
+        )
+
         return kwargs
 
     def _apply_non_overlapping_constraints(self, pred_masks: torch.Tensor) -> torch.Tensor:
diff --git a/src/transformers/models/siglip/image_processing_siglip.py b/src/transformers/models/siglip/image_processing_siglip.py
index bde8913fc13e..152447bb5b87 100644
--- a/src/transformers/models/siglip/image_processing_siglip.py
+++ b/src/transformers/models/siglip/image_processing_siglip.py
@@ -181,6 +181,7 @@ def preprocess(
         image_std = image_std if image_std is not None else self.image_std
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/siglip2/image_processing_siglip2.py b/src/transformers/models/siglip2/image_processing_siglip2.py
index 54dc1360226f..30b5f1b958af 100644
--- a/src/transformers/models/siglip2/image_processing_siglip2.py
+++ b/src/transformers/models/siglip2/image_processing_siglip2.py
@@ -267,6 +267,7 @@ def preprocess(
         # Image processor does not support different output formats, because it returns patches.
         data_format = ChannelDimension.LAST
 
+        images = self.fetch_images(images)
         images = make_flat_list_of_images(images)
 
         if not valid_images(images):
diff --git a/src/transformers/models/smolvlm/image_processing_smolvlm.py b/src/transformers/models/smolvlm/image_processing_smolvlm.py
index 2c7e34a9864f..d92d9a3385eb 100644
--- a/src/transformers/models/smolvlm/image_processing_smolvlm.py
+++ b/src/transformers/models/smolvlm/image_processing_smolvlm.py
@@ -684,6 +684,7 @@ def preprocess(
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
         do_pad = do_pad if do_pad is not None else self.do_pad
 
+        images = self.fetch_images(images)
         images_list = make_nested_list_of_images(images)
 
         if not valid_images(images_list[0]):
diff --git a/src/transformers/models/smolvlm/processing_smolvlm.py b/src/transformers/models/smolvlm/processing_smolvlm.py
index c8dd26631bf0..e9ae2099a43f 100644
--- a/src/transformers/models/smolvlm/processing_smolvlm.py
+++ b/src/transformers/models/smolvlm/processing_smolvlm.py
@@ -120,6 +120,9 @@ class SmolVLMProcessorKwargs(ProcessingKwargs, total=False):
         "images_kwargs": {
             "return_row_col_info": True,
         },
+        "videos_kwargs": {
+            "return_metadata": True,
+        },
     }
 
 
@@ -174,23 +177,7 @@ def __init__(
 
         super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template, **kwargs)
 
-    def process_vision(self, text, images, output_kwargs):
-        if text is not None:
-            n_images_in_text = [sample.count(self.image_token) for sample in text]
-
-        n_images_in_images = [len(sublist) for sublist in images]
-        image_inputs = self.image_processor(images, **output_kwargs["images_kwargs"])
-
-        if text is None:
-            return None, image_inputs
-
-        if n_images_in_images != n_images_in_text:
-            raise ValueError(
-                f"The number of images in the text {n_images_in_text} and images {n_images_in_images} should be the same."
-            )
-        image_rows = image_inputs.pop("rows", [[0] * len(text)])
-        image_cols = image_inputs.pop("cols", [[0] * len(text)])
-
+    def expand_text_with_image_tokens(self, text, image_rows, image_cols):
         prompt_strings = []
         for sample, sample_rows, sample_cols in zip(text, image_rows, image_cols):
             # Replace the image token with fake tokens around the expanded image token sequence of length `image_seq_len`
@@ -216,32 +203,25 @@ def process_vision(self, text, images, output_kwargs):
                 sample += image_prompt_string + split_sample[i + 1]
             prompt_strings.append(sample)
 
-        return prompt_strings, image_inputs
-
-    def process_video(self, text, videos, output_kwargs):
-        if text is not None:
-            n_videos_in_text = [sample.count(self.video_token) for sample in text]
-
-        n_videos_in_videos = [len(sublist) for sublist in videos]
-        video_inputs = self.video_processor(videos, **output_kwargs["videos_kwargs"])
+        return prompt_strings
 
+    def expand_text_with_video_tokens(self, text, video_inputs):
         num_frames = video_inputs["pixel_values"].shape[1]
-        batch_timestamps = iter(video_inputs.pop("timestamps"))
-        batch_durations = iter(video_inputs.pop("durations"))
-
-        if text is None:
-            return None, video_inputs
-
-        if n_videos_in_videos != n_videos_in_text:
-            raise ValueError(
-                f"The number of videos in the text {n_videos_in_text} and videos {n_videos_in_videos} should be the same."
-            )
+        video_metadata = iter(video_inputs["video_metadata"])
 
         prompt_strings = []
         for sample in text:
             while self.video_token in sample:
-                timestamps = next(batch_timestamps)
-                duration = next(batch_durations)
+                metadata = next(video_metadata)
+                if metadata.fps is None:
+                    logger.warning_once(
+                        "SmolVLM requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
+                        "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
+                        "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+                    )
+                    metadata.fps = 24  # Set the default fps to 24 for BC, otherwise `timestamps` can't be inferred
+                timestamps = [(int(second // 60), int(second % 60)) for second in metadata.timestamps]
+                duration = int(metadata.duration) if metadata.duration is not None else int(metadata.timestamps[-1])
                 duration_td = timedelta(seconds=int(duration))
                 image_prompt_strings = DEFAULT_VIDEO_INTRO.format(
                     frame_count=num2words(num_frames), video_duration=str(duration_td)
@@ -260,7 +240,7 @@ def process_video(self, text, videos, output_kwargs):
                 image_prompt_strings += DEFAULT_MEDIA_OUTTRO
                 sample = sample.replace(self.video_token, image_prompt_strings, 1)
             prompt_strings.append(sample)
-        return prompt_strings, video_inputs
+        return prompt_strings
 
     def __call__(
         self,
@@ -341,19 +321,38 @@ def __call__(
         inputs = {}
         # Images and videos are mutually exclusive, so process one which is present
         if images is not None:
+            images = self.image_processor.fetch_images(images)
             images = make_nested_list_of_images(images)
-            text, vision_inputs = self.process_vision(
-                text,
-                images,
-                output_kwargs,
-            )
+            vision_inputs = self.image_processor(images, **output_kwargs["images_kwargs"])
+
+            image_rows = vision_inputs.pop("rows", [[0] * len(text)])
+            image_cols = vision_inputs.pop("cols", [[0] * len(text)])
             inputs.update(vision_inputs)
+
+            if text is not None:
+                n_images_in_text = [sample.count(self.image_token) for sample in text]
+                n_images_in_images = [len(sublist) for sublist in images]
+                if n_images_in_images != n_images_in_text:
+                    raise ValueError(
+                        f"The number of images in the text {n_images_in_text} and images {n_images_in_images} should be the same."
+                    )
+                text = self.expand_text_with_image_tokens(text, image_rows=image_rows, image_cols=image_cols)
+
         elif videos is not None:
-            text, vision_inputs = self.process_video(
-                text,
-                videos,
-                output_kwargs,
-            )
+            vision_inputs = self.video_processor(videos, **output_kwargs["videos_kwargs"])
+            if text is not None:
+                n_videos_in_text = [sample.count(self.video_token) for sample in text]
+                n_videos_in_videos = [len(sublist) for sublist in videos]
+                if n_videos_in_videos != n_videos_in_text:
+                    raise ValueError(
+                        f"The number of videos in the text {n_videos_in_text} and videos {n_videos_in_videos} should be the same."
+                    )
+                text = self.expand_text_with_video_tokens(text, vision_inputs)
+
+            # If user has not requested video metadata, pop it. By default metadata
+            # is always returned to expand video tokens correctly
+            if "return_metadata" not in kwargs:
+                vision_inputs.pop("video_metadata")
             inputs.update(vision_inputs)
 
         return_tensors = output_kwargs["text_kwargs"].pop("return_tensors", None)
diff --git a/src/transformers/models/smolvlm/video_processing_smolvlm.py b/src/transformers/models/smolvlm/video_processing_smolvlm.py
index b66facefa41b..020f2d4c8e93 100644
--- a/src/transformers/models/smolvlm/video_processing_smolvlm.py
+++ b/src/transformers/models/smolvlm/video_processing_smolvlm.py
@@ -246,11 +246,11 @@ def pad(
 
     def sample_frames(
         self,
-        video: "torch.Tensor",
-        metadata: Union[VideoMetadata, dict],
+        metadata: VideoMetadata,
         num_frames: Optional[int] = None,
         fps: Optional[Union[int, float]] = None,
         skip_secs: Optional[int] = 1,
+        **kwargs,
     ):
         """
         Video sampling function which:
@@ -260,8 +260,6 @@ def sample_frames(
             - Uniformly samples the desired number of frames between the start and end indices.
 
         Args:
-            video (`torch.Tensor`):
-                Video that need to be sampled.
             metadata (`VideoMetadata`):
                 Metadata of the video containing information about total duration, fps and total number of frames.
             num_frames (`int`, *optional*):
@@ -272,13 +270,18 @@ def sample_frames(
                 Number of seconds to skip from the start and end if the video is long enough.
 
         Returns:
-            torch.Tensor:
-                Sampled video frames.
+            np.ndarray:
+                Indices to sample video frames.
         """
+        if metadata is None or getattr(metadata, "fps", None) is None:
+            raise ValueError(
+                "Asked to sample frames per second but no video metadata was provided which is required when sampling in SmolVLM. "
+                "Please pass in `VideoMetadata` object or set `do_sample_frames=False`"
+            )
+
         num_frames = num_frames if num_frames is not None else self.num_frames
         fps = fps if fps is not None else self.fps
-
-        total_num_frames = video.shape[0]
+        total_num_frames = metadata.total_num_frames
 
         # Step 1) Estimate how many frames we'd sample at `target_fps`, fallback if target_fps <= 0
         estimated_frames = int(round(fps * metadata["duration"]))
@@ -303,20 +306,12 @@ def sample_frames(
 
         indices = np.linspace(start_idx, end_idx, desired_frames, dtype=int)
         indices = np.unique(indices)
-        video = video[indices].contiguous()
 
-        timestamps = []
-        for idx in indices:
-            sec = idx / metadata["fps"]
-            mm = int(sec // 60)
-            ss = int(sec % 60)
-            timestamps.append([mm, ss])
-        return video, timestamps, int(metadata["duration"])
+        return indices
 
     def _preprocess(
         self,
         videos: list["torch.Tensor"],
-        video_metadata: Union[list[VideoMetadata], list[dict]],
         do_convert_rgb: bool,
         do_resize: bool,
         size: SizeDict,
@@ -325,44 +320,12 @@ def _preprocess(
         rescale_factor: float,
         do_normalize: bool,
         do_pad: bool,
-        do_sample_frames: bool,
         image_mean: Optional[Union[float, list[float]]],
         image_std: Optional[Union[float, list[float]]],
-        fps: Optional[Union[int, float]] = None,
-        num_frames: Optional[int] = None,
-        skip_secs: Optional[int] = 0,
         return_tensors: Optional[Union[str, TensorType]] = None,
-        device: Optional["torch.Tensor"] = None,
         **kwargs,
     ):
-        # Group videos by size for batched resizing
-        if do_sample_frames:
-            if video_metadata[0] is None:
-                raise ValueError(
-                    "Frame sampling is enabled but no video metadata was found. SmolVLM requires metadata to correctly sample frames. "
-                    "Please pass in `VideoMetadata` object per each input video or set `do_sample_frames=False`"
-                )
-            processed_videos = []
-            timestamps_list, durations_list = [], []
-            for video, metadata in zip(videos, video_metadata):
-                video, timestamps, duration = self.sample_frames(video, metadata, num_frames, fps, skip_secs)
-                timestamps_list.append(timestamps)
-                durations_list.append(duration)
-                processed_videos.append(video)
-        else:
-            # Assume 24 fps by default and prepare timestamps for the whole video when all frames are sampled
-            processed_videos = videos
-            timestamps_list = [
-                [(int((idx / 24) // 60), int((idx / 24) % 60)) for idx in range(len(video))] for video in videos
-            ]
-            durations_list = [len(video) // 24 for video in videos]
-
-        # We need to sample frames first before moving to device, if `do_sample_frames=True`. Otherwise
-        # moving the whole video incurs high GPU mem usage for long videos
-        if device is not None:
-            videos = [video.to(device) for video in videos]
-
-        grouped_videos, grouped_videos_index = group_videos_by_shape(processed_videos)
+        grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
         resized_videos_grouped = {}
         for shape, stacked_videos in grouped_videos.items():
             if do_convert_rgb:
@@ -400,7 +363,7 @@ def _preprocess(
             pixel_attention_mask = reorder_videos(processed_padded_mask_grouped, grouped_videos_index)
 
         processed_videos = torch.stack(processed_videos, dim=0) if return_tensors else processed_videos
-        data = {"pixel_values": processed_videos, "timestamps": timestamps_list, "durations": durations_list}
+        data = {"pixel_values": processed_videos}
 
         if do_pad:
             data["pixel_attention_mask"] = (
diff --git a/src/transformers/models/video_llava/image_processing_video_llava.py b/src/transformers/models/video_llava/image_processing_video_llava.py
index fa3aeeeb2b3a..47b51869f600 100644
--- a/src/transformers/models/video_llava/image_processing_video_llava.py
+++ b/src/transformers/models/video_llava/image_processing_video_llava.py
@@ -257,6 +257,7 @@ def preprocess(
         do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
 
         if images is not None:
+            images = self.fetch_images(images)
             images = make_list_of_images(images)
 
         if images is not None and not valid_images(images):
diff --git a/src/transformers/processing_utils.py b/src/transformers/processing_utils.py
index eb836db1ff27..ec2e0dd57518 100644
--- a/src/transformers/processing_utils.py
+++ b/src/transformers/processing_utils.py
@@ -31,14 +31,12 @@
 import typing_extensions
 from huggingface_hub.errors import EntryNotFoundError
 
-from transformers.utils import is_torch_available
-
 from .audio_utils import load_audio
 from .dynamic_module_utils import custom_object_save
 from .feature_extraction_utils import BatchFeature
-from .image_utils import ChannelDimension, is_vision_available, load_image
+from .image_utils import ChannelDimension, is_vision_available
 from .utils.chat_template_utils import render_jinja_template
-from .video_utils import VideoMetadata, load_video
+from .video_utils import VideoMetadata
 
 
 if is_vision_available():
@@ -66,6 +64,7 @@
     download_url,
     is_offline_mode,
     is_remote_url,
+    is_torch_available,
     list_repo_templates,
     logging,
 )
@@ -250,7 +249,7 @@ class VideosKwargs(TypedDict, total=False):
             Whether to center crop the video.
         do_sample_frames (`bool`, *optional*):
             Whether to sample frames from the video before processing or to process the whole video.
-        video_metadata (`VideoMetadata`, *optional*):
+        video_metadata (`Union[VideoMetadata, dict]`, *optional*):
             Metadata of the video containing information about total duration, fps and total number of frames.
         num_frames (`int`, *optional*):
             Maximum number of frames to sample when `do_sample_frames=True`.
@@ -262,6 +261,8 @@ class VideosKwargs(TypedDict, total=False):
             The channel dimension format for the output video.
         input_data_format (`ChannelDimension` or `str`, *optional*):
             The channel dimension format for the input video.
+        return_metadata (`ChannelDimension` or `str`, *optional*):
+            Whether to return video metadata or not.
     """
 
     do_convert_rgb: Optional[bool]
@@ -285,6 +286,7 @@ class VideosKwargs(TypedDict, total=False):
     video_metadata: Optional[Union[VideoMetadata, dict]]
     fps: Optional[Union[int, float]]
     num_frames: Optional[int]
+    return_metadata: Optional[bool]
 
 
 class AudioKwargs(TypedDict, total=False):
@@ -430,23 +432,11 @@ class ChatTemplateLoadKwargs(TypedDict, total=False):
 
     num_frames (`int`, *optional*):
         Number of frames to sample uniformly. If not passed, the whole video is loaded.
-    video_load_backend (`str`, *optional*, defaults to `"pyav"`):
-        The backend to use when loading the video which will be used only when there are videos in the conversation.
-        Can be any of ["decord", "pyav", "opencv", "torchvision"]. Defaults to "pyav" because it is the only backend
-        that supports all types of sources to load from.
-    sample_indices_fn (`Callable`, *optional*):
-            A callable function that will return indices at which the video should be sampled. If the video has to be loaded using
-            by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`.
-            If not provided, simple uniformt sampling with fps is performed, otherwise `sample_indices_fn` has priority over other args.
-            The function expects at input the all args along with all kwargs passed to `load_video` and should output valid
-            indices at which the video should be sampled. For example:
-
-            def sample_indices_fn(num_frames, fps, metadata, **kwargs):
-                # add you sampling logic here ...
-                return np.linspace(start_idx, end_idx, num_frames, dtype=int)
+    load_audio_from_video (`bool`, *optional*):
+            Whether to use the audio track of input video. If `True` the audio track will be loaded and passed to the
+            processor. This flag has no effect if the model doesn't support audio modality.
     """
 
-    video_load_backend: Optional[str] = "pyav"
     sampling_rate: Optional[int] = 16_000
     load_audio_from_video: Optional[bool] = False
 
@@ -1438,6 +1428,11 @@ def validate_init_kwargs(processor_config, valid_kwargs):
         return unused_kwargs, valid_kwargs
 
     @deprecate_kwarg("video_fps", version="4.58", new_name="fps")
+    @deprecate_kwarg(
+        "video_load_backend",
+        version="4.59",
+        additional_message=". This function will use `torchcodec` by default, or `torchvision` if `torchcodec` is not installed.",
+    )
     def apply_chat_template(
         self,
         conversation: Union[list[dict[str, str]], list[list[dict[str, str]]]],
@@ -1525,6 +1520,9 @@ def apply_chat_template(
                 if value is not None and not isinstance(value, dict):
                     processed_kwargs[kwarg_type][key] = value
 
+        # pop unused and deprecated kwarg
+        kwargs.pop("video_load_backend", None)
+
         # Pass unprocessed custom kwargs
         processed_kwargs["template_kwargs"].update(kwargs)
 
@@ -1544,10 +1542,7 @@ def apply_chat_template(
         if tokenize:
             batch_images, batch_videos = [], []
             batch_audios = []
-            batch_video_metadata = []
             for conversation in conversations:
-                images, videos = [], []
-                video_metadata = []
                 for message in conversation:
                     visuals = [content for content in message["content"] if content["type"] in ["image", "video"]]
                     audio_fnames = [
@@ -1569,9 +1564,6 @@ def apply_chat_template(
                         if key in vision_info and vision_info["type"] == "video"
                     ]
 
-                    for fname in image_fnames:
-                        images.append(load_image(fname))
-
                     # Audio models do not accept nested list of audios (yet!) so we construct a flat input audio list
                     if not mm_load_kwargs["load_audio_from_video"]:
                         for fname in audio_fnames:
@@ -1580,32 +1572,12 @@ def apply_chat_template(
                         for fname in video_fnames:
                             batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
 
-                    for fname in video_fnames:
-                        if isinstance(fname, (list, tuple)) and isinstance(fname[0], str):
-                            # Case a: Video is provided as a list of image file names
-                            video = [np.array(load_image(image_fname)) for image_fname in fname]
-                            video = np.stack(video)
-                            metadata = None
-                            logger.warning(
-                                "When loading the video from list of images, we cannot infer metadata such as `fps` or `duration`. "
-                                "If your model requires metadata during processing, please load the whole video and let the processor sample frames instead."
-                            )
-                        else:
-                            # Case b: Video is provided as a single file path or URL or decoded frames in a np.ndarray or torch.tensor
-                            video, metadata = load_video(
-                                fname,
-                                backend=mm_load_kwargs["video_load_backend"],
-                            )
-                        videos.append(video)
-                        video_metadata.append(metadata)
-
-                # Currently all processors can accept nested list of batches, but not flat list of visuals
-                # So we'll make a batched list of images and let the processor handle it
-                if images:
-                    batch_images.append(images)
-                if videos:
-                    batch_videos.append(videos)
-                    batch_video_metadata.append(video_metadata)
+                    # Currently all processors can accept nested list of batches, but not flat list of visuals
+                    # So we'll make a batched list of images and let the processor handle it
+                    if image_fnames:
+                        batch_images.append(image_fnames)
+                    if video_fnames:
+                        batch_videos.append(video_fnames)
 
         prompt, generation_indices = render_jinja_template(
             conversations=conversations,
@@ -1638,7 +1610,6 @@ def apply_chat_template(
                 images=batch_images if batch_images else None,
                 videos=batch_videos if batch_videos else None,
                 audio=batch_audios if batch_audios else None,
-                video_metadata=batch_video_metadata,
                 **kwargs,
             )
 
diff --git a/src/transformers/video_processing_utils.py b/src/transformers/video_processing_utils.py
index 51df926ed658..17781ec8a3e9 100644
--- a/src/transformers/video_processing_utils.py
+++ b/src/transformers/video_processing_utils.py
@@ -17,7 +17,8 @@
 import os
 import warnings
 from copy import deepcopy
-from typing import Any, Optional, Union
+from functools import partial
+from typing import Any, Callable, Optional, Union
 
 import numpy as np
 
@@ -43,9 +44,9 @@
     is_offline_mode,
     is_remote_url,
     is_torch_available,
+    is_torchcodec_available,
     is_torchvision_available,
     is_torchvision_v2_available,
-    is_vision_available,
     logging,
 )
 from .utils.import_utils import requires
@@ -53,22 +54,19 @@
     VideoInput,
     VideoMetadata,
     group_videos_by_shape,
+    is_valid_video,
     load_video,
+    make_batched_metadata,
     make_batched_videos,
     reorder_videos,
     to_channel_dimension_format,
 )
 
 
-if is_vision_available():
-    from .image_utils import PILImageResampling
-
 if is_torch_available():
     import torch
 
 if is_torchvision_available():
-    from .image_utils import pil_torch_interpolation_mapping
-
     if is_torchvision_v2_available():
         from torchvision.transforms.v2 import functional as F
     else:
@@ -141,7 +139,10 @@
             - `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format.
             - `"none"` or `ChannelDimension.NONE`: video in (height, width) format.
         device (`torch.device`, *optional*):
-            The device to process the videos on. If unset, the device is inferred from the input videos."""
+            The device to process the videos on. If unset, the device is inferred from the input videos.
+        return_metadata (`bool`, *optional*):
+            Whether to return video metadata or not.
+        """
 
 
 @add_start_docstrings(
@@ -170,6 +171,7 @@ class BaseVideoProcessor(BaseImageProcessorFast):
     fps = None
     num_frames = None
     video_metadata = None
+    return_metadata = False
     valid_kwargs = VideosKwargs
     model_input_names = ["pixel_values_videos"]
 
@@ -234,10 +236,10 @@ def convert_to_rgb(
 
     def sample_frames(
         self,
-        video: "torch.Tensor",
-        metadata: Optional[Union[VideoMetadata, dict]] = None,
+        metadata: VideoMetadata,
         num_frames: Optional[int] = None,
         fps: Optional[Union[int, float]] = None,
+        **kwargs,
     ):
         """
         Default sampling function which uniformly samples the desired number of frames between 0 and total number of frames.
@@ -245,9 +247,7 @@ def sample_frames(
         and `fps` are mutually exclusive.
 
         Args:
-            video (`torch.Tensor`):
-                Video that need to be sampled.
-            metadata (`VideoMetadata`, *optional*):
+            metadata (`VideoMetadata`):
                 Metadata of the video containing information about total duration, fps and total number of frames.
             num_frames (`int`, *optional*):
                 Maximum number of frames to sample. Defaults to `self.num_frames`.
@@ -255,8 +255,8 @@ def sample_frames(
                 Target frames to sample per second. Defaults to `self.fps`.
 
         Returns:
-            torch.Tensor:
-                Sampled video frames.
+            np.ndarray:
+                Indices to sample video frames.
         """
         if fps is not None and num_frames is not None:
             raise ValueError(
@@ -265,16 +265,16 @@ def sample_frames(
 
         num_frames = num_frames if num_frames is not None else self.num_frames
         fps = fps if fps is not None else self.fps
-        total_num_frames = video.shape[0]
+        total_num_frames = metadata.total_num_frames
 
         # If num_frames is not given but fps is, calculate num_frames from fps
         if num_frames is None and fps is not None:
-            if metadata is None:
+            if metadata is None or metadata.fps is None:
                 raise ValueError(
                     "Asked to sample `fps` frames per second but no video metadata was provided which is required when sampling with `fps`. "
                     "Please pass in `VideoMetadata` object or use a fixed `num_frames` per input video"
                 )
-            num_frames = int(total_num_frames / metadata["fps"] * fps)
+            num_frames = int(total_num_frames / metadata.fps * fps)
 
         if num_frames > total_num_frames:
             raise ValueError(
@@ -285,25 +285,53 @@ def sample_frames(
             indices = torch.arange(0, total_num_frames, total_num_frames / num_frames).int()
         else:
             indices = torch.arange(0, total_num_frames).int()
+        return indices
 
-        video = video[indices].contiguous()
-        return video
+    def _decode_and_sample_videos(
+        self,
+        videos: VideoInput,
+        video_metadata: Union[VideoMetadata, dict],
+        do_sample_frames: Optional[bool] = None,
+        sample_indices_fn: Optional[Callable] = None,
+    ) -> list["torch.Tensor"]:
+        """
+        Decode input videos and sample frames if needed.
+        """
+        videos = make_batched_videos(videos)
+        video_metadata = make_batched_metadata(videos, video_metadata=video_metadata)
+
+        # Only sample frames if an array video is passed, otherwise first decode -> then sample
+        if is_valid_video(videos[0]) and do_sample_frames:
+            sampled_videos = []
+            for video, metadata in zip(videos, video_metadata):
+                indices = sample_indices_fn(metadata=metadata)
+                sampled_videos.append(video[indices])
+            videos = sampled_videos
+        elif not is_valid_video(videos[0]):
+            if isinstance(videos[0], list):
+                # Videos sometimes are passed as a list of image URLs, especially through templates
+                videos = [
+                    torch.stack([F.pil_to_tensor(image) for image in images], dim=0)
+                    for images in self.fetch_images(videos)
+                ]
+                if do_sample_frames:
+                    raise ValueError(
+                        "Sampling frames from a list of images is not supported! Set `do_sample_frames=False`."
+                    )
+            else:
+                videos, video_metadata = self.fetch_videos(videos, sample_indices_fn=sample_indices_fn)
+
+        return videos, video_metadata
 
     def _prepare_input_videos(
         self,
         videos: VideoInput,
-        video_metadata: VideoMetadata = None,
         input_data_format: Optional[Union[str, ChannelDimension]] = None,
+        device: Optional[str] = None,
     ) -> list["torch.Tensor"]:
         """
         Prepare the input videos for processing.
         """
-        videos = make_batched_videos(videos)
-        if video_metadata is not None:
-            batch_metadata = [metadata for batch_list in video_metadata for metadata in batch_list]
-        else:
-            batch_metadata = [None] * len(videos)
-
         processed_videos = []
         for video in videos:
             # `make_batched_videos` always returns a 4D array per video
@@ -312,10 +340,15 @@ def _prepare_input_videos(
                 # not using F.to_tensor as it doesn't handle (C, H, W) numpy arrays
                 video = torch.from_numpy(video).contiguous()
 
+            if device is not None:
+                video = video.to(device)
+
             processed_videos.append(video)
-        return processed_videos, batch_metadata
+        return processed_videos
 
-    @add_start_docstrings(BASE_VIDEO_PROCESSOR_DOCSTRING)
+    @add_start_docstrings(
+        BASE_VIDEO_PROCESSOR_DOCSTRING,
+    )
     def preprocess(
         self,
         videos: VideoInput,
@@ -331,30 +364,34 @@ def preprocess(
             kwargs.setdefault(kwarg_name, getattr(self, kwarg_name, None))
 
         input_data_format = kwargs.pop("input_data_format")
+        do_sample_frames = kwargs.pop("do_sample_frames")
+        device = kwargs.pop("device")
         video_metadata = kwargs.pop("video_metadata")
-        videos, video_metadata = self._prepare_input_videos(
-            videos=videos, video_metadata=video_metadata, input_data_format=input_data_format
+
+        sample_indices_fn = partial(self.sample_frames, **kwargs) if do_sample_frames else None
+        videos, video_metadata = self._decode_and_sample_videos(
+            videos,
+            video_metadata=video_metadata,
+            do_sample_frames=do_sample_frames,
+            sample_indices_fn=sample_indices_fn,
         )
+        videos = self._prepare_input_videos(videos=videos, input_data_format=input_data_format, device=device)
 
         kwargs = self._further_process_kwargs(**kwargs)
         self._validate_preprocess_kwargs(**kwargs)
 
-        # torch resize uses interpolation instead of resample
-        resample = kwargs.pop("resample")
-        kwargs["interpolation"] = (
-            pil_torch_interpolation_mapping[resample] if isinstance(resample, (PILImageResampling, int)) else resample
-        )
-
         # Pop kwargs that are not needed in _preprocess
-        kwargs.pop("default_to_square")
         kwargs.pop("data_format")
+        return_metadata = kwargs.pop("return_metadata")
 
-        return self._preprocess(videos=videos, video_metadata=video_metadata, **kwargs)
+        preprocessed_videos = self._preprocess(videos=videos, **kwargs)
+        if return_metadata:
+            preprocessed_videos["video_metadata"] = video_metadata
+        return preprocessed_videos
 
     def _preprocess(
         self,
         videos: list["torch.Tensor"],
-        video_metadata: Union[list[VideoMetadata], list[dict]],
         do_convert_rgb: bool,
         do_resize: bool,
         size: SizeDict,
@@ -368,24 +405,9 @@ def _preprocess(
         do_normalize: bool,
         image_mean: Optional[Union[float, list[float]]],
         image_std: Optional[Union[float, list[float]]],
-        do_sample_frames: Optional[bool] = None,
-        fps: Optional[Union[int, float]] = None,
-        num_frames: Optional[int] = None,
         return_tensors: Optional[Union[str, TensorType]] = None,
-        device: Optional["torch.Tensor"] = None,
+        **kwargs,
     ) -> BatchFeature:
-        if do_sample_frames:
-            # Sample video frames
-            videos = [
-                self.sample_frames(video, metadata=metadata, num_frames=num_frames, fps=fps)
-                for video, metadata in zip(videos, video_metadata)
-            ]
-
-        # We need to sample frames first before moving to device, if `do_sample_frames=True`. Otherwise
-        # moving the whole video incurs high GPU mem usage for long videos
-        if device is not None:
-            videos = [video.to(device) for video in videos]
-
         # Group videos by size for batched resizing
         grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
         resized_videos_grouped = {}
@@ -861,19 +883,25 @@ def register_for_auto_class(cls, auto_class="AutoVideoProcessor"):
 
         cls._auto_class = auto_class
 
-    def fetch_videos(self, video_url_or_urls: Union[str, list[str]]):
+    def fetch_videos(self, video_url_or_urls: Union[str, list[str], list[list[str]]], sample_indices_fn=None):
         """
         Convert a single or a list of urls into the corresponding `np.array` objects.
 
         If a single url is passed, the return value will be a single object. If a list is passed a list of objects is
         returned.
         """
+        backend = "torchcodec"
+        if not is_torchcodec_available():
+            warnings.warn(
+                "`torchcodec` is not installed and cannot be used to decode the video by default. "
+                "Falling back to `torchvision`. Note that `torchvision` decoding is deprecated and will be removed in future versions. "
+            )
+            backend = "torchvision"
+
         if isinstance(video_url_or_urls, list):
-            return [self.fetch_videos(x) for x in video_url_or_urls]
-        elif isinstance(video_url_or_urls, str):
-            return load_video(video_url_or_urls)
+            return list(zip(*[self.fetch_videos(x, sample_indices_fn=sample_indices_fn) for x in video_url_or_urls]))
         else:
-            raise TypeError(f"only a single or a list of entries is supported but got type={type(video_url_or_urls)}")
+            return load_video(video_url_or_urls, backend=backend, sample_indices_fn=sample_indices_fn)
 
 
 BaseVideoProcessor.push_to_hub = copy_func(BaseVideoProcessor.push_to_hub)
diff --git a/src/transformers/video_utils.py b/src/transformers/video_utils.py
index ea5a93f97ca7..7576b05857df 100644
--- a/src/transformers/video_utils.py
+++ b/src/transformers/video_utils.py
@@ -19,7 +19,7 @@
 from contextlib import redirect_stdout
 from dataclasses import dataclass
 from io import BytesIO
-from typing import Callable, Optional, Union
+from typing import Callable, NewType, Optional, Union
 from urllib.parse import urlparse
 
 import numpy as np
@@ -56,6 +56,8 @@
 
 logger = logging.get_logger(__name__)
 
+URL = NewType("URL", str)
+Path = NewType("Path", str)
 
 VideoInput = Union[
     list["PIL.Image.Image"],
@@ -66,19 +68,43 @@
     list[list["PIL.Image.Image"]],
     list[list["np.ndarrray"]],
     list[list["torch.Tensor"]],
+    URL,
+    list[URL],
+    list[list[URL]],
+    Path,
+    list[Path],
+    list[list[Path]],
 ]  # noqa
 
 
 @dataclass
 class VideoMetadata:
     total_num_frames: int
-    fps: float
-    duration: float
-    video_backend: str
+    fps: float = None
+    width: int = None
+    height: int = None
+    duration: float = None
+    video_backend: str = None
+    frames_indices: list[int] = None
 
     def __getitem__(self, item):
         return getattr(self, item)
 
+    def __setitem__(self, key, value):
+        return setattr(self, key, value)
+
+    @property
+    def timestamps(self) -> float:
+        "Timestamps of the sampled frames in seconds."
+        if self.fps is None:
+            raise ValueError("Cannot infer video `timestamps` when `fps` is None.")
+        return [frame_idx / self.fps for frame_idx in self.frames_indices]
+
+    def update(self, dictionary):
+        for key, value in dictionary.items():
+            if hasattr(self, key):
+                setattr(self, key, value)
+
 
 def is_valid_video_frame(frame):
     return isinstance(frame, PIL.Image.Image) or (
@@ -130,7 +156,7 @@ def convert_pil_frames_to_video(videos: list[VideoInput]) -> list[Union["np.ndar
             Video inputs to turn into a list of videos.
     """
 
-    if not isinstance(videos[0], (list, tuple)):
+    if not (isinstance(videos[0], (list, tuple)) and is_valid_image(videos[0][0])):
         return videos
 
     video_converted = []
@@ -141,7 +167,7 @@ def convert_pil_frames_to_video(videos: list[VideoInput]) -> list[Union["np.ndar
     return video_converted
 
 
-def make_batched_videos(videos) -> list[Union["np.ndarray", "torch.Tensor"]]:
+def make_batched_videos(videos) -> list[Union["np.ndarray", "torch.Tensor", "URL", "Path"]]:
     """
     Ensure that the input is a list of videos. If the input is a single video, it is converted to a list of length 1.
     If the input is a batch of videos, it is converted to a list of 4D video arrays. Videos passed as list `PIL.Image`
@@ -153,23 +179,64 @@ def make_batched_videos(videos) -> list[Union["np.ndarray", "torch.Tensor"]]:
         videos (`VideoInput`):
             Video inputs to turn into a list of videos.
     """
-    if not valid_videos:
+    # Early exit for deeply nested list of image frame paths. We shouldn't flatten them
+    try:
+        if isinstance(videos[0][0][0], str):
+            return [image_paths for sublist in videos for image_paths in sublist]
+    except (IndexError, TypeError):
+        pass
+
+    if isinstance(videos, str) or is_valid_video(videos):
+        return convert_pil_frames_to_video([videos])
+    # only one frame passed, thus we unsqueeze time dim
+    elif is_valid_image(videos):
+        return [np.array(videos)[None, ...]]
+    elif not isinstance(videos, list):
         raise ValueError(
             f"Invalid video input. Expected either a list of video frames or an input of 4 or 5 dimensions, but got"
             f" type {type(videos)}."
         )
 
-    if is_batched_video(videos):
-        pass
-    elif is_valid_video(videos):
-        videos = [videos]
-    # only one frame passed, thus we unsqueeze time dim
-    elif is_valid_image(videos):
-        videos = [np.array(videos)[None, ...]]
-    # nested batch so we need to unflatten
-    elif isinstance(videos[0], (list, tuple)) and is_valid_video(videos[0][0]):
-        videos = [video for sublist in videos for video in sublist]
-    return convert_pil_frames_to_video(videos)
+    # Recursively flatten any nested structure
+    flat_videos_list = []
+    for item in videos:
+        if isinstance(item, str) or is_valid_video(item):
+            flat_videos_list.append(item)
+        elif isinstance(item, list):
+            flat_videos_list.extend(make_batched_videos(item))
+
+    flat_videos_list = convert_pil_frames_to_video(flat_videos_list)
+    return flat_videos_list
+
+
+def make_batched_metadata(videos: VideoInput, video_metadata: Union[VideoMetadata, dict]):
+    if video_metadata is None:
+        # Create default metadata and fill attrbiutes we can infer from given video
+        video_metadata = [
+            {
+                "total_num_frames": len(video),
+                "fps": None,
+                "duration": None,
+                "frames_indices": list(range(len(video))),
+                "height": get_video_size(video)[0] if is_valid_video(video) else None,
+                "width": get_video_size(video)[1] if is_valid_video(video) else None,
+            }
+            for video in videos
+        ]
+
+    if isinstance(video_metadata, list):
+        # Flatten if nested list
+        if isinstance(video_metadata[0], list):
+            video_metadata = [
+                VideoMetadata(**metadata) for metadata_list in video_metadata for metadata in metadata_list
+            ]
+        # Simply wrap in VideoMetadata if simple dict
+        elif isinstance(video_metadata[0], dict):
+            video_metadata = [VideoMetadata(**metadata) for metadata in video_metadata]
+    else:
+        # Create a batched list from single object
+        video_metadata = [VideoMetadata(**video_metadata)]
+    return video_metadata
 
 
 def get_video_size(video: np.ndarray, channel_dim: ChannelDimension = None) -> tuple[int, int]:
@@ -186,7 +253,7 @@ def get_video_size(video: np.ndarray, channel_dim: ChannelDimension = None) -> t
         A tuple of the video's height and width.
     """
     if channel_dim is None:
-        channel_dim = infer_channel_dimension_format(video)
+        channel_dim = infer_channel_dimension_format(video, num_channels=(1, 3, 4))
 
     if channel_dim == ChannelDimension.FIRST:
         return video.shape[-2], video.shape[-1]
@@ -253,7 +320,7 @@ def default_sample_indices_fn(metadata: VideoMetadata, num_frames=None, fps=None
 
 
 def read_video_opencv(
-    video_path: str,
+    video_path: Union["URL", "Path"],
     sample_indices_fn: Callable,
     **kwargs,
 ):
@@ -285,7 +352,12 @@ def sample_indices_fn(metadata, **kwargs):
     video_fps = video.get(cv2.CAP_PROP_FPS)
     duration = total_num_frames / video_fps if video_fps else 0
     metadata = VideoMetadata(
-        total_num_frames=int(total_num_frames), fps=float(video_fps), duration=float(duration), video_backend="opencv"
+        total_num_frames=int(total_num_frames),
+        fps=float(video_fps),
+        duration=float(duration),
+        video_backend="opencv",
+        height=int(video.get(cv2.CAP_PROP_FRAME_HEIGHT)),
+        width=int(video.get(cv2.CAP_PROP_FRAME_WIDTH)),
     )
     indices = sample_indices_fn(metadata=metadata, **kwargs)
 
@@ -310,8 +382,8 @@ def sample_indices_fn(metadata, **kwargs):
 
 
 def read_video_decord(
-    video_path: str,
-    sample_indices_fn: Optional[Callable] = None,
+    video_path: Union["URL", "Path"],
+    sample_indices_fn: Callable,
     **kwargs,
 ):
     """
@@ -320,7 +392,7 @@ def read_video_decord(
     Args:
         video_path (`str`):
             Path to the video file.
-        sample_indices_fn (`Callable`, *optional*):
+        sample_indices_fn (`Callable`):
             A callable function that will return indices at which the video should be sampled. If the video has to be loaded using
             by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`.
             If not provided, simple uniform sampling with fps is performed.
@@ -342,18 +414,27 @@ def sample_indices_fn(metadata, **kwargs):
     total_num_frames = len(vr)
     duration = total_num_frames / video_fps if video_fps else 0
     metadata = VideoMetadata(
-        total_num_frames=int(total_num_frames), fps=float(video_fps), duration=float(duration), video_backend="decord"
+        total_num_frames=int(total_num_frames),
+        fps=float(video_fps),
+        duration=float(duration),
+        video_backend="decord",
     )
 
     indices = sample_indices_fn(metadata=metadata, **kwargs)
-
-    frames = vr.get_batch(indices).asnumpy()
-    metadata.frames_indices = indices
-    return frames, metadata
+    video = vr.get_batch(indices).asnumpy()
+
+    metadata.update(
+        {
+            "frames_indices": indices,
+            "height": video.shape[1],
+            "width": video.shape[2],
+        }
+    )
+    return video, metadata
 
 
 def read_video_pyav(
-    video_path: str,
+    video_path: Union["URL", "Path"],
     sample_indices_fn: Callable,
     **kwargs,
 ):
@@ -385,7 +466,12 @@ def sample_indices_fn(metadata, **kwargs):
     video_fps = container.streams.video[0].average_rate  # should we better use `av_guess_frame_rate`?
     duration = total_num_frames / video_fps if video_fps else 0
     metadata = VideoMetadata(
-        total_num_frames=int(total_num_frames), fps=float(video_fps), duration=float(duration), video_backend="pyav"
+        total_num_frames=int(total_num_frames),
+        fps=float(video_fps),
+        duration=float(duration),
+        video_backend="pyav",
+        height=container.streams.video[0].height,
+        width=container.streams.video[0].width,
     )
     indices = sample_indices_fn(metadata=metadata, **kwargs)
 
@@ -404,7 +490,7 @@ def sample_indices_fn(metadata, **kwargs):
 
 
 def read_video_torchvision(
-    video_path: str,
+    video_path: Union["URL", "Path"],
     sample_indices_fn: Callable,
     **kwargs,
 ):
@@ -423,8 +509,8 @@ def sample_indices_fn(metadata, **kwargs):
                 return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int)
 
     Returns:
-        tuple[`np.array`, `VideoMetadata`]: A tuple containing:
-            - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]).
+        tuple[`torch.Tensor`, `VideoMetadata`]: A tuple containing:
+            - Torch tensor of frames in RGB (shape: [num_frames, height, width, 3]).
             - `VideoMetadata` object.
     """
     warnings.warn(
@@ -436,7 +522,7 @@ def sample_indices_fn(metadata, **kwargs):
         start_pts=0.0,
         end_pts=None,
         pts_unit="sec",
-        output_format="THWC",
+        output_format="TCHW",
     )
     video_fps = info["video_fps"]
     total_num_frames = video.size(0)
@@ -450,13 +536,19 @@ def sample_indices_fn(metadata, **kwargs):
 
     indices = sample_indices_fn(metadata=metadata, **kwargs)
 
-    video = video[indices].contiguous().numpy()
-    metadata.frames_indices = indices
+    video = video[indices].contiguous()
+    metadata.update(
+        {
+            "frames_indices": indices,
+            "height": video.shape[1],
+            "width": video.shape[2],
+        }
+    )
     return video, metadata
 
 
 def read_video_torchcodec(
-    video_path: str,
+    video_path: Union["URL", "Path"],
     sample_indices_fn: Callable,
     **kwargs,
 ):
@@ -466,7 +558,7 @@ def read_video_torchcodec(
     Args:
         video_path (`str`):
             Path to the video file.
-        sample_indices_fn (`Callable`, *optional*):
+        sample_indices_fn (`Callable`):
             A callable function that will return indices at which the video should be sampled. If the video has to be loaded using
             by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`.
             If not provided, simple uniform sampling with fps is performed.
@@ -476,7 +568,7 @@ def sample_indices_fn(metadata, **kwargs):
 
     Returns:
         Tuple[`torch.Tensor`, `VideoMetadata`]: A tuple containing:
-            - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]).
+            - Torch tensor of frames in RGB (shape: [num_frames, height, width, 3]).
             - `VideoMetadata` object.
     """
     # Lazy import torchcodec
@@ -485,15 +577,19 @@ def sample_indices_fn(metadata, **kwargs):
 
     decoder = VideoDecoder(
         video_path,
-        dimension_order="NHWC",  # to be consistent with other decoders
         # Interestingly `exact` mode takes less than approximate when we load the whole video
         seek_mode="exact",
+        # Allow FFmpeg decide on the number of threads for efficiency
+        num_ffmpeg_threads=0,
+        device=kwargs.get("device"),
     )
     metadata = VideoMetadata(
         total_num_frames=decoder.metadata.num_frames,
         fps=decoder.metadata.average_fps,
         duration=decoder.metadata.duration_seconds,
         video_backend="torchcodec",
+        height=decoder.metadata.height,
+        width=decoder.metadata.width,
     )
     indices = sample_indices_fn(metadata=metadata, **kwargs)
 
@@ -512,7 +608,7 @@ def sample_indices_fn(metadata, **kwargs):
 
 
 def load_video(
-    video: Union[str, "VideoInput"],
+    video: VideoInput,
     num_frames: Optional[int] = None,
     fps: Optional[Union[int, float]] = None,
     backend: str = "pyav",
@@ -523,7 +619,7 @@ def load_video(
     Loads `video` to a numpy array.
 
     Args:
-        video (`str` or `VideoInput`):
+        video (`VideoInput`):
             The video to convert to the numpy array format. Can be a link to video or local path.
         num_frames (`int`, *optional*):
             Number of frames to sample uniformly. If not passed, the whole video is loaded.
@@ -563,13 +659,10 @@ def sample_indices_fn_func(metadata, **fn_kwargs):
 
         sample_indices_fn = sample_indices_fn_func
 
-    if is_valid_image(video) or (isinstance(video, (list, tuple)) and is_valid_image(video[0])):
-        # Case 1: Video is provided as a 4D numpy array or torch tensor (frames, height, width, channels)
-        if not is_valid_video(video):
-            raise ValueError(
-                f"When passing video as decoded frames, video should be a 4D numpy array or torch tensor, but got {video.ndim} dimensions instead."
-            )
-        return video, None
+    # Early exit if provided an array or `PIL` frames
+    if not isinstance(video, str):
+        metadata = [None] * len(video)
+        return video, metadata
 
     if urlparse(video).netloc in ["www.youtube.com", "youtube.com"]:
         if not is_yt_dlp_available():
@@ -593,13 +686,8 @@ def sample_indices_fn_func(metadata, **fn_kwargs):
     # can also load with decord, but not cv2/torchvision
     # both will fail in case of url links
     video_is_url = video.startswith("http://") or video.startswith("https://")
-    if video_is_url and backend in ["opencv", "torchvision"]:
-        raise ValueError(
-            "If you are trying to load a video from URL, you can decode the video only with `pyav`, `decord` or `torchcodec` as backend"
-        )
-
-    if file_obj is None:
-        return video
+    if video_is_url and backend in ["opencv"]:
+        raise ValueError("If you are trying to load a video from URL, you cannot use 'opencv' as backend")
 
     if (
         (not is_decord_available() and backend == "decord")
diff --git a/tests/models/glm4v/test_processor_glm4v.py b/tests/models/glm4v/test_processor_glm4v.py
index b5d6a2a9d7e8..7025294a0f73 100644
--- a/tests/models/glm4v/test_processor_glm4v.py
+++ b/tests/models/glm4v/test_processor_glm4v.py
@@ -29,7 +29,6 @@
 if is_vision_available():
     from transformers import Glm4vProcessor
 
-
 if is_torch_available():
     import torch
 
@@ -163,16 +162,6 @@ def _test_apply_chat_template(
         for k in out_dict:
             self.assertIsInstance(out_dict[k], return_tensor_to_type[return_tensors])
 
-    @require_av
-    @unittest.skip("GLM4V can't sample frames from image frames")
-    def test_apply_chat_template_video_1(self):
-        pass
-
-    @require_av
-    @unittest.skip("GLM4V can't sample frames from image frames")
-    def test_apply_chat_template_video_2(self):
-        pass
-
     @require_av
     def test_apply_chat_template_video_frame_sampling(self):
         processor = self.get_processor()
@@ -224,7 +213,7 @@ def test_apply_chat_template_video_frame_sampling(self):
             video_fps=video_fps,
         )
         self.assertTrue(self.videos_input_name in out_dict_with_video)
-        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 40)
+        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 20)
 
         # Load without any arg should load the whole video
         out_dict_with_video = processor.apply_chat_template(
@@ -232,10 +221,9 @@ def test_apply_chat_template_video_frame_sampling(self):
             add_generation_prompt=True,
             tokenize=True,
             return_dict=True,
-            do_sample_frames=False,
         )
         self.assertTrue(self.videos_input_name in out_dict_with_video)
-        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 600)
+        self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 40)
 
         # Load video as a list of frames (i.e. images). NOTE: each frame should have same size
         # because we assume they come from one video
@@ -256,6 +244,19 @@ def test_apply_chat_template_video_frame_sampling(self):
         self.assertTrue(self.videos_input_name in out_dict_with_video)
         self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 4)
 
+        # When the inputs are frame URLs/paths we expect that those are already
+        # sampled and will raise an error is asked to sample again.
+        with self.assertRaisesRegex(
+            ValueError, "Sampling frames from a list of images is not supported! Set `do_sample_frames=False`"
+        ):
+            out_dict_with_video = processor.apply_chat_template(
+                messages,
+                add_generation_prompt=True,
+                tokenize=True,
+                return_dict=True,
+                do_sample_frames=True,
+            )
+
     def test_model_input_names(self):
         processor = self.get_processor()
 
diff --git a/tests/models/glm4v/test_video_processing_glm4v.py b/tests/models/glm4v/test_video_processing_glm4v.py
index 5f597aae79ca..1dcd4bdecca6 100644
--- a/tests/models/glm4v/test_video_processing_glm4v.py
+++ b/tests/models/glm4v/test_video_processing_glm4v.py
@@ -96,7 +96,7 @@ def prepare_video_metadata(self, videos):
             metadata = {
                 "fps": 2,
                 "duration": num_frames / 2,
-                "total_frames": num_frames,
+                "total_num_frames": num_frames,
             }
             video_metadata.append(metadata)
         return video_metadata
diff --git a/tests/models/qwen2_5_omni/test_processing_qwen2_5_omni.py b/tests/models/qwen2_5_omni/test_processing_qwen2_5_omni.py
index 2a584efe8099..f9231d3b905a 100644
--- a/tests/models/qwen2_5_omni/test_processing_qwen2_5_omni.py
+++ b/tests/models/qwen2_5_omni/test_processing_qwen2_5_omni.py
@@ -519,11 +519,21 @@ def test_apply_chat_template_video_frame_sampling(self):
         self.assertTrue(self.videos_input_name in out_dict_with_video)
         self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 2904)
 
+        # When the inputs are frame URLs/paths we expect that those are already
+        # sampled and will raise an error is asked to sample again.
+        with self.assertRaisesRegex(
+            ValueError, "Sampling frames from a list of images is not supported! Set `do_sample_frames=False`"
+        ):
+            out_dict_with_video = processor.apply_chat_template(
+                messages,
+                add_generation_prompt=True,
+                tokenize=True,
+                return_dict=True,
+                do_sample_frames=True,
+            )
+
     @require_librosa
     @require_av
-    @unittest.skip(
-        "@raushan: librosa can'r decode this audio in CI runner, fix after adding moviepy or another decoder"
-    )
     def test_chat_template_audio_from_video(self):
         processor = self.get_processor()
         if processor.chat_template is None:
@@ -570,7 +580,7 @@ def test_chat_template_audio_from_video(self):
             add_generation_prompt=True,
             tokenize=True,
             return_dict=True,
-            return_tensors="np",
+            return_tensors="pt",
             load_audio_from_video=True,
         )
         self.assertTrue(self.audio_input_name in out_dict)
diff --git a/tests/models/qwen2_5_vl/test_processing_qwen2_5_vl.py b/tests/models/qwen2_5_vl/test_processing_qwen2_5_vl.py
index 879f07526fd7..8aaf869e329b 100644
--- a/tests/models/qwen2_5_vl/test_processing_qwen2_5_vl.py
+++ b/tests/models/qwen2_5_vl/test_processing_qwen2_5_vl.py
@@ -338,6 +338,19 @@ def test_apply_chat_template_video_frame_sampling(self):
         self.assertTrue(self.videos_input_name in out_dict_with_video)
         self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 160)
 
+        # When the inputs are frame URLs/paths we expect that those are already
+        # sampled and will raise an error is asked to sample again.
+        with self.assertRaisesRegex(
+            ValueError, "Sampling frames from a list of images is not supported! Set `do_sample_frames=False`"
+        ):
+            out_dict_with_video = processor.apply_chat_template(
+                messages,
+                add_generation_prompt=True,
+                tokenize=True,
+                return_dict=True,
+                do_sample_frames=True,
+            )
+
     def test_kwargs_overrides_custom_image_processor_kwargs(self):
         processor = self.get_processor()
         self.skip_processor_without_typed_kwargs(processor)
diff --git a/tests/models/qwen2_vl/test_processing_qwen2_vl.py b/tests/models/qwen2_vl/test_processing_qwen2_vl.py
index b346a1b802be..2d6e9ef5da17 100644
--- a/tests/models/qwen2_vl/test_processing_qwen2_vl.py
+++ b/tests/models/qwen2_vl/test_processing_qwen2_vl.py
@@ -338,6 +338,19 @@ def test_apply_chat_template_video_frame_sampling(self):
         self.assertTrue(self.videos_input_name in out_dict_with_video)
         self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 160)
 
+        # When the inputs are frame URLs/paths we expect that those are already
+        # sampled and will raise an error is asked to sample again.
+        with self.assertRaisesRegex(
+            ValueError, "Sampling frames from a list of images is not supported! Set `do_sample_frames=False`"
+        ):
+            out_dict_with_video = processor.apply_chat_template(
+                messages,
+                add_generation_prompt=True,
+                tokenize=True,
+                return_dict=True,
+                do_sample_frames=True,
+            )
+
     def test_kwargs_overrides_custom_image_processor_kwargs(self):
         processor = self.get_processor()
         self.skip_processor_without_typed_kwargs(processor)
diff --git a/tests/models/qwen2_vl/test_video_processing_qwen2_vl.py b/tests/models/qwen2_vl/test_video_processing_qwen2_vl.py
index 097c35edebe5..4ffb70fc40df 100644
--- a/tests/models/qwen2_vl/test_video_processing_qwen2_vl.py
+++ b/tests/models/qwen2_vl/test_video_processing_qwen2_vl.py
@@ -326,11 +326,6 @@ def test_call_sample_frames(self):
             self.assertListEqual(list(encoded_videos.shape), expected_output_video_shape)
             self.assertListEqual(list(encoded_videos_batched.shape), expected_output_video_shape_batched)
 
-            # Sample with `fps` requires metadata to infer number of frames from total duration
-            with self.assertRaises(ValueError):
-                encoded_videos = video_processing(video_inputs[0], return_tensors="pt", fps=3)[self.input_name]
-                encoded_videos_batched = video_processing(video_inputs, return_tensors="pt", fps=3)[self.input_name]
-
             metadata = [[{"duration": 2.0, "total_num_frames": 8, "fps": 4}]]
             batched_metadata = metadata * len(video_inputs)
             encoded_videos = video_processing(video_inputs[0], return_tensors="pt", fps=3, video_metadata=metadata)[
diff --git a/tests/models/smolvlm/test_video_processing_smolvlm.py b/tests/models/smolvlm/test_video_processing_smolvlm.py
index fa229af99ce9..22e7c1d4f7bd 100644
--- a/tests/models/smolvlm/test_video_processing_smolvlm.py
+++ b/tests/models/smolvlm/test_video_processing_smolvlm.py
@@ -136,15 +136,6 @@ def test_call_sample_frames(self):
             metadata = [[{"duration": 2.0, "total_num_frames": 8, "fps": 4}]]
             batched_metadata = metadata * len(video_inputs)
 
-            # Sample with `fps` requires metadata to infer number of frames from total duration
-            with self.assertRaises(ValueError):
-                encoded_videos = video_processing(video_inputs[0], return_tensors="pt", num_frames=6, fps=3)[
-                    self.input_name
-                ]
-                encoded_videos_batched = video_processing(video_inputs, return_tensors="pt", num_frames=6, fps=3)[
-                    self.input_name
-                ]
-
             encoded_videos = video_processing(
                 video_inputs[0], return_tensors="pt", num_frames=6, fps=3, video_metadata=metadata
             )[self.input_name]
@@ -154,14 +145,5 @@ def test_call_sample_frames(self):
             self.assertEqual(encoded_videos.shape[1], 6)
             self.assertEqual(encoded_videos_batched.shape[1], 6)
 
-            # We should raise error when asked to sample more frames than there are in input video
-            with self.assertRaises(ValueError):
-                encoded_videos = video_processing(video_inputs[0], return_tensors="pt", fps=10, num_frames=20)[
-                    self.input_name
-                ]
-                encoded_videos_batched = video_processing(video_inputs, return_tensors="pt", fps=10, num_frames=20)[
-                    self.input_name
-                ]
-
             # Assign back the actual num frames in tester
             self.video_processor_tester.num_frames = prev_num_frames
diff --git a/tests/test_processing_common.py b/tests/test_processing_common.py
index d4ec8d183674..97506e6f1207 100644
--- a/tests/test_processing_common.py
+++ b/tests/test_processing_common.py
@@ -51,7 +51,7 @@
     ],
     "videos": [
         "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/Big_Buck_Bunny_720_10s_10MB.mp4",
-        ["https://www.ilankelman.org/stopsigns/australia.jpg", "https://www.ilankelman.org/stopsigns/australia.jpg"],
+        "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4",
     ],
     "audio": [
         "https://huggingface.co/datasets/raushan-testing-hf/audio-test/resolve/main/glass-breaking-151256.mp3",
@@ -977,7 +977,7 @@ def test_apply_chat_template_audio(self, batch_size: int, return_tensors: str):
         )
 
     @require_av
-    @parameterized.expand([(1, "pt"), (2, "pt"), (3, "pt")])  # video processor supports only torchvision
+    @parameterized.expand([(1, "pt"), (2, "pt")])  # video processor supports only torchvision
     def test_apply_chat_template_video(self, batch_size: int, return_tensors: str):
         self._test_apply_chat_template(
             "video", batch_size, return_tensors, "videos_input_name", "video_processor", MODALITY_INPUT_DATA["videos"]
@@ -1082,8 +1082,8 @@ def test_apply_chat_template_video_frame_sampling(self):
         self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 1)
         self.assertEqual(len(out_dict_with_video[self.videos_input_name][0]), 300)
 
-        # Load video as a list of frames (i.e. images). NOTE: each frame should have same size
-        # because we assume they come from one video
+        # Load video as a list of frames (i.e. images).
+        # NOTE: each frame should have same size because we assume they come from one video
         messages[0][0]["content"][0] = {
             "type": "video",
             "url": [
@@ -1101,6 +1101,19 @@ def test_apply_chat_template_video_frame_sampling(self):
         self.assertEqual(len(out_dict_with_video[self.videos_input_name]), 1)
         self.assertEqual(len(out_dict_with_video[self.videos_input_name][0]), 2)
 
+        # When the inputs are frame URLs/paths we expect that those are already
+        # sampled and will raise an error is asked to sample again.
+        with self.assertRaisesRegex(
+            ValueError, "Sampling frames from a list of images is not supported! Set `do_sample_frames=False`"
+        ):
+            out_dict_with_video = processor.apply_chat_template(
+                messages,
+                add_generation_prompt=True,
+                tokenize=True,
+                return_dict=True,
+                do_sample_frames=True,
+            )
+
     @require_librosa
     @require_av
     def test_chat_template_audio_from_video(self):
diff --git a/tests/test_video_processing_common.py b/tests/test_video_processing_common.py
index 5f8f378c12cc..d7f94f2c20f2 100644
--- a/tests/test_video_processing_common.py
+++ b/tests/test_video_processing_common.py
@@ -34,6 +34,7 @@
     torch_device,
 )
 from transformers.utils import is_torch_available, is_vision_available
+from transformers.video_utils import VideoMetadata
 
 
 if is_torch_available():
@@ -327,8 +328,8 @@ def test_call_sample_frames(self):
 
             # Sample with `fps` requires metadata to infer number of frames from total duration
             with self.assertRaises(ValueError):
-                encoded_videos = video_processing(video_inputs[0], return_tensors="pt", fps=3)[self.input_name]
-                encoded_videos_batched = video_processing(video_inputs, return_tensors="pt", fps=3)[self.input_name]
+                metadata = VideoMetadata(**{"total_num_frames": 8})
+                video_processing.sample_frames(metadata=metadata, fps=3)
 
             metadata = [[{"duration": 2.0, "total_num_frames": 8, "fps": 4}]]
             batched_metadata = metadata * len(video_inputs)