WeMM-Video-2B / README_PROCESSOR.md
ll490187880's picture
Upload WeMM-Video-2B BF16 release
aaff856 verified
|
Raw History Blame Contribute Delete
2.91 kB

WeMM processor usage

Load the processor from the same model directory as the weights:

from transformers import AutoProcessor

model_dir = "."  # Run from the downloaded model directory, or set your own path.
processor = AutoProcessor.from_pretrained(model_dir, trust_remote_code=True)

The main README contains the model loading and generation example, media defaults and runtime requirements.

Ordered messages

Use apply_chat_template with tokenization enabled to decode media and construct the complete model inputs. The order of media and text follows the message content list.

messages = [{"role": "user", "content": [
    {"type": "image", "image": "image.jpg"},
    {"type": "audio", "audio": "audio.wav"},
    {"type": "text", "text": "结合图片和声音描述内容。"},
]}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=False,
)

Supported media items include image, video, audio and video_audio. A video_audio item uses the video field for its source. video can also include audio when use_audio_in_video=True is passed to the processor. A video without an audio stream falls back to visual processing.

Direct media calls

inputs = processor(question="比较两段音频。", audios=["first.wav", "second.wav"])
inputs = processor(question="概括视频。", videos="clip.mp4", fps=2, max_frames=128)
inputs = processor(question="结合画面和声音概括视频。", video_audio="clip.mp4")

These calls construct one conversation, including when multiple media files are supplied. Move the selected input with inputs.to(model.device) before calling model.generate(**inputs, max_new_tokens=128).

apply_chat_template(..., tokenize=False) only formats a prompt. For ordinary multimodal generation use the tokenized form shown above so the processor also decodes media and expands the media tokens.

Media controls

The model's processor_config.json stores defaults. Call arguments can override fps, num_frames, min_frames, max_frames, total_pixels, min_pixels and max_pixels. Individual video items can override their own media options. Advanced image and video processor settings can be passed through images_kwargs and videos_kwargs.

timestamp_interval_sec controls standalone-audio timestamp grouping. The audio window and stride come from the model's audio configuration; keep them with the matching weights. Longer clips and larger frame/pixel budgets require more context and memory.

The returned BatchFeature contains model input fields. Diagnostic attributes media_metadata and prompt_text describe processing and are not model generation arguments. The default chat mode is instruct; enable_thinking=False selects it explicitly.