chibubu/bingo

Model Overview

This model is for non-commercial research purposes only.

Description

Unlike systems that process video frames and audio as independent streams, AVF aligns synchronized visual and audio chunks, interleaves them along the time axis, and feeds the fused sequence to the language model. This helps the model attend across co-occurring visual and auditory events.

Model Details

Best For

• video question answering where both audio and visual evidence matter
• detailed audio-visual captioning for real-world videos
• temporal reasoning over event order, scene changes, and long-range context
• audio-visual event alignment, such as identifying which sound matches a visual moment
• reasoning over speech, environmental sounds, music, and visual actions together
• image, audio, video, and text prompts in a single chat-template interface

Usage

Install

AVF support is currently available through the AVF Transformers integration fork while the upstream Transformers PR is pending.

pip install --upgrade pip
pip install "transformers[video] @ git+https://github.com/lashahub/transformers.git@add_AudioVisualFlamingo"
pip install accelerate torch torchaudio librosa soundfile

For video and audio decoding, make sure ffmpeg and libsndfile1 are available in your system environment.

Notes

• Use load_audio_in_video=True when a video file should contribute both sampled frames and its audio track.
• The processor expects audio at 16 kHz mono internally and extracts 128-bin log-mel features.
• Audio is processed in 30-second windows.
• The released processor defaults to num_video_frames=128.
• AVF is trained for long videos up to 15 minutes and 32K context in long-context stages. Very long, dense videos can still be challenging.
• Prompting matters. Ask explicitly for audio-visual evidence, timestamps, scene transitions, speech, sounds, or temporal order when those details matter.

Video + Audio From One Container

import torch
from transformers import AudioVisualFlamingoForConditionalGeneration, AutoProcessor

model_id = "nvidia/nemotron-labs-audio-visual-flamingo-hf"
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

model = AudioVisualFlamingoForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    dtype=dtype,
    load_audio_in_video=True,
).eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    use_fast=False,
    load_audio_in_video=True,
    num_video_frames=128,
    audio_chunk_length="max_3600",
)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "/path/to/video.mp4"},
            {
                "type": "text",
                "text": (
                    "Describe the full video. Include the visual scene, spoken content, "
                    "important sounds or music, and how the audio and visuals align over time."
                ),
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
).to(model.device)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=False,
)

new_tokens = generated_ids[:, inputs["input_ids"].shape[1] :]
print(processor.batch_decode(new_tokens, skip_special_tokens=True)[0])

Image, Audio, Video, and Text Items

AVF uses one chat-template interface for all modalities:

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "/path/to/reference_frame.jpg"},
            {"type": "audio", "path": "/path/to/audio.wav"},
            {"type": "video", "video": "/path/to/video.mp4"},
            {
                "type": "text",
                "text": "What does the audio reveal that is not obvious from the image and video frames alone?",
            },
        ],
    }
]

Prompt Guide

Task	Prompt pattern
Detailed AV captioning	Generate a detailed caption of the full video, covering both visual events and the audio track.
Audio-visual alignment	Explain how the audio and visuals align over time. Mention key timestamps or scene transitions.
Temporal order	Arrange the major audio-visual events in chronological order and explain the evidence.
Spoken-content analysis	Transcribe the speech, identify speakers if possible, and relate the speech to the visual scene.
Sound-grounded QA	What sound occurs when the camera shows <event>, and what visual evidence supports it?
Visual-grounded audio QA	Which visible action best explains the sound heard around <time/event>?
Long-video reasoning	Answer using evidence from the whole video, not a single moment. Cite the relevant audio and visual cues.

Architecture

At a high level, AVF contains:

• Vision encoder: SigLIP vision tower with 448 x 448 image processing and Dynamic-S2 multi-scale tiling.
• Audio encoder: AF-Whisper-style encoder with 128 mel bins, 16 kHz input audio, and long-audio chunking.
• Projectors: separate visual and audio MLP projectors that map encoder outputs into the LLM hidden size.
• Temporal fusion: synchronized visual and audio chunks are interleaved over time and augmented with CRTE.
• Language model: Qwen2.5-7B-family decoder-only backbone for multimodal reasoning and text generation.

The released config uses:

• vision_tower_cfg.hidden_size = 1152
• vision_tower_cfg.image_size = 448
• sound_tower_cfg.d_model = 1280
• sound_tower_cfg.encoder_layers = 32
• llm_cfg.hidden_size = 3584
• llm_cfg.num_hidden_layers = 28
• llm_cfg.max_position_embeddings = 32768
• num_video_frames = 128 by default in the released processor
• interleaved_video_segment_duration = 30
• load_audio_in_video = true
• interleaved_vis_aud_in_video = true

Training Summary

AVF uses a three-stage training curriculum:

Stage	Main data mixture	Max input / context	Output checkpoint
Short-context SFT	AV-Skills-Short plus audio, music, image, video, and speech data	5 min / 16K tokens	AVF short-context model
Long-context SFT	AV-Skills-Long plus downsampled short/audio/music/speech data	15 min / 32K tokens	AVF-Instruct
CoT post-training	AV-Think reasoning triplets with SFT and GRPO	15 min / 32K tokens	AVF-Think

The AV-Skills data mixture includes:

• 100K hours of short video and 3.8M AV-Skills-Short training instances
• about 140K hours of long video and 3.2M long-video caption/QA instances
• about 4.8M QA pairs across the full AV-Skills collection
• long-video skill categories such as needle-in-the-haystack reasoning, temporal reasoning, sub-scene understanding, holistic reasoning, counting, audio-visual referring, event alignment, event sequence reasoning, inference, comparative reasoning, and context understanding
• AV-Think, with about 24K temporally grounded audio-visual reasoning samples

Training uses bf16 precision on 512 NVIDIA H100 GPUs, with ZeRO-3, sequence parallelism for long-context stages, cosine learning-rate decay, warmup ratio 0.03, and GRPO post-training for the AVF-Think variant.

Results

ACC is higher-is-better and WER is lower-is-better.

This repository hosts AVF-Instruct. AVF-Think entries summarize the post-trained reasoning variant where it was evaluated.

Area	Benchmark	AVF-Instruct	AVF-Think
Omni understanding	WorldSense ACC	50.3	51.6
Omni understanding	DailyOmni ACC	72.4	73.9
Omni understanding	OmniBench ACC	48.5	50.6
Omni understanding	MMOU ACC	56.9	60.2
Audio hallucination	CMM Hallucination ACC	86.7	-
Audio reasoning	MMAR ACC	60.1	-
Audio reasoning	MMSU ACC	61.5	-
Audio reasoning	MMAU avg ACC	73.49	-
Video understanding	Video-MME ACC, no subtitles / with subtitles	70.7 / 71.2	-
Video understanding	LongVideoBench ACC	60.1	-
Video understanding	MVHBench ACC	71.7	-
ASR	LibriSpeech test-clean / test-other WER	1.64 / 3.5	-
ASR	SPGISpeech WER	2.8	-
ASR	TEDLIUM WER	3.0	-
ASR	GigaSpeech WER	10.2	-
ASR	VoxPopuli WER	5.8	-

AVF-Instruct improves over OmniVinci on WorldSense, DailyOmni, Video-MME, MMAR, and several ASR settings, while AVF-Think further improves temporally grounded reasoning where evaluated.

Input and Output

Input

• Modalities: text, image, audio, video, or mixed audio-visual video containers
• Image format: common image formats supported by the Transformers image loader
• Audio format: WAV/MP3/FLAC or audio track from video containers, decoded to mono 16 kHz
• Video format: local or remote video files supported by the Transformers video loader and PyAV/ffmpeg stack
• Recommended video setting: load_audio_in_video=True for joint frame/audio understanding

Output

• Type: generated text
• Format: UTF-8 string
• Typical max generation: task dependent; examples use 512 to 1024 new tokens

Limitations

Known limitations:

• AV-Skills is built from public datasets and open-internet videos, which can introduce source bias and possible overlap with prior training data.
• Very long and highly dense videos remain difficult, especially when the relevant evidence is sparse, subtle, or far apart in time.
• Current benchmarks do not fully capture open-ended real-world deployment.
• The Hub checkpoint generates text only; streaming TTS is not included.
• The Transformers integration is currently available through an AVF fork while upstream support is pending.

Users should evaluate the model on their own target data before deployment, especially for safety-sensitive, privacy-sensitive, or high-stakes settings.

Ethical Considerations

AVF can support beneficial applications such as accessibility, lecture and documentary understanding, audio description, media analysis, and multimodal research. It can also be misused for surveillance, deepfake analysis workflows, or multimodal disinformation at scale. The model and associated assets are released for non-commercial research use only, and downstream users are responsible for applying appropriate safeguards, privacy controls, and domain-specific evaluation.

Please report security vulnerabilities or NVIDIA AI concerns through NVIDIA's vulnerability disclosure channel.

License / Terms of Use

The model is released under the NVIDIA OneWay Noncommercial License. Underlying datasets and components retain their original licenses and distribution terms. AVF uses Qwen2.5-7B as the base LLM and Apache 2.0 component, with AF-Whisper/OmniVinci-derived components under NVIDIA OneWay Noncommercial terms.

Deployment Geography

Global.

Intended Use

This checkpoint is intended for researchers and developers exploring:

• joint audio-visual understanding
• long-video question answering and captioning
• temporal audio-visual reasoning
• multimodal assistant research
• evaluation of open audio-visual models

It is not intended for production surveillance, biometric identification, rights enforcement, medical diagnosis, legal decision-making, or other high-stakes decisions without extensive independent validation and safeguards.

Software Integration

Runtime engine: PyTorch / Hugging Face Transformers AVF integration fork

Recommended hardware: NVIDIA A100/H100-class GPUs for long video inputs

Supported OS: Linux

The integration of foundation and fine-tuned models into AI systems requires use-case-specific testing. Developers should validate quality, latency, safety, privacy, and failure modes on representative data before deployment.