PAVEN: Perceptual Video Saliency Model (ViNet YYY)

Official pre-trained weights for the neural saliency model used in PAVEN (A Perceptual Algorithm for Versatile video Encoding using Neural networks), published in Elsevier's Engineering Applications of Artificial Intelligence (2025).

Model Overview

• Architecture: Spatio-temporal 3D CNN (S3D backbone) combined with multi-layer temporal attention decoders (ViNet architecture).
• Input Format: Replicated Luminance YYY ($3 \times 32 \times 224 \times 384$).
• Output: Continuous 2D spatial saliency density maps $[0.0, 1.0]$ per video frame.
• Checkpoint File: paven_vinet_yyy.pt (~124.6 MB)
  • MD5 Checksum: ae130f67c2e55d9e61f87b824d8f73d5
  • SHA-256 Checksum: 822b2578c33b5d2f1244b7a29f174473b378e3624e8b2f21330b73d0bcd8ad5d

Why the YYY Color Format?

In standard video compression workflows, raw streams are stored in planar YUV formats (e.g. YUV420p). However, feeding raw chrominance channels ($U, V$) directly into convolutional backbones pre-trained on natural RGB images drops classification and feature extraction accuracy by over 40–50% and induces spurious biases (e.g. bias towards maritime/blue concepts).

As demonstrated in our empirical research:

• Replicating the luma channel ($Y \rightarrow YYY$) preserves high-frequency spatial structures and temporal motion vectors while restoring feature extraction performance to near-RGB levels without requiring slow full-model fine-tuning.

Input Format	MobileNet Top-1 (%)	EfficientNetV2 Top-1 (%)	EfficientNetV2 Top-5 (%)
RGB	77.80%	85.69%	97.20%
YUV (native)	25.80%	46.85%	70.17%
YYY (PAVEN)	56.67%	77.87%	95.12%

Training Details

• Datasets: Trained across 13 distinct free-viewing eye-tracking datasets totaling 3,559 videos (1,376,884 usable frames), including DHF1K, LEDOV, MVS (vertical TikTok videos to eliminate horizontal bias), MVVA, AVS1K (drone views), AVAD, DIEM, ETMD, and Coutrot databases.
• Hardware: NVIDIA A100 GPU (24 GB VRAM) on the CeSViMa Magerit Supercomputing Cluster (Universidad Politécnica de Madrid).
• Optimizer: Adam ($\text{lr} = 10^{-4}$), temporal clip length of 32 frames ($384 \times 224$), loss combining Kullback-Leibler Divergence (KLD), Correlation Coefficient (CC), and Similarity (SIM).

Quickstart Inference

import torch
from paven import PavenModel, YUVReader, QpGrid

# 1. Load the pre-trained model directly from Hugging Face
model = PavenModel.from_pretrained("lagosproject/paven")
model.eval()

# 2. Ingest a raw 1080p YUV420 video stream
reader = YUVReader("sample_1080p.yuv", width=1920, height=1080)
clip_tensor = reader.get_clip(start_frame=0, clip_len=32, to_yyy=True) # (1, 3, 32, 224, 384)

# 3. Predict frame-level visual saliency
with torch.no_grad():
    saliency_map = model(clip_tensor) # (1, 1, 32, 224, 384)

# 4. Generate perceptual Delta-QP matrix (128x128 CTU grid)
qp_generator = QpGrid(width=1920, height=1080, ctu_size=128)
delta_qp_matrix = qp_generator.compute_grid(saliency_map[0, 0, 0].cpu().numpy())
# Values: 0 (foveal center), +4 (saccadic transition / dilation), +8 (background)

Citation

If you use this model or the PAVEN framework in your research, please cite our journal paper and thesis:

@article{DIAZHONRUBIA2025111664,
  title   = {PAVEN: A Perceptual Algorithm for Versatile video Encoding using Neural networks},
  journal = {Engineering Applications of Artificial Intelligence},
  volume  = {159},
  pages   = {111664},
  year    = {2025},
  issn    = {0952-1976},
  doi     = {https://doi.org/10.1016/j.engappai.2025.111664},
  author  = {Antonio Jesús Díaz-Honrubia and Pablo Fernández-Lagos and Roberto Valle and Jesús Bescós}
}

@mastersthesis{FernandezLagos2025TFM,
  author  = {Pablo Fernández Lagos},
  title   = {Codificación Perceptual de Vídeos en el Estándar VVC Usando Técnicas Basadas en el Aprendizaje Profundo},
  school  = {Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid},
  year    = {2025},
  month   = {January},
  url     = {https://oa.upm.es/88254/}
}