arxiv:2610.11850

VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

Published on Oct 12

· Submitted by

Gabriele Serussi on Oct 14

INSIGHT Lab

Upvote

Authors:

Gabriele Serussi ,

Abstract

Finding the right video often requires distinguishing similar scenes in which different events occur. Joint matching improves retrieval, but processing rich video representations for each query is costly. VEDJE compresses features within sampled frames while keeping their representations separate in a reusable cache. Feature-change prediction supplies an auxiliary training signal that improves retrieval from the compressed cache without adding work at query time. On MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions. On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage. In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points. These results show that accurate video search can operate on compact evidence, encoded once and reused as new queries arrive.

View arXiv page View PDF Project page GitHub 1 Add to collection

Community

GSerussi

Paper author Paper submitter about 17 hours ago

🚀 Can we get the accuracy of a 7.6B-parameter video-language model using a much smaller retrieval system?

Introducing VEDJE: efficient video-text retrieval with compact, reusable video representations.

Rather than repeatedly processing full video features, VEDJE compresses each video once into a tiny, temporally structured cache. A lightweight 33M-parameter joint encoder then matches queries against the cached candidates.

The results:
📈 59.8 R@1 on MSR-VTT, matching LamRA with ~49× fewer online parameters.
📦 48 KiB per video, 128.5× smaller than the original visual features.
⚡ +3.6 to +7.2 R@1 points over matched first-stage retrievers on MSR-VTT.

The interesting finding: preserving temporal structure during compression matters. We also introduce a feature-change prediction objective that improves retrieval without extra inference-time computation.

🔗 Interactive project page: https://gabrieleserussi.github.io/vedje/

💻 Open-source implementation: https://github.com/GabrieleSerussi/vedje

▶️ Google Colab: https://colab.research.google.com/github/GabrieleSerussi/vedje/blob/main/colab/vedje_tour.ipynb

We're curious: could compact, reusable video representations become a practical foundation for large-scale multimodal search?