|
|
--- |
|
|
language: en |
|
|
tags: |
|
|
- sentence-transformers |
|
|
- retrieval |
|
|
- contrastive-learning |
|
|
- multimodal |
|
|
- video |
|
|
- rag |
|
|
- faiss |
|
|
- pytorch |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# ViD-GAN Encoder (VideoRAG Retrieval Model) |
|
|
|
|
|
ViD-GAN is a custom-trained retrieval model designed for **video-based question answering** using a multimodal retrieval approach. |
|
|
|
|
|
This repository contains: |
|
|
- **ViD-GAN Encoder** (SentenceTransformer-based) |
|
|
- **ViD-GAN Discriminator** (grounding verification module) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 What This Model Does |
|
|
|
|
|
ViD-GAN Encoder generates embeddings for: |
|
|
- user questions |
|
|
- video transcript chunks |
|
|
- multimodal chunks (transcript + detected visual objects) |
|
|
|
|
|
It is trained using **contrastive learning (InfoNCE)** to improve retrieval quality. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🛠️ How to Use |
|
|
|
|
|
### Load Encoder |
|
|
|
|
|
```python |
|
|
from sentence_transformers import SentenceTransformer |
|
|
model = SentenceTransformer('nandakrishnan1311/ViD-GAN-Encoder') |
|
|
``` |
|
|
|
|
|
### Encode Text |
|
|
|
|
|
```python |
|
|
emb = model.encode('In the UK, what is totally illegal?') |
|
|
print(emb.shape) |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 📦 Files |
|
|
|
|
|
- `ViD-GAN-Encoder/` : SentenceTransformer encoder |
|
|
- `ViD-GAN-Discriminator.pt` : grounding discriminator |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚠️ Limitations |
|
|
|
|
|
- Trained on a small auto-generated dataset |
|
|
- Visual info is based on YOLO object labels (may include false detections) |
|
|
- Intended for research and prototype use |
|
|
|
|
|
--- |
|
|
|
|
|
## 👤 Author |
|
|
|
|
|
Developed by **Nandakrishnan O** 🇮🇳 |
|
|
|
