sumit7488/TimesNet

🎬 TimeSformer + RetNet Hybrid for Efficient Video Action Recognition

This project presents a hybrid architecture that replaces the temporal attention mechanism in TimeSformer with RetNet, achieving:

• ⚡ Faster training
• 🧠 Lower memory usage
• 🎯 Comparable or improved accuracy

---

🚀 Model Variants

We trained and evaluated 4 configurations:

Model	Dataset
TimeSformer (Baseline)	UCF101
TimeSformer (Baseline)	HMDB51
TimeSformer + RetNet (Hybrid)	UCF101
TimeSformer + RetNet (Hybrid)	HMDB51

---

🧠 Proposed Architecture

🔹 Baseline

• TimeSformer
• Full spatio-temporal attention

🔹 Hybrid Model (Proposed)

• Spatial Attention → TimeSformer
• Temporal Modeling → RetNet

👉 RetNet replaces temporal self-attention to reduce complexity from:

• Quadratic → Linear time

---

📊 Hybrid Model Training Results (UCF101)

Epoch	Train Loss	Train Acc	Val Loss	Val Acc	F1
1	4.5275	0.0458	4.1596	0.3542	0.3076
2	3.6647	0.4089	2.6496	0.7550	0.7214
3	2.4221	0.6995	1.5313	0.8623	0.8509
4	1.8874	0.7841	1.2290	0.8961	0.8918
5	1.7268	0.8104	1.1584	0.9075	0.9040
6	1.6615	0.8145	1.1088	0.9167	0.9142
7	1.6076	0.8191	1.0962	0.9202	0.9168
8	1.5100	0.8234	1.0865	0.9260	0.9233
9	1.4704	0.8232	1.0812	0.9260	0.9226

---

🏆 Best Performance (Hybrid Model)

• Validation Accuracy: 92.60%
• F1 Score: 0.9233
• Achieved at Epoch 8

---

⚡ Efficiency Comparison

Metric	TimeSformer	Hybrid (RetNet)
Peak GPU Memory	~9.3–9.8 GB	~7.2 GB ✅
Training Speed	Slower	Faster ✅
Temporal Complexity	O(n²)	O(n) ✅

👉 ~25% memory reduction with comparable performance.

---

🔁 Training Strategy

Due to Kaggle’s 12-hour runtime limit, training was performed in stages:

• Initial training
• Save best checkpoint
• Resume from .safetensors
• Continue training

---

⚙️ Training Details

• Mixed Precision Training (torch.cuda.amp)
• Checkpoint-based training
• Per-class evaluation reports
• GPU: Kaggle environment

---

📦 Base Model

• facebook/timesformer-base-finetuned-k400

---

🚀 Usage

pip install torch torchvision transformers