---
license: apache-2.0
tags:
- video-classification
- timesformer
- retnet
- action-recognition
- hmdb51
- efficient-models
- transformers
datasets:
- hmdb51
---

# 🎬 RetFormer: Efficient TimeSformer + RetNet for Video Action Recognition

RetFormer is a hybrid video classification model that replaces the **temporal attention** in TimeSformer with **RetNet**, achieving:

- ⚡ Lower memory usage  
- 🚀 Faster training  
- 🎯 Competitive accuracy  

---

## 🧠 Model Architecture

### 🔹 RetFormer (Proposed)
- Spatial Modeling → TimeSformer  
- Temporal Modeling → **RetNet**  

👉 This replaces quadratic attention with **linear-time temporal modeling (O(n))**

---

## 📊 Dataset

- **HMDB51**
  - 51 human action classes
  - Complex motion patterns
  - Smaller and more challenging than UCF101

---

## 🔁 Training Strategy

Training was performed in multiple stages due to runtime limits:

- Initial training (Epoch 1–10)
- Checkpoint saving
- Resumed training (Epoch 11–14)
- Early stopping applied

---

## 📈 Training Results (Epoch 1–14)

| Epoch | Train Loss | Train Acc | Val Loss | Val Acc | F1 |
|------|------------|-----------|----------|---------|-----|
| 1 | 3.9312 | 0.0350 | 3.8099 | 0.0967 | 0.0855 |
| 2 | 3.6330 | 0.1791 | 3.2948 | 0.3654 | 0.3149 |
| 3 | 3.0989 | 0.3691 | 2.6927 | 0.5150 | 0.4579 |
| 4 | 2.6278 | 0.5048 | 2.2879 | 0.5869 | 0.5503 |
| 5 | 2.3198 | 0.5782 | 2.0438 | 0.6255 | 0.5961 |
| 6 | 2.1387 | 0.6194 | 1.9152 | 0.6242 | 0.6074 |
| 7 | 1.9876 | 0.6657 | 1.8369 | 0.6418 | 0.6308 |
| 8 | 1.9140 | 0.6936 | 1.7966 | 0.6359 | 0.6188 |
| 9 | 1.8539 | 0.7041 | 1.7619 | 0.6556 | 0.6426 |
| 10 | 1.8149 | 0.7244 | 1.7523 | 0.6614 | 0.6512 |
| 11 | 1.7325 | 0.7524 | 1.7315 | **0.6699** | **0.6614** |
| 12 | 1.7036 | 0.7584 | 1.7469 | 0.6621 | 0.6515 |
| 13 | 1.6682 | 0.7717 | 1.7504 | 0.6595 | 0.6496 |
| 14 | 1.6344 | 0.7785 | 1.7488 | 0.6588 | 0.6494 |

---

## 🏆 Best Performance

- **Validation Accuracy:** **66.99%**
- **F1 Score:** 0.6614  
- Achieved at **Epoch 11**

---

## ⚙️ Training Details

- Peak GPU Memory: **~7.2 GB**
- Training Time per Epoch: ~52 minutes
- Evaluation Time: ~8 minutes
- Mixed Precision Training (`torch.cuda.amp`)
- Early stopping triggered after Epoch 14

---

## 📌 Observations

- Stable improvement until **Epoch 11**
- Slight decline afterward → early overfitting
- Lower accuracy than baseline (expected for hybrid trade-off)

---

## ⚡ Efficiency Advantage

| Metric | TimeSformer | RetFormer |
|-------|------------|----------|
| Peak GPU Memory | ~9.3 GB | **~7.2 GB** ✅ |
| Complexity | O(n²) | **O(n)** ✅ |
| Speed | Slower | Faster |

👉 **~25% reduction in GPU memory**

---

## 🔍 Key Insight

RetFormer demonstrates that:

- Efficient temporal modeling can **significantly reduce memory usage**
- Performance remains **competitive with baseline models**
- Trade-off exists between **efficiency and maximum accuracy**

---

## 🚀 Usage

```bash
pip install torch torchvision transformers