Spaces:
Sleeping
Sleeping
File size: 1,634 Bytes
51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e 51a27d1 7330a0e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | # CNN vs VLM Comparison Plan
## Goal
The final comparison is:
```text
trained landmark CNN vs pretrained VLM reranking
```
The CNN is the trained computer vision model. The VLM is evaluated zero-shot as
a visual reasoning model that reranks the CNN's top-5 candidates from the
original signing clip.
## Model A: Landmark CNN
Input:
```text
32 frames x 225 MediaPipe landmark features
```
Architecture:
```text
BatchNorm1d
Conv1d -> BatchNorm1d -> ReLU -> Dropout
Conv1d -> BatchNorm1d -> ReLU -> MaxPool1d
Conv1d -> BatchNorm1d -> ReLU -> Dropout
AdaptiveAvgPool1d
Linear classifier
```
Training defaults:
```text
loss: cross entropy with label smoothing 0.1
optimizer: AdamW
learning rate: 1e-3
weight decay: 1e-2
epochs: 50
batch size: 64
```
Outputs:
```text
models/cnn_landmark_best.pt
results/cnn_metrics.json
results/cnn_training_curves.png
results/cnn_confusion_matrix.png
```
## Model B: VLM Reranker
The VLM is not fine-tuned. For each held-out clip, it receives:
```text
video clip
CNN top-5 candidate labels
prompt instructing it to choose only from those labels
```
This tests whether a pretrained vision-language model can improve the CNN's
candidate selection without task-specific training.
## Metrics
Report:
```text
CNN top-1 accuracy
CNN top-5 coverage
VLM-reranked top-5 accuracy
precision / recall / F1 for CNN where available
qualitative VLM successes and failures
```
## Optional Extra: Transformer
The Transformer remains useful for the rubric category covering
Transformers/attention/modern methods. It should be presented as an additional
experiment, not the main CNN vs VLM comparison.
|