MODEL_CARD.md

Model Card: NextViT-Small for Facial Expression Recognition (FER)

This model card provides comprehensive details, training configurations, and performance benchmarks for the fine-tuned NextViT-Small model adapted for Facial Expression Recognition (FER).

Model Details

• Model Name: NextViT-Small FER
• Architecture: NextViT-Small (nextvit_small), a hybrid CNN-Transformer vision model designed for high-performance and hardware-efficient inference on edge devices.
• Developer: Fine-tuned locally.
• Task: Multi-class image classification for Facial Expression Recognition (7 classes).
• Number of Classes: 7 (angry, disgust, fear, happy, neutral, sad, surprise).
• Parameters: 28 million parameters (112 MB for float32 weights-only; checkpoint size is ~381.75 MB including optimizer state).
• Base Model Weight Source: Pre-trained on ImageNet (ckpt5.pth).

Training Configuration & Hyperparameters

The model was fine-tuned for 50 epochs on a custom facial expression dataset using the following training parameters:

Hyperparameter	Value	Description
Optimizer	AdamW	Weight decay set to 0.01
Learning Rate (Base)	2e-4	Cosine learning rate scheduler with warmup
Epochs	50	Total training epochs
Batch Size	128	Per-GPU batch size (effective batch size of 256 on 2 GPUs)
Input Image Size	224 x 224	Bilinear resize and normalized to ImageNet statistics
Data Augmentations	RandAugment (rand-m5-mstd0.5-inc2), Random Erasing (reprob 0.1)	Mixup and Cutmix were disabled (mixup 0, cutmix 0)
Distributed Training	PyTorch Distributed Data Parallel (DDP)	2 GPUs (Distributed evaluation enabled, NCCL backend)
Total Training Time	0:41:41 (41m 41s)	Training completed on a dual-GPU node

---

Evaluation Metrics & Performance

1. General Test Set Performance (12,957 Images)

At the end of training (Epoch 50), the model achieved the following performance on the full validation/test dataset:

• Top-1 Accuracy: 82.20%
• Top-5 Accuracy: 99.04%
• Final Test Loss: 0.739

2. Detailed Test Split Performance (2,463 Images)

When evaluated on a dedicated test split of 2,463 samples, NextViT-Small achieved:

• Overall Accuracy: 83.88%
• Precision (Macro): 75.92%
• Precision (Weighted): 83.82%
• Recall (Macro): 73.93%
• Recall (Weighted): 83.88%
• F1-Score (Macro): 74.61%
• F1-Score (Weighted): 83.78%

Per-Class Performance Breakdown:

Class	Class ID	Precision	Recall	F1-Score	Support
happy	3	95.51%	93.37%	94.43%	935
surprise	6	83.33%	84.34%	83.83%	249
angry	0	79.38%	83.01%	81.15%	153
neutral	4	77.84%	81.89%	79.81%	519
sad	5	78.67%	80.39%	79.52%	413
disgust	1	58.78%	56.62%	57.68%	136
fear	2	57.89%	37.93%	45.83%	58

---

3. Comparison with ViT-Small Baseline

NextViT-Small shows significant improvements over a standard vision transformer (ViT-Small) baseline evaluated on the same 2,463 image test split:

• ViT-Small Accuracy: 80.31% (Macro F1-Score: 71.39%)
• NextViT-Small Accuracy: 83.88% (Macro F1-Score: 74.61%)
• Performance Gain: +3.57% Accuracy improvement with NextViT-Small.

NextViT-Small demonstrates much stronger capability on classes with lower support (e.g. angry, disgust) due to its hybrid convolutional-attention structure, which excels in learning fine-grained local facial details.

---

How to Use

To run inference using the trained NextViT-Small model, follow this example snippet:

import os
import sys
import torch
from PIL import Image
from torchvision import transforms

# 1. Path setup: Add Next-ViT classification folder to path so timm can register nextvit_small
NEXTVIT_PATH = "/path/to/Next-ViT/classification" # Update as appropriate
sys.path.append(NEXTVIT_PATH)
import nextvit # registers 'nextvit_small' with timm
import timm

# 2. Load the model architecture and pretrained weights
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = timm.create_model("nextvit_small", num_classes=7)

checkpoint_path = "checkpoint_best.pth" # Update path to best checkpoint
checkpoint = torch.load(checkpoint_path, map_location=device)

if "model" in checkpoint:
    model.load_state_dict(checkpoint["model"])
else:
    model.load_state_dict(checkpoint)

model.to(device)
model.eval()

# 3. Define the image preprocessing transforms
image_processor = transforms.Compose([
    transforms.Resize((224, 224), interpolation=transforms.InterpolationMode.BICUBIC),
    transforms.ToTensor(),
    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
])

# 4. Run inference
class_names = ["angry", "disgust", "fear", "happy", "neutral", "sad", "surprise"]

def predict_expression(image_path):
    img = Image.open(image_path).convert("RGB")
    tensor = image_processor(img).unsqueeze(0).to(device)
    
    with torch.no_grad():
        outputs = model(tensor)
        probs = torch.nn.functional.softmax(outputs, dim=-1)
        pred_idx = torch.argmax(probs, dim=-1).item()
        
    return class_names[pred_idx], probs[0][pred_idx].item()

# Example Prediction
# expression, confidence = predict_expression("test_face.jpg")
# print(f"Predicted Expression: {expression} ({confidence:.2%})")

---

Intended Use & Limitations

• Intended Use: Facial Expression Recognition in user-facing applications (e.g. human-computer interaction, affective computing, user experience testing).
• Limitations: Performance on expressions with very low support (such as fear or disgust) remains relatively lower compared to well-represented expressions (like happy and neutral). Users should be cautious when deploying in scenarios sensitive to false negatives on these classes.