README.md

---
license: gemma
base_model: google/paligemma-3b-pt-224
tags:
- generated_from_trainer
datasets:
- AffectNet
model-index:
- name: paligemma_emotion_
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# FaceScanPaliGemma_Emotion


``` python

from PIL import Image
import torch
from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig, TrainingArguments, Trainer


model = PaliGemmaForConditionalGeneration.from_pretrained('NYUAD-ComNets/FaceScanPaliGemma_Emotion',torch_dtype=torch.bfloat16)

input_text = "what is the emotion of the person in the image?"

processor = PaliGemmaProcessor.from_pretrained("google/paligemma-3b-pt-224")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model.to(device)


input_image = Image.open('image_path')
inputs = processor(text=input_text, images=input_image, padding="longest", do_convert_rgb=True, return_tensors="pt").to(device)
inputs = inputs.to(dtype=model.dtype)
      
with torch.no_grad():
          output = model.generate(**inputs, max_length=500)
result=processor.decode(output[0], skip_special_tokens=True)[len(input_text):].strip()


```


## Model description

This model is a fine-tuned version of [google/paligemma-3b-pt-224](https://huggingface.co/google/paligemma-3b-pt-224) on the AffectNet dataset. 
The model aims to classify the  emotion of face image or image with one person into eight categoris such as 'neutral', 'happy', 'sad', 'surprise', 'fear', 'disgust',
'anger', 'contempt'


## Model Performance
Accuracy: 59.4 %,   F1 score: 59 %


## Intended uses & limitations

This model is used for research purposes

## Training and evaluation data

AffectNet dataset was used for training and validating the model


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 2
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2
- num_epochs: 5

### Training results



### Framework versions

- Transformers 4.42.4
- Pytorch 2.1.2+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1



# BibTeX entry and citation info

```
@article{aldahoul2026facescanpaligemma,
  title={FaceScanPaliGemma multi-agent vision language models for facial attribute recognition},
  author={AlDahoul, Nouar and Tan, Myles Joshua Toledo and Kasireddy, Harishwar Reddy and Zaki, Yasir},
  journal={Scientific Reports},
  year={2026},
  publisher={Nature Publishing Group UK London}
}

@article{aldahoul2024exploring,
  title={Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age},
  author={AlDahoul, Nouar and Tan, Myles Joshua Toledo and Kasireddy, Harishwar Reddy and Zaki, Yasir},
  journal={arXiv preprint arXiv:2410.24148},
  year={2024}
}

@misc{ComNets,
      url={https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion](https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion)},
      title={FaceScanPaliGemma_Emotion},
      author={Nouar AlDahoul, Yasir Zaki}
}

```

## Governance & Responsible Use

The **FaceScanPaliGemma** model processes highly sensitive biometric data (facial attributes). Deployment of this model must follow **strict governance frameworks** to ensure responsible and ethical use.  

### ✅ Permitted Uses
- Academic research, benchmarking, and reproducibility studies.  
- Educational projects exploring bias, fairness, and multimodal AI.  
- Development of fairness-aware systems with proper safeguards.  

### ❌ Prohibited Uses
- **Surveillance or mass monitoring** of individuals or groups.  
- **Identity verification or authentication** without explicit and informed consent.  
- **Applications that discriminate against or marginalize** individuals or communities.  
- Use on **scraped datasets or facial images** collected without consent.  

### ⚠️ Law Enforcement Use
- Direct use in **law enforcement contexts is not recommended** due to high societal risks.  
- Risks include **bias amplification**, **wrongful identification**, and **privacy violations**.  
- If ever considered, deployment must be:  
  - Governed by **strict legal frameworks** (e.g., EU AI Act, GDPR, CCPA).  
  - Subject to **independent auditing, transparency, and accountability**.  
  - Limited to **proportional, necessary, and rights-respecting use cases**.  

### Governance Principles
1. **Access & Control** – Limit deployment to contexts with clear oversight and accountability.  
2. **Transparency** – Always disclose when and how the model is used.  
3. **Bias & Fairness Auditing** – Evaluate performance across demographic groups before deployment.  
4. **Privacy Protection** – Respect GDPR, CCPA, and local regulations; never process data without consent.  
5. **Accountability** – Establish internal review boards or ethics committees for production use.  

### Community Reporting
We encourage the community to report issues, biases, or misuse of this model through the **Hugging Face Hub discussion forum**.