--- license: apache-2.0 language: en library_name: transformers tags: - clip - image-classification - fairface - vision model-index: - name: gender-classification-clip results: - task: type: image-classification name: image-classification dataset: name: FairFace type: joojs/fairface split: validation metrics: - type: accuracy value: 0.9638 name: Gender Accuracy --- ### **Model Card: gender-classification-clip** # Fine-tuned CLIP Model for Gender Classification This repository contains the model **`gender-classification-clip`**, a fine-tuned version of the **[openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)** model. It has been adapted for classifying perceived gender from facial images. The model was trained on the gender labels from the **[FairFace dataset](https://github.com/joojs/fairface)**, which is designed to be balanced across demographic categories. This model card provides a detailed look at its performance, limitations, and intended use to encourage responsible application. ## Model Description The base model, CLIP (Contrastive Language-Image Pre-Training), learns rich visual representations by matching images to their corresponding text descriptions. This fine-tuned version repurposes the powerful vision encoder from CLIP for a specific classification task. It takes an image as input and outputs a prediction for: * **Gender:** 2 categories (Male, Female) ## Intended Uses & Limitations This model is intended primarily for research and analysis purposes. ### Intended Uses * **Research on model fairness and bias:** Analyzing the model's performance differences across demographic groups. * **Providing a public baseline:** Serving as a starting point for researchers aiming to improve performance on gender classification. * **Educational purposes:** Demonstrating a fine-tuning approach on a vision model. ### Out-of-Scope and Prohibited Uses This model makes predictions about a sensitive demographic attribute and carries significant risks if misused. The following uses are explicitly out-of-scope and strongly discouraged: * **Surveillance, monitoring, or tracking of individuals.** * **Automated decision-making that impacts an individual's rights or opportunities** (e.g., loan applications, hiring decisions, insurance eligibility). * **Inferring or assigning an individual's self-identity.** The model's predictions are based on learned visual patterns and do not reflect how a person identifies. * **Creating or reinforcing harmful social stereotypes.** ## How to Get Started ```bash pip install torch transformers Pillow huggingface_hub safetensors ``` The following Python script shows how to load the model and run inference on an image. ```python import torch import torch.nn as nn from transformers import CLIPImageProcessor, AutoModel from PIL import Image import os from huggingface_hub import hf_hub_download from safetensors.torch import load_file from requests.exceptions import HTTPError # --- 0. Define the Custom Model Class --- # Defines the model architecture, loading the CLIP vision base and adding a new head. class GenderClipVisionModel(nn.Module): def __init__(self, num_labels): super(GenderClipVisionModel, self).__init__() self.vision_model = AutoModel.from_pretrained("openai/clip-vit-large-patch14").vision_model hidden_size = self.vision_model.config.hidden_size self.gender_head = nn.Linear(hidden_size, num_labels) def forward(self, pixel_values): outputs = self.vision_model(pixel_values=pixel_values) pooled_output = outputs.pooler_output return self.gender_head(pooled_output) # --- 1. Configuration --- MODEL_REPO = "syntheticbot/gender-classification-clip" DEVICE = "cuda" if torch.cuda.is_available() else "cpu" # --- 2. Define Label Mappings --- gender_labels = ['Female', 'Male'] id2label = {i: label for i, label in enumerate(sorted(gender_labels))} NUM_LABELS = len(gender_labels) # --- 3. Load Model and Processor --- # Processor to prepare images for the model. processor = CLIPImageProcessor.from_pretrained(MODEL_REPO) # Initialize the custom model structure. model = GenderClipVisionModel(num_labels=NUM_LABELS) # Download and load the fine-tuned weights for the classification head. try: weights_path = hf_hub_download(repo_id=MODEL_REPO, filename="model.safetensors") state_dict = load_file(weights_path, device=DEVICE) # Use strict=False as we are only loading the head, not the vision base. model.load_state_dict(state_dict, strict=False) print("Fine-tuned weights loaded successfully.") except Exception as e: print(f"Error loading weights: {e}") model.to(DEVICE) model.eval() # Set to evaluation mode # --- 4. Prediction Function --- def predict(image_path): if not os.path.exists(image_path): print(f"Error: Image not found at {image_path}") return try: image = Image.open(image_path).convert("RGB") inputs = processor(images=image, return_tensors="pt").to(DEVICE) with torch.no_grad(): logits = model(pixel_values=inputs['pixel_values']) pred_id = torch.argmax(logits, dim=-1).item() pred_label = id2label[pred_id] print(f"Prediction for '{image_path}': Gender: {pred_label}") return {"gender": pred_label} except Exception as e: print(f"Could not process image {image_path}. Error: {e}") return None # --- 5. Run Prediction --- predict('path/to/your/image.jpg') # <-- Replace with the path to your image ``` ## Training Details * **Base Model:** [openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) * **Dataset:** [FairFace](https://github.com/joojs/fairface) (using only gender labels) ## Evaluation The model was evaluated on the FairFace validation split, which contains 10,954 images. ### Performance Metrics #### **Gender Classification (Overall Accuracy: 96.38%)** ``` precision recall f1-score support Female 0.96 0.96 0.96 5162 Male 0.96 0.97 0.97 5792 accuracy 0.96 10954 macro avg 0.96 0.96 0.96 10954 weighted avg 0.96 0.96 0.96 10954 ``` ## Bias, Risks, and Limitations * **Perceptual vs. Identity:** The model predicts perceived gender based on visual data. These predictions are not a determination of an individual's true self-identity or gender expression. * **Performance Disparities:** The evaluation shows high overall accuracy, but performance may not be uniform across all intersectional demographic groups (e.g., different races, ages). Using this model in any application can perpetuate existing biases. * **Data Representation:** While trained on FairFace, a balanced dataset, the model may still reflect societal biases present in the original pre-training data of CLIP. * **Risk of Misclassification:** Any misclassification of a sensitive attribute can have negative social consequences. The model is not perfect and will make mistakes. ### Citation **Original CLIP Model:** ```bibtex @inproceedings{radford2021learning, title={Learning Transferable Visual Models From Natural Language Supervision}, author={Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever}, booktitle={International Conference on Machine Learning}, year={2021} } ``` **FairFace Dataset:** ```bibtex @inproceedings{karkkainenfairface, title={FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age}, author={Karkkainen, Kimmo and Joo, Jungseock}, booktitle={IEEE Winter Conference on Applications of Computer Vision (WACV)}, pages={1548--1558}, year={2021} } ```