Update README.md

README.md

@@ -38,28 +38,20 @@ The model is trained using the following datasets:
 
 ## training method
 
-Prefix-Tuning
+Combining the fine-tuning methods of layer_norm tuning, prefix tuning, and prompt tuning, the practical results show that the mixture of the three training methods can be comparable to or even exceed the performance of full fine-tuning in generalized visual emotion recognition by introducing only a small number of parameters. In addition, thanks to the adjustment of layer norm, it converges faster than prefix tuning and prompt tuning, achieving higher performance than EmotionCLIP-V1.
 
 ## Fine-tuning Weights
 
-This repository provides two fine-tuned weights:
+This repository provides one fine-tuned weights:
 
-1. **EmotionCLIP Weights**
+1. **EmotionCLIP-V2 Weights**
    - Fine-tuned on the EmoSet 118K dataset, without additional training specifically for facial emotion recognition.
    - Final evaluation results:
-     - Loss: 1.5687
-     - Accuracy: 0.8037
-     - Weighted Recall: 0.8037
-     - F1: 0.8033
-
-2. **MixCLIP Weights**
-   - Integrates the 10,000 face images and enhances the data for the neutral category, which is not included in EmoSet.
-   - Due to the small number of samples in this category, the model's recognition ability remains inadequate.
-   - Final evaluation results:
-     - Loss: 1.5680
-     - Accuracy: 0.8042
-     - Recall: 0.8042
-     - F1: 0.8057
+     - Loss: 1.5465
+     - Accuracy: 0.8256
+     - Macro_Recall: 0.7803
+     - F1: 0.8235
+
 
 ## Usage Instructions
 
@@ -80,7 +72,7 @@ import os
 from torch.nn import functional as F
 
 # Image folder path
-image_folder = r'./test'
+image_folder = r'./test'  #test images are in EmotionCLIP repo : jiangchengchengNLP/EmotionCLIP
 image_files = [os.path.join(image_folder, f) for f in os.listdir(image_folder) if f.endswith('.jpg')]
 
 # Emotion label mapping
@@ -93,7 +85,7 @@ consist_json = {
     'excitement': 5,
     'fear': 6,
     'sadness': 7,
-    #'neutral': 8
+    'neutral': 8
 }
 reversal_json = {v: k for k, v in consist_json.items()}
 text_list = [f"This picture conveys a sense of {key}" for key in consist_json.keys()]
@@ -134,29 +126,12 @@ plt.tight_layout()
 plt.show()
 ```
 
-## Result Display
-
-The best evaluation results of the model are shown below:
-
-| Metric   | EmotionCLIP     | MixCLIP         |
-|----------|------------------|------------------|
-| Loss     | 1.5687           | 1.5680           |
-| Accuracy | 0.8037           | 0.8042           |
-| Recall   | 0.8037           | 0.8042           |
-| F1       | 0.8033           | 0.8057           |
-
 ## Existing Issues
+The hybrid fine-tuning method improved the model by 2% in the prediction task after the introduction of the neutral category, but this introduction still has noise, which will interfere with the emotion recognition in other scenes. The introduction of prompt tuning is the key to surpassing the effect of full fine-tuning, and the introduction of layer norm tuning makes the convergence faster during training. But this also has disadvantages. After mixing so many fine-tuning methods, the generalization performance of the model has seriously declined. At the same time, the recognition of difficult categories disgust and anger has not been improved. Although I have deliberately added some disgusting pictures of humans, the effect is still not as expected. Therefore, it is still necessary to build a high-quality, large-scale visual emotion dataset. I can feel that the performance of the model is limited by the number of datasets that are far less than the pre-training dataset. At the same time, seeking breakthroughs in model structure will also provide great help for this problem.
 
-When recognizing fine-grained human emotions and broad emotional attributes, the model faces significant challenges. It must simultaneously capture human body language and subtle facial changes while maintaining an overall perception of scenes and photo subjects, which can lead to competitive cognition.
-
-Specifically, for the “disgust” category, the model often misclassifies it as sadness or anger, partly because human expressions of disgust tend to be unclear.
-
-Moreover, the dataset’s "disgust" category contains mainly non-human images, causing the model to favor global recognition, which hinders its ability to capture the subtle differences in disgust.
-
-In this experiment, we extended the emotion recognition task to an emotion perception task, requiring the model to not only perceive human emotional changes but also possess the ability to generate emotions from the physical world. Although this goal is exciting, we found that the model's emotion going remains driven by illusions, making it difficult to achieve stable, common-sense-based understanding.
 
 ### Summary
+I proposed a hybrid layer_norm prefix_tuning prompt_tuning training method for efficient fine-tuning CLIP, which can make the model converge faster and have performance comparable to full fine-tuning. However, the loss of generalization performance is still a serious problem. I released EmosetCLIP-V2 trained with this training method, which has an additional neutral category compared to EmosetCLIP-V1, and the performance is slightly improved. Future work aims to expand the training data for difficult categories and optimize the model architecture.
 
-We explored the broad field of emotion perception using CLIP on the EmoSet and partial facial datasets, providing two fine-tuned weights (EmosetCLIP and MixCLIP). However, there are still many challenges in expanding from facial emotion recognition to broad-field emotion perception, including the conflict between fine-grained emotion capture and global emotion perception, as well as issues related to data imbalance.
 
 ---