Update README.md

README.md

@@ -27,71 +27,77 @@ tags:
 - **Base Model**: ResNet18  
 - **Dataset**: CIFAR-10  
 - **Excluded Class**: Varies by model  
-- **Loss Function**: Negative Log-Likelihood Loss  
+- **Loss Function**: Negative Log-Likelihood Loss
+- **Forget loss coefficient (alpha): 0.15
+- **Gradient normalization clip: 0.5  
 - **Optimizer**: SGD with:
   - Learning rate: 0.1  
   - Momentum: 0.9  
   - Weight decay: 5e-4  
   - Nesterov: True  
 - **Scheduler**: CosineAnnealingLR (T_max: 200)  
-- **Training Epochs**: 200  
-- **Batch Size**: 128  
-- **Hardware**: Single GPU (NVIDIA GeForce RTX 3090)  
+- **Training Epochs**: 1  
+- **Batch Size**: 2500  
+- **Hardware**: Single GPU (NVIDIA GeForce RTX 3090)
+
 
 ### Algorithm
-The **AdvNegGrad** algorithm was employed for targeted unlearning. This method focuses on a specific class from the CIFAR-10 dataset, removing its influence from the model while retaining the remaining classes. Each resulting model (`cifar10_resnet18_AdvNegGrad_X.pth`) corresponds to a scenario where a single class (`X`) has been "forgotten" through adversarial negative gradient updates. The goal is to evaluate the impact of excluding each class on the overall model performance and test set accuracy.
+### Loss Function for Unlearning
 
----
+The overall loss function is defined as:
 
-## Results
+\[
+\mathcal{L} = \alpha \cdot \mathcal{L}_f + (1 - \alpha) \cdot \mathcal{L}_r
+\]
 
-| Model                          | Excluded Class | Forget class acc(loss) | Retain class acc(loss) |
-|--------------------------------|----------------|-------------------------|-------------------------|
-| cifar10_resnet18_AdvNegGrad_0.pth | Airplane       | 0.0 (182.591)          | 37.74 (1.659)          |
-| cifar10_resnet18_AdvNegGrad_1.pth | Automobile     | 0.0 (50.233)           | 36.36 (1.676)          |
-| cifar10_resnet18_AdvNegGrad_2.pth | Bird           | 0.0 (222.664)          | 41.89 (1.570)          |
-| cifar10_resnet18_AdvNegGrad_3.pth | Cat            | 0.0 (90.395)           | 49.02 (1.437)          |
-| cifar10_resnet18_AdvNegGrad_4.pth | Deer           | 0.0 (243.175)          | 41.94 (1.505)          |
-| cifar10_resnet18_AdvNegGrad_5.pth | Dog            | 0.0 (76.483)           | 41.80 (1.516)          |
-| cifar10_resnet18_AdvNegGrad_6.pth | Frog           | 0.0 (86.987)           | 49.31 (1.333)          |
-| cifar10_resnet18_AdvNegGrad_7.pth | Horse          | 0.0 (93.724)           | 42.44 (1.481)          |
-| cifar10_resnet18_AdvNegGrad_8.pth | Ship           | 0.0 (78.647)           | 33.76 (1.695)          |
-| cifar10_resnet18_AdvNegGrad_9.pth | Truck          | 0.0 (132.552)          | 30.61 (1.848)          |
+where:
 
+\[
+\mathcal{L}_f = - \sum_{i \in \mathcal{D}_f} \log p(y_i | x_i, \theta)
+\]
 
----
+\[
+\mathcal{L}_r = \sum_{j \in \mathcal{D}_r} \log p(y_j | x_j, \theta)
+\]
+
+- \( \mathcal{D}_f \) is the forget dataset.
+- \( \mathcal{D}_r \) is the retain dataset.
+- \( \alpha \) (denoted as `forget_coefficient` in the code) controls the balance between forgetting and retaining.
 
-### Notes
+### Gradient Update:
 
-1. **Forget Class Accuracy and Loss**:
-   - The forget class accuracy is consistently `0.0` for all excluded classes, confirming the effectiveness of the **AdvNegGrad** method in fully excluding the targeted classes.
-   - The forget class loss varies significantly, ranging from `50.233` ("Automobile") to `243.175` ("Deer"). The high loss values for certain classes, such as "Deer" and "Bird," suggest that these classes may require more resources to achieve complete exclusion.
+- **Forget loss gradient ascent** (negating gradients):
 
-2. **Retain Class Accuracy and Loss**:
-   - Retain class accuracy shows significant variability, ranging from `30.61%` ("Truck") to `49.31%` ("Frog"). This indicates that the method's ability to preserve performance on the retained classes is inconsistent across classes.
-   - Retain class loss generally aligns with the retain accuracy, with the lowest loss observed for "Frog" (1.333) and the highest for "Truck" (1.848). Classes with lower retain accuracy tend to have higher retain losses, highlighting a trade-off in performance.
+\[
+\theta \leftarrow \theta - \eta \nabla_{\theta} \mathcal{L}_r + \eta \alpha \nabla_{\theta} \mathcal{L}_f
+\]
+
+- **Gradient clipping**:
+
+\[
+\nabla_{\theta} \mathcal{L} \leftarrow \frac{\nabla_{\theta} \mathcal{L}}{\max(1, \frac{\|\nabla_{\theta} \mathcal{L}\|}{C})}
+\]
+
+where \( C \) is the clipping threshold (`grad_norm_clip` in the code).
 
-3. **Class-Specific Observations**:
-   - "Truck" exhibits the lowest retain class accuracy (30.61%) and highest retain class loss (1.848), suggesting that the exclusion of this class may significantly impact the model's overall balance.
-   - In contrast, "Frog" achieves the highest retain class accuracy (49.31%) with the lowest retain loss (1.333), demonstrating that some classes are more robust to the exclusion process.
-   - The high forget class loss for "Deer" and "Bird" indicates that these classes may have more overlapping features with other classes, making their exclusion computationally more challenging.
 
 ---
 
-### Conclusion
+| Model                          | Forget Class | Forget class acc(loss) | Retain class acc(loss) |
+|--------------------------------|--------------|-------------------------|-------------------------|
+| cifar10_resnet18_AdvNegGrad_0.pth | Airplane     | 0.0 (28.448)            | 90.52 (0.631)          |
+| cifar10_resnet18_AdvNegGrad_1.pth | Automobile   | 0.0 (31.394)            | 91.27 (0.516)          |
+| cifar10_resnet18_AdvNegGrad_2.pth | Bird         | 0.0 (30.110)            | 92.72 (0.475)          |
+| cifar10_resnet18_AdvNegGrad_3.pth | Cat          | 0.0 (26.171)            | 92.44 (0.512)          |
+| cifar10_resnet18_AdvNegGrad_4.pth | Deer         | 0.0 (27.805)            | 91.19 (0.561)          |
+| cifar10_resnet18_AdvNegGrad_5.pth | Dog          | 0.0 (28.574)            | 92.81 (0.456)          |
+| cifar10_resnet18_AdvNegGrad_6.pth | Frog         | 0.0 (28.360)            | 92.18 (0.486)          |
+| cifar10_resnet18_AdvNegGrad_7.pth | Horse        | 0.0 (32.505)            | 92.89 (0.401)          |
+| cifar10_resnet18_AdvNegGrad_8.pth | Ship         | 0.0 (29.307)            | 91.34 (0.543)          |
+| cifar10_resnet18_AdvNegGrad_9.pth | Truck        | 0.0 (28.959)            | 92.47 (0.474)          |
 
-The results demonstrate that the **AdvNegGrad method** is effective in achieving complete exclusion of targeted classes while partially preserving performance on the retained classes. However, the method shows variability in its ability to maintain high accuracy and low loss for the retained classes, indicating potential areas for improvement.
 
-- **Strengths**:
-  - The forget class accuracy is consistently `0.0`, ensuring complete suppression of the excluded classes.
-  - Some classes, such as "Frog" and "Cat," achieve relatively high retain class accuracy (49.31% and 49.02%, respectively), demonstrating that the method can preserve knowledge in certain scenarios.
 
-- **Weaknesses**:
-  - Retain class accuracy is low overall, with several classes scoring below 40%. This suggests that the method struggles to maintain performance on the retained classes.
-  - High forget class loss for certain classes (e.g., "Deer" and "Bird") indicates that the exclusion process is more resource-intensive for these classes, which may reflect class-specific feature overlaps or shared characteristics.
+---
 
-- **Future Work**:
-  - Explore adaptive strategies to improve retain class accuracy and reduce loss, particularly for classes like "Truck" and "Ship."
-  - Investigate why certain classes, such as "Deer" and "Bird," have significantly higher forget class loss and optimize the exclusion process for such challenging classes.
-  - Test the **AdvNegGrad method** on other datasets and architectures to evaluate its generalizability and identify consistent patterns in performance.