Sync from GitHub via hub-sync

.python-version

@@ -0,0 +1 @@
+3.12

LICENSE

@@ -0,0 +1,6 @@
+Licenses
+
+Unless specifically labeled otherwise, these Datasets are provided to You under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License (“CC BY-NC-SA 4.0”), with the additional terms included herein. 
+The CC BY-NC-SA 4.0 may be accessed at https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode. When You download or use the Datasets from the Website or elsewhere, You are agreeing to comply with the terms of CC BY-NC-SA 4.0, and also agreeing to the Dataset Terms. 
+Where these Dataset Terms conflict with the terms of CC BY-NC-SA 4.0, these Dataset Terms shall prevail. We reiterate once again that this dataset is used only for non-commercial purposes such as academic research, teaching, or scientific publications. 
+We prohibits You from using the dataset or any derivative works for commercial purposes, such as selling data or using it for commercial gain.

README.md

@@ -1,14 +1,189 @@
 ---
-title: DeForge AIGIBench
-emoji: 👀
-colorFrom: pink
+title: DeForge AI
+emoji: 📉
+colorFrom: yellow
 colorTo: gray
 sdk: gradio
 sdk_version: 6.14.0
-python_version: '3.13'
+python_version: '3.12'
 app_file: app.py
 pinned: false
-license: cc-by-nc-sa-4.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+<div align="center">
+ <br>
+<h1>Is Artificial Intelligence Generated Image Detection a Solved Problem?</h1>
+ 
+[Ziqiang Li](https://scholar.google.com/citations?user=mj5a8WgAAAAJ&hl=zh-CN)<sup>1</sup>, [Jiazhen Yan](https://scholar.google.com/citations?user=QkURh8EAAAAJ&hl=zh-CN)<sup>1</sup>, [Ziwen He](https://scholar.google.com/citations?user=PjkDK9cAAAAJ&hl=zh-CN)<sup>1</sup>, [Kai Zeng](https://scholar.google.com.hk/citations?user=TsI93SIAAAAJ&hl=zh-CN)<sup>2</sup>, [Weiwei Jiang](https://scholar.google.co.jp/citations?user=mbPN0hgAAAAJ&hl=zh-CN)<sup>1</sup>, [Lizhi Xiong](https://scholar.google.com/citations?user=-FzrEP4AAAAJ&hl=zh-CN)<sup>1</sup>, [Zhangjie Fu](https://scholar.google.com/citations?user=fO9NmagAAAAJ&hl=zh-CN)<sup>1‡</sup>
+
+
+<div class="is-size-6 publication-authors">
+  <p class="footnote">
+    <span class="footnote-symbol"><sup>‡</sup></span>Corresponding author
+  </p>
+</div>
+
+<sup>1</sup>Nanjing University of Information Science and Technology <sup>2</sup>University of Siena
+<p align="center">
+  <a href='https://github.com/HorizonTEL/AIGIBench'>
+    <img src='https://img.shields.io/badge/Project-Page-pink?style=flat&logo=Google%20chrome&logoColor=pink'>
+  </a>
+  <a href='https://arxiv.org/abs/2505.12335'>
+    <img src='https://img.shields.io/badge/Arxiv-2406.19435-A42C25?style=flat&logo=arXiv&logoColor=A42C25'>
+  </a>
+  <a href='https://arxiv.org/pdf/2505.12335'>
+    <img src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv&logoColor=yellow'>
+  </a>
+</p>
+</div>
+
+## 🔥 News
+* [2025-09-19]🎉🎉🎉 AIGIBench is accepted by NeurIPS 2025 Datasets and Benchmarks.
+
+##
+
+**This repository is the official repository of the AIGIBench.**
+
+> [!NOTE]
+> This is a **modified version** of the original [AIGIBench](https://github.com/HorizonTEL/AIGIBench) repository. In addition to the original dataset and methods, it includes my custom detection solutions: **DeForge-AI** and **C2P-DINOv2** (intermediary solution).
+
+**This repository contains the AIGIBench dataset and the evaluated methods.**
+
+ **AIGIBench** dataset contains two types of training and 25 test subsets. This dataset has the following advantages:
+- Comprehensive generate types: including GAN-based Noise-to-Image Generation, Diffusion for Text-to-Image Generation, GANs for Deepfake, Diffusion for Personalized Generation, and Open-source Platforms.
+- State-of-the-art Generators: MidjourneyV6, Stable Diffusion 3, Imagen, DALLE3, InstantID, FaceSwap, StyleGAN-XL and so on.
+- Completely unknown generation method: Crawl pictures from communities and social media to build datasets CommunityAI & SocialRF, making detection more challenging.
+
+![example](https://github.com/user-attachments/assets/36250270-6fc1-4919-8078-1865f80913c0)
+
+If this project helps you, please fork, watch, and give a star to this repository.  
+
+## 📚Dataset
+The training set and testing set used in the paper can be downloaded on [Huggingface](https://huggingface.co/datasets/HorizonTEL/AIGIBench)/[Baidu Netdisk](https://pan.baidu.com/s/1XTwfXlfqkGxAwYLxXuZbfA?pwd=sm6v).
+
+Each folder contains compressed files. After unzip the file, files under the data root directory can be organized as follows.
+### Train
+AIGIBench introduces two training dataset settings: **(i) Setting-I:** Training on 144K images generated by ProGAN across four object categories—car, cat, chair, and horse. **(ii) Setting-II:** Training on 144K images generated by both SD-v1.4 and ProGAN, covering the same four object categories. The data of ProGAN comes from ForenSynths, and the data of sdv1.4 comes from GenImage. In order to maintain the fairness of the training data, we randomly select the sdv1.4 training images of GenImage to keep the same number as ProGAN, and then merge the data. The file directory is as follows:
+
+```
+├── train
+│   ├── car
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   ├── cat
+│   │   ├── ...
+│   ├── chair
+│   │   ├── ...
+│   ├── horse
+│   │   ├── ...
+│   ├── sdv1.4
+│   │   ├── 0_real
+│   │   ├── 1_fake
+├── val
+│   ├── ...
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   │   ...
+```
+
+### Test
+AIGIBench comprehensively tests the performance of the detector and builds a test dataset from five perspectives: GAN-based Noise-to-Image Generation, Diffusion for Text-to-Image Generation, GANs for Deepfake, Diffusion for Personalized Generation, and Open-source Platforms. The file directory is as follows:
+```
+├── test
+│   ├── ProGAN
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   ├── R3GAN
+│   │   ├── ...
+│   │   ...
+│   ├── BlendFace
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   ├── InSwap
+│   │   ├── ...
+│   │   ...
+│   ├── FLUX1-dev
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   ├── Midjourney-V6
+│   │   ├── ...
+│   │   ...
+│   ├── BLIP
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   ├── Infinite-ID
+│   │   ├── ...
+│   │   ...
+│   ├── CommunityAI
+│   │   ├── 0_real
+│   │   ├── 1_fake
+│   ├── SocialRF
+│   │   ├── ...
+```
+
+*Note: The test set count in the paper contained some errors, which we are correcting here. Please note that the number of real images and generated images are consistent; only the number of generated images is listed below.*
+| Generator    |   Number  |
+|:------:      |:---------:|
+| CommunityAI  | 6000      |
+| SocialRF     | 3000      |
+| FaceSwap     | 4000      |
+| ImSwap       | 4000      |
+| WFIR         | 1000      |
+
+## 🔍Detection Methods
+We use the official code for all detection codes and make unified modifications to the input and output. The code we use for training in Setting-II is publicly available above, the corresponding pre-trained checkpoints are publicly available on [Huggingface](https://huggingface.co/HorizonTEL/AIGIBench). Of course, if you need the code from the original paper, the following is the corresponding detection code in the paper：
+- [ResNet-50](https://github.com/huggingface/pytorch-image-models/tree/v0.6.12/timm): Deep Residual Learning for Image Recognition
+- [CNNDetection](https://github.com/PeterWang512/CNNDetection): CNN-generated images are surprisingly easy to spot...for now
+- [GramNet](https://github.com/liuzhengzhe/Global_Texture_Enhancement_for_Fake_Face_Detection_in_the-Wild): Global Texture Enhancement for Fake Face Detection in the Wild
+- [LGrad](https://github.com/chuangchuangtan/LGrad): Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection
+- [CLIPDetection](https://github.com/WisconsinAIVision/UniversalFakeDetect): Towards Universal Fake Image Detectors that Generalize Across Generative Models
+- [FreqNet](https://github.com/chuangchuangtan/FreqNet-DeepfakeDetection): FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform
+- [NPR](https://github.com/chuangchuangtan/NPR-DeepfakeDetection): Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection
+- [DFFreq](https://github.com/HorizonTEL/DFFreq-main): Dual Frequency Branch Framework with Reconstructed Sliding Windows Attention for AI-Generated Image Detection
+- [LaDeDa](https://github.com/barcavia/RealTime-DeepfakeDetection-in-the-RealWorld): Real-Time Deepfake Detection in the Real-World
+- [AIDE](https://github.com/shilinyan99/AIDE): A Sanity Check for AI-generated Image Detection
+- [SAFE](https://github.com/Ouxiang-Li/SAFE): Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspectives
+- [Effort](https://github.com/YZY-stack/Effort-AIGI-Detection): Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection
+
+## ⏳Detection Results (Continuously updating)
+**To ensure a fair comparison, we retrain all baseline methods on the Setting-II of AIGIBench.**
+
+_If your retrained results differ significantly from those shown, please contact us._
+| Method       |    Paper                                                                                                      |    Ref    | R.Acc. | F.Acc. | Acc. | A.P. |
+|:------:      |:---------:                                                                                                    |:---------:|:------:|:------:|:----:|:----:|
+| CNNDetection | CNN-generated images are surprisingly easy to spot... for now                                                 | CVPR 2020 |**98.2**|  11.6  | 54.9 | 67.0 |
+| Gram-Net     | Global Texture Enhancement for Fake Face Detection In the Wild                                                | CVPR 2020 |  90.5  |  26.6  | 58.6 | 62.4 |
+| LGrad        | Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection                | CVPR 2023 |  85.8  |  39.6  | 62.9 | 66.6 |
+| UniFD        | Towards Universal Fake Image Detectors that  Generalize Across Generative Models                              | CVPR 2023 |  73.3  |  71.5  | 72.5 | 75.6 |
+| FreqNet      | Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning               | AAAI 2024 |  65.9  |  66.4  | 66.2 | 70.1 |
+| NPR          | Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection    | CVPR 2024 |  93.8  |  41.9  | 67.9 | 73.9 |
+| Ladeda       | Real-Time Deepfake Detection in the Real-World                                                                | Arxiv 2024|  91.7  |  54.9  | 73.4 | 79.3 |
+| DFFreq       | Dual Frequency Branch Framework with Reconstructed Sliding Windows Attention for AI-Generated Image Detection | TIFS 2026 |  91.8  |  58.0  | 75.1 | 82.2 |
+| C2P-CLIP*    | C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection            | AAAI 2025 |  93.8  |  49.8  | 71.8 | 82.2 |
+| AIDE         | A Sanity Check for AI-generated Image Detection                                                               | ICLR 2025 |  88.1  |  67.0  | 77.6 | 82.7 |
+| SAFE         | Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspectives              | KDD 2025  |  89.0  |  66.6  | 78.1 | 83.6 |
+| VIB-Net      | Towards Universal AI-Generated Image Detection by Variational Information Bottleneck Network                  | CVPR 2025 |  60.6  |**78.1**| 69.3 | 70.9 |
+| $D^3$        | $D^3$: Scaling Up Deepfake Detection by Learning from Discrepancy                                             | CVPR 2025 |  81.0  |  46.4  | 63.7 | 68.9 |
+| Effort       | Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection                              | ICML 2025 |  96.9  |  57.1  | 77.1 |**87.2**|
+| FerretNet    | FerretNet: Efficient Synthetic Image Detection via Local Pixel Dependencies                                   | NIPS 2025 |  96.6  |  61.8  |**79.4**| 85.8 |
+| LOTA         | LOTA: Bit-Planes Guided AI-Generated Image Detection                                                          | ICCV 2025 |  89.3  |  65.1  | 77.4 | 83.1 |
+| BSF          | Beyond Semantic Features: Pixel-level Mapping for Generalized AI-Generated Image Detection                    | AAAI 2026 |  91.5  |  65.6  | 78.8 | 81.1 |
+| LTD          | Layer Consistency Matters: Elegant Latent Transition Discrepancy for Generalizable Synthetic Image Detection  | CVPR 2026 |  82.0  |  67.7  | 74.9 | 77.6 |
+
+**For specific reasons, in the following method, we directly utilize the official pre-trained weights for inference.**
+| Method       |    Paper                                                                                                      |    Ref    | R.Acc. | F.Acc. | Acc. | A.P. |
+|:------:      |:---------:                                                                                                    |:---------:|:------:|:------:|:----:|:----:|
+| DDA          | Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizable                                    | NIPS 2025 |  93.9  |  69.3  | 81.6 | 90.2 |
+ 
+## Citation
+```
+@inproceedings{li2025artificial,
+  title={Is Artificial Intelligence Generated Image Detection a Solved Problem?},
+  author={Li, Ziqiang and Yan, Jiazhen and He, Ziwen and Zeng, Kai and Jiang, Weiwei and Xiong, Lizhi and Fu, Zhangjie},
+  booktitle={Advances in Neural Information Processing Systems},
+  year={2025}
+}
+```
+
+## Contact
+If you have any question about this project, please feel free to contact 247918horizon@gmail.com

app.py

@@ -0,0 +1,81 @@
+import gradio as gr
+
+from detector_codes import DEVICE, detector_classes, weight_mapping
+
+# Model cache to avoid reloading
+model_cache = {'name': None, 'instance': None}
+
+
+def predict(model_name, input_image):
+    if input_image is None:
+        return 'Please upload an image.'
+
+    global model_cache
+
+    # Load model if not in cache or if model changed
+    if model_cache['name'] != model_name:
+        print(f'Loading model: {model_name}...')
+        try:
+            detector_class = detector_classes[model_name]
+            weights = weight_mapping[model_name]
+            model_cache['instance'] = detector_class(weights)
+            model_cache['name'] = model_name
+        except Exception as e:
+            return f'Error loading model {model_name}: {str(e)}'
+
+    detector = model_cache['instance']
+
+    # Preprocess image
+    try:
+        img_tensor = detector.transform(input_image).unsqueeze(0).to(DEVICE)
+
+        # Inference
+        p_fake = detector.detect(img_tensor).item()
+        p_real = 1.0 - p_fake
+
+        return {'Real Image': p_real, 'AI Generated': p_fake}
+    except Exception as e:
+        return f'Error during inference: {str(e)}'
+
+
+# Define the Gradio interface
+with gr.Blocks(title='AIGI Detector Bench') as demo:
+    gr.Markdown('# AIGI Detector Benchmark')
+    gr.Markdown(
+        "Select a model and upload an image to check if it's AI-generated or real."
+    )
+
+    with gr.Row():
+        with gr.Column():
+            model_dropdown = gr.Dropdown(
+                choices=sorted(list(detector_classes.keys())),
+                value='DeForge-AI',
+                label='Select Detection Model',
+            )
+            input_img = gr.Image(type='pil', label='Upload Image')
+            btn = gr.Button('Detect', variant='primary')
+
+        with gr.Column():
+            output_label = gr.Label(num_top_classes=2, label='Prediction')
+
+    btn.click(fn=predict, inputs=[model_dropdown, input_img], outputs=output_label)
+
+    gr.Markdown("""
+    ### About
+    This tool is a **modified version** of the official [AIGIBench](https://github.com/HorizonTEL/AIGIBench) repository, featuring state-of-the-art AI-Generated Image (AIGI) detectors.
+
+    In this version, I have integrated the original baselines along with my own proposed solutions: **DeForge-AI** and **C2P-DINOv2**.
+
+    - **Project Page**: [NeurIPS 2025] Is Artificial Intelligence Generated Image Detection a Solved Problem?
+    - **Original Repository**: [HorizonTEL/AIGIBench](https://github.com/HorizonTEL/AIGIBench)
+
+    #### Featured Models:
+    - **DeForge-AI**: My proposed multi-modal forensic detector (optimized for diverse generators).
+    - **C2P-DINOv2**: My solution leveraging DINOv2 features (intermediary solution).
+    - **RIGID, AIDE, SAFE, Effort, NPR, LaDeDa, etc.**: Original SOTA baselines.
+
+    Each model has different strengths. DeForge-AI generally provides the best performance across diverse generators.
+    """)
+
+if __name__ == '__main__':
+    demo.launch()

detector_codes/AIDE-main/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Shilin Yan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

detector_codes/AIDE-main/data/datasets.py

@@ -0,0 +1,199 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+
+# All rights reserved.
+
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+import os
+from torchvision import transforms
+from torch.utils.data import Dataset
+
+from PIL import Image
+import io
+import torch
+from .dct import DCT_base_Rec_Module
+import random
+
+try:
+    from torchvision.transforms import InterpolationMode
+    BICUBIC = InterpolationMode.BICUBIC
+except ImportError:
+    BICUBIC = Image.BICUBIC
+
+from PIL import ImageFile
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+import kornia.augmentation as K
+
+Perturbations = K.container.ImageSequential(
+    K.RandomGaussianBlur(kernel_size=(3, 3), sigma=(0.1, 3.0), p=0.1),
+    K.RandomJPEG(jpeg_quality=(30, 100), p=0.1)
+)
+
+transform_before = transforms.Compose([
+    transforms.ToTensor(),
+    transforms.Lambda(lambda x: Perturbations(x)[0])
+    ]
+)
+transform_before_test = transforms.Compose([
+    transforms.ToTensor(),
+    ]
+)
+
+transform_train = transforms.Compose([
+    transforms.Resize([256, 256]),
+    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]
+)
+
+transform_test_normalize = transforms.Compose([
+    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]
+)
+
+
+class TrainDataset(Dataset):
+    def __init__(self, is_train, args):
+        
+        root = args.data_path if is_train else args.eval_data_path
+
+        self.data_list = []
+
+        if'GenImage' in root and root.split('/')[-1] != 'train':
+            file_path = root
+
+            if '0_real' not in os.listdir(file_path):
+                for folder_name in os.listdir(file_path):
+                
+                    assert (os.listdir(os.path.join(file_path, folder_name)) == ['0_real', '1_fake']) or (os.listdir(os.path.join(file_path, folder_name)) == ['1_fake', '0_real'])
+
+                    for image_path in os.listdir(os.path.join(file_path, folder_name, '0_real')):
+                        self.data_list.append({"image_path": os.path.join(file_path, folder_name, '0_real', image_path), "label" : 0})
+                 
+                    for image_path in os.listdir(os.path.join(file_path, folder_name, '1_fake')):
+                        self.data_list.append({"image_path": os.path.join(file_path, folder_name, '1_fake', image_path), "label" : 1})
+            
+            else:
+                for image_path in os.listdir(os.path.join(file_path, '0_real')):
+                    self.data_list.append({"image_path": os.path.join(file_path, '0_real', image_path), "label" : 0})
+                for image_path in os.listdir(os.path.join(file_path, '1_fake')):
+                    self.data_list.append({"image_path": os.path.join(file_path, '1_fake', image_path), "label" : 1})
+        else:
+
+            for filename in os.listdir(root):
+
+                file_path = os.path.join(root, filename)
+
+                if '0_real' not in os.listdir(file_path):
+                    for folder_name in os.listdir(file_path):
+                    
+                        assert (os.listdir(os.path.join(file_path, folder_name)) == ['0_real', '1_fake']) or (os.listdir(os.path.join(file_path, folder_name)) == ['1_fake', '0_real'])
+
+                        for image_path in os.listdir(os.path.join(file_path, folder_name, '0_real')):
+                            self.data_list.append({"image_path": os.path.join(file_path, folder_name, '0_real', image_path), "label" : 0})
+                    
+                        for image_path in os.listdir(os.path.join(file_path, folder_name, '1_fake')):
+                            self.data_list.append({"image_path": os.path.join(file_path, folder_name, '1_fake', image_path), "label" : 1})
+                
+                else:
+                    for image_path in os.listdir(os.path.join(file_path, '0_real')):
+                        self.data_list.append({"image_path": os.path.join(file_path, '0_real', image_path), "label" : 0})
+                    for image_path in os.listdir(os.path.join(file_path, '1_fake')):
+                        self.data_list.append({"image_path": os.path.join(file_path, '1_fake', image_path), "label" : 1})
+                
+        self.dct = DCT_base_Rec_Module()
+
+
+    def __len__(self):
+        return len(self.data_list)
+
+    def __getitem__(self, index):
+        
+        sample = self.data_list[index]
+                
+        image_path, targets = sample['image_path'], sample['label']
+
+        try:
+            image = Image.open(image_path).convert('RGB')
+        except:
+            print(f'image error: {image_path}')
+            return self.__getitem__(random.randint(0, len(self.data_list) - 1))
+
+
+        image = transform_before(image)
+
+        try:
+            x_minmin, x_maxmax, x_minmin1, x_maxmax1 = self.dct(image)
+        except:
+            print(f'image error: {image_path}, c, h, w: {image.shape}')
+            return self.__getitem__(random.randint(0, len(self.data_list) - 1))
+
+        x_0 = transform_train(image)
+        x_minmin = transform_train(x_minmin) 
+        x_maxmax = transform_train(x_maxmax)
+
+        x_minmin1 = transform_train(x_minmin1) 
+        x_maxmax1 = transform_train(x_maxmax1)
+        
+
+
+        return torch.stack([x_minmin, x_maxmax, x_minmin1, x_maxmax1, x_0], dim=0), torch.tensor(int(targets))
+
+    
+
+class TestDataset(Dataset):
+    def __init__(self, is_train, args):
+        
+        root = args.data_path if is_train else args.eval_data_path
+
+        self.data_list = []
+
+        file_path = root
+
+        if '0_real' not in os.listdir(file_path):
+            for folder_name in os.listdir(file_path):
+    
+                assert (os.listdir(os.path.join(file_path, folder_name)) == ['0_real', '1_fake']) or (os.listdir(os.path.join(file_path, folder_name)) == ['1_fake', '0_real'])
+                
+                for image_path in os.listdir(os.path.join(file_path, folder_name, '0_real')):
+                    self.data_list.append({"image_path": os.path.join(file_path, folder_name, '0_real', image_path), "label" : 0})
+                
+                for image_path in os.listdir(os.path.join(file_path, folder_name, '1_fake')):
+                    self.data_list.append({"image_path": os.path.join(file_path, folder_name, '1_fake', image_path), "label" : 1})
+        
+        else:
+            for image_path in os.listdir(os.path.join(file_path, '0_real')):
+                self.data_list.append({"image_path": os.path.join(file_path, '0_real', image_path), "label" : 0})
+            for image_path in os.listdir(os.path.join(file_path, '1_fake')):
+                self.data_list.append({"image_path": os.path.join(file_path, '1_fake', image_path), "label" : 1})
+
+
+        self.dct = DCT_base_Rec_Module()
+
+
+    def __len__(self):
+        return len(self.data_list)
+
+    def __getitem__(self, index):
+        
+        sample = self.data_list[index]
+                
+        image_path, targets = sample['image_path'], sample['label']
+
+        image = Image.open(image_path).convert('RGB')
+
+        image = transform_before_test(image)
+
+        # x_max, x_min, x_max_min, x_minmin = self.dct(image)
+
+        x_minmin, x_maxmax, x_minmin1, x_maxmax1 = self.dct(image)
+
+
+        x_0 = transform_train(image)                # 上采样到256*256
+        x_minmin = transform_train(x_minmin) 
+        x_maxmax = transform_train(x_maxmax)
+
+        x_minmin1 = transform_train(x_minmin1) 
+        x_maxmax1 = transform_train(x_maxmax1)
+        
+        return torch.stack([x_minmin, x_maxmax, x_minmin1, x_maxmax1, x_0], dim=0), torch.tensor(int(targets))
+

detector_codes/AIDE-main/data/dct.py

@@ -0,0 +1,151 @@
+import os
+import torch
+import torch.nn as nn
+import numpy as np
+import torch.nn.functional as F
+
+
+def DCT_mat(size):
+    m = [[ (np.sqrt(1./size) if i == 0 else np.sqrt(2./size)) * np.cos((j + 0.5) * np.pi * i / size) for j in range(size)] for i in range(size)]
+    return m
+
+
+def generate_filter(start, end, size):
+    return [[0. if i + j > end or i + j < start else 1. for j in range(size)] for i in range(size)]
+
+
+def norm_sigma(x):
+    return 2. * torch.sigmoid(x) - 1.
+
+
+class Filter(nn.Module):
+    def __init__(self, size, band_start, band_end, use_learnable=False, norm=False):
+        super(Filter, self).__init__()
+        self.use_learnable = use_learnable
+        self.base = nn.Parameter(torch.tensor(generate_filter(band_start, band_end, size)), requires_grad=False)
+        if self.use_learnable:
+            self.learnable = nn.Parameter(torch.randn(size, size), requires_grad=True)
+            self.learnable.data.normal_(0., 0.1)
+        self.norm = norm
+        if norm:
+            self.ft_num = nn.Parameter(torch.sum(torch.tensor(generate_filter(band_start, band_end, size))), requires_grad=False)
+
+
+    def forward(self, x):
+        if self.use_learnable:
+            filt = self.base + norm_sigma(self.learnable)
+        else:
+            filt = self.base
+
+        if self.norm:
+            y = x * filt / self.ft_num
+        else:
+            y = x * filt
+        return y
+
+
+class DCT_base_Rec_Module(nn.Module):
+    """_summary_
+
+    Args:
+        x: [C, H, W] -> [C*level, output, output]
+    """
+    def __init__(self, window_size=32, stride=16, output=256, grade_N=6, level_fliter=[0]):
+        super().__init__()
+        
+        assert output % window_size == 0
+        assert len(level_fliter) > 0
+        
+        self.window_size = window_size
+        self.grade_N = grade_N
+        self.level_N = len(level_fliter)
+        self.N = (output // window_size) * (output // window_size)
+
+        self._DCT_patch = nn.Parameter(torch.tensor(DCT_mat(window_size)).float(), requires_grad=False)
+        self._DCT_patch_T = nn.Parameter(torch.transpose(torch.tensor(DCT_mat(window_size)).float(), 0, 1), requires_grad=False)
+        
+        self.unfold = nn.Unfold(
+            kernel_size=(window_size, window_size), stride=stride
+        )
+        self.fold0 = nn.Fold(
+            output_size=(window_size, window_size), 
+            kernel_size=(window_size, window_size), 
+            stride=window_size
+        )
+        
+        lm, mh = 2.82, 2
+        level_f = [
+            Filter(window_size, 0, window_size * 2)
+        ]
+
+        self.level_filters = nn.ModuleList([level_f[i] for i in level_fliter])
+        self.grade_filters = nn.ModuleList([Filter(window_size, window_size * 2. / grade_N * i, window_size * 2. / grade_N * (i+1), norm=True) for i in range(grade_N)])
+
+    def forward(self, x):
+        
+        N = self.N
+        grade_N = self.grade_N
+        level_N = self.level_N
+        window_size = self.window_size
+        C, W, H = x.shape
+        x_unfold = self.unfold(x.unsqueeze(0)).squeeze(0)  
+        
+
+        _, L = x_unfold.shape
+        x_unfold = x_unfold.transpose(0, 1).reshape(L, C, window_size, window_size) 
+        x_dct = self._DCT_patch @ x_unfold @ self._DCT_patch_T
+        
+        y_list = []
+        for i in range(self.level_N):
+            x_pass = self.level_filters[i](x_dct)
+            y = self._DCT_patch_T @ x_pass @ self._DCT_patch
+            y_list.append(y)
+        level_x_unfold = torch.cat(y_list, dim=1)
+        
+        grade = torch.zeros(L).to(x.device)
+        w, k = 1, 2
+        for _ in range(grade_N):
+            _x = torch.abs(x_dct)
+            _x = torch.log(_x + 1)
+            _x = self.grade_filters[_](_x)
+            _x = torch.sum(_x, dim=[1,2,3])
+            grade += w * _x            
+            w *= k
+        
+        _, idx = torch.sort(grade)
+        max_idx = torch.flip(idx, dims=[0])[:N]
+        maxmax_idx = max_idx[0]
+        if len(max_idx) == 1:
+            maxmax_idx1 = max_idx[0]
+        else:
+            maxmax_idx1 = max_idx[1]
+
+        min_idx = idx[:N]
+        minmin_idx = idx[0]
+        if len(min_idx) == 1:
+            minmin_idx1 = idx[0]
+        else:
+            minmin_idx1 = idx[1]
+
+        x_minmin = torch.index_select(level_x_unfold, 0, minmin_idx)
+        x_maxmax = torch.index_select(level_x_unfold, 0, maxmax_idx)
+        x_minmin1 = torch.index_select(level_x_unfold, 0, minmin_idx1)
+        x_maxmax1 = torch.index_select(level_x_unfold, 0, maxmax_idx1)
+
+        x_minmin = x_minmin.reshape(1, level_N*C*window_size* window_size).transpose(0, 1)
+        x_maxmax = x_maxmax.reshape(1, level_N*C*window_size* window_size).transpose(0, 1)
+        x_minmin1 = x_minmin1.reshape(1, level_N*C*window_size* window_size).transpose(0, 1)
+        x_maxmax1 = x_maxmax1.reshape(1, level_N*C*window_size* window_size).transpose(0, 1)
+
+        x_minmin = self.fold0(x_minmin)
+        x_maxmax = self.fold0(x_maxmax)
+        x_minmin1 = self.fold0(x_minmin1)
+        x_maxmax1 = self.fold0(x_maxmax1)
+
+       
+        return x_minmin, x_maxmax, x_minmin1, x_maxmax1
+
+
+        
+
+

detector_codes/AIDE-main/engine_finetune.py

@@ -0,0 +1,202 @@
+import math
+from typing import Iterable, Optional
+
+import torch
+import utils
+from scipy.special import softmax
+from sklearn.metrics import accuracy_score, average_precision_score
+from timm.data import Mixup
+from timm.utils import ModelEma, accuracy
+from utils import adjust_learning_rate
+
+
+def train_one_epoch(
+    model: torch.nn.Module,
+    criterion: torch.nn.Module,
+    data_loader: Iterable,
+    optimizer: torch.optim.Optimizer,
+    device: torch.device,
+    epoch: int,
+    loss_scaler,
+    max_norm: float = 0,
+    model_ema: Optional[ModelEma] = None,
+    mixup_fn: Optional[Mixup] = None,
+    log_writer=None,
+    args=None,
+):
+    model.train(True)
+    metric_logger = utils.MetricLogger(delimiter='  ')
+    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
+    header = 'Epoch: [{}]'.format(epoch)
+    print_freq = 100
+
+    update_freq = args.update_freq
+    use_amp = args.use_amp
+    optimizer.zero_grad()
+
+    for data_iter_step, (samples, targets) in enumerate(
+        metric_logger.log_every(data_loader, print_freq, header)
+    ):
+        # we use a per iteration (instead of per epoch) lr scheduler
+        if data_iter_step % update_freq == 0:
+            adjust_learning_rate(
+                optimizer, data_iter_step / len(data_loader) + epoch, args
+            )
+
+        samples = samples.to(device, non_blocking=True)
+        targets = targets.to(device, non_blocking=True)
+
+        if mixup_fn is not None:
+            samples, targets = mixup_fn(samples, targets)
+
+        if use_amp:
+            with torch.cuda.amp.autocast():
+                output = model(samples)
+                loss = criterion(output, targets)
+        else:  # full precision
+            output = model(samples)
+            loss = criterion(output, targets)
+
+        loss_value = loss.item()
+
+        if not math.isfinite(loss_value):
+            print('Loss is {}, stopping training'.format(loss_value))
+            assert math.isfinite(loss_value)
+
+        if use_amp:
+            # this attribute is added by timm on one optimizer (adahessian)
+            is_second_order = (
+                hasattr(optimizer, 'is_second_order') and optimizer.is_second_order
+            )
+            loss /= update_freq
+            grad_norm = loss_scaler(
+                loss,
+                optimizer,
+                clip_grad=max_norm,
+                parameters=model.parameters(),
+                create_graph=is_second_order,
+                update_grad=(data_iter_step + 1) % update_freq == 0,
+            )
+            if (data_iter_step + 1) % update_freq == 0:
+                optimizer.zero_grad()
+                if model_ema is not None:
+                    model_ema.update(model)
+        else:  # full precision
+            loss /= update_freq
+            loss.backward()
+            if (data_iter_step + 1) % update_freq == 0:
+                optimizer.step()
+                optimizer.zero_grad()
+                if model_ema is not None:
+                    model_ema.update(model)
+
+        torch.cuda.synchronize()
+
+        if mixup_fn is None:
+            class_acc = (output.max(-1)[-1] == targets).float().mean()
+        else:
+            class_acc = None
+
+        metric_logger.update(loss=loss_value)
+        metric_logger.update(class_acc=class_acc)
+        min_lr = 10.0
+        max_lr = 0.0
+        for group in optimizer.param_groups:
+            min_lr = min(min_lr, group['lr'])
+            max_lr = max(max_lr, group['lr'])
+
+        metric_logger.update(lr=max_lr)
+        metric_logger.update(min_lr=min_lr)
+        weight_decay_value = None
+        for group in optimizer.param_groups:
+            if group['weight_decay'] > 0:
+                weight_decay_value = group['weight_decay']
+        metric_logger.update(weight_decay=weight_decay_value)
+        if use_amp:
+            metric_logger.update(grad_norm=grad_norm)
+        if log_writer is not None:
+            log_writer.update(loss=loss_value, head='loss')
+            log_writer.update(class_acc=class_acc, head='loss')
+            log_writer.update(lr=max_lr, head='opt')
+            log_writer.update(min_lr=min_lr, head='opt')
+            log_writer.update(weight_decay=weight_decay_value, head='opt')
+            if use_amp:
+                log_writer.update(grad_norm=grad_norm, head='opt')
+            log_writer.set_step()
+
+    print('Averaged stats:', metric_logger)
+    return {k: meter.global_avg for k, meter in metric_logger.meters.items()}
+
+
+@torch.no_grad()
+def evaluate(data_loader, model, device, use_amp=False):
+    criterion = torch.nn.CrossEntropyLoss()
+
+    metric_logger = utils.MetricLogger(delimiter='  ')
+    header = 'Test:'
+
+    # switch to evaluation mode
+    model.eval()
+
+    predictions = []
+    labels = []
+
+    for index, batch in enumerate(metric_logger.log_every(data_loader, 10, header)):
+        images = batch[0]
+        target = batch[-1]
+
+        images = images.to(device, non_blocking=True)
+        target = target.to(device, non_blocking=True)
+
+        # compute output
+        if use_amp:
+            with torch.cuda.amp.autocast(dtype=torch.bfloat16):
+                output = model(images)
+                if isinstance(output, dict):
+                    output = output['logits']
+                loss = criterion(output, target)
+        else:
+            output = model(images)  # [bs, num_cls]
+            if isinstance(output, dict):
+                output = output['logits']
+
+            loss = criterion(output, target)
+
+        predictions.append(output)
+        labels.append(target)
+
+        torch.cuda.synchronize()
+
+        acc1, acc5 = accuracy(output, target, topk=(1, 2))
+
+        batch_size = images.shape[0]
+        metric_logger.update(loss=loss.item())
+        metric_logger.meters['acc1'].update(acc1.item(), n=batch_size)
+        metric_logger.meters['acc5'].update(acc5.item(), n=batch_size)
+
+    print(
+        '* Acc@1 {top1.global_avg:.3f} Acc@5 {top5.global_avg:.3f} loss {losses.global_avg:.3f}'.format(
+            top1=metric_logger.acc1, top5=metric_logger.acc5, losses=metric_logger.loss
+        )
+    )
+
+    # Concatenate predictions and labels
+    predictions = torch.cat(predictions, dim=0)
+    labels = torch.cat(labels, dim=0)
+
+    y_pred = softmax(predictions.detach().cpu().numpy(), axis=1)[:, 1]
+    y_true = labels.detach().cpu().numpy()
+    y_true = y_true.astype(int)
+
+    acc = accuracy_score(y_true, y_pred > 0.5)
+    r_acc = accuracy_score(y_true[y_true == 0], y_pred[y_true == 0] > 0.5)
+    f_acc = accuracy_score(y_true[y_true == 1], y_pred[y_true == 1] > 0.5)
+    ap = average_precision_score(y_true, y_pred)
+
+    return (
+        {k: meter.global_avg for k, meter in metric_logger.meters.items()},
+        acc,
+        ap,
+        r_acc,
+        f_acc,
+    )

detector_codes/AIDE-main/main_finetune.py

@@ -0,0 +1,657 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import csv
+import datetime
+import json
+import os
+import time
+import warnings
+from pathlib import Path
+
+import models.AIDE as AIDE
+import numpy as np
+import torch
+import torch.backends.cudnn as cudnn
+import utils
+from data.datasets import TestDataset, TrainDataset
+from engine_finetune import evaluate, train_one_epoch
+from optim_factory import create_optimizer
+from timm.data.mixup import Mixup
+from timm.loss import LabelSmoothingCrossEntropy, SoftTargetCrossEntropy
+from timm.utils import ModelEma
+from utils import NativeScalerWithGradNormCount as NativeScaler
+from utils import str2bool
+
+warnings.filterwarnings('ignore')
+
+
+def get_args_parser():
+    parser = argparse.ArgumentParser('Resnet fine-tuning', add_help=False)
+    parser.add_argument('--batch_size', default=32, type=int, help='Per GPU batch size')
+    parser.add_argument('--epochs', default=100, type=int)
+    parser.add_argument(
+        '--update_freq', default=1, type=int, help='gradient accumulation steps'
+    )
+
+    # Model parameters
+    parser.add_argument(
+        '--model',
+        default='AIDE',
+        type=str,
+        metavar='MODEL',
+        help='Name of model to train',
+    )
+    parser.add_argument(
+        '--resnet_path',
+        default=None,
+        type=str,
+        metavar='MODEL',
+        help='Path of resnet model',
+    )
+    parser.add_argument(
+        '--convnext_path',
+        default=None,
+        type=str,
+        metavar='MODEL',
+        help='Path of ConvNeXt of model',
+    )
+
+    # EMA related parameters
+    parser.add_argument('--model_ema', type=str2bool, default=False)
+    parser.add_argument('--model_ema_decay', type=float, default=0.9999, help='')
+    parser.add_argument('--model_ema_force_cpu', type=str2bool, default=False, help='')
+    parser.add_argument(
+        '--model_ema_eval',
+        type=str2bool,
+        default=False,
+        help='Using ema to eval during training.',
+    )
+
+    # Optimization parameters
+    parser.add_argument(
+        '--clip_grad',
+        type=float,
+        default=None,
+        metavar='NORM',
+        help='Clip gradient norm (default: None, no clipping)',
+    )
+    parser.add_argument(
+        '--weight_decay', type=float, default=0.0, help='weight decay (default: 0.05)'
+    )
+    parser.add_argument(
+        '--lr',
+        type=float,
+        default=None,
+        metavar='LR',
+        help='learning rate (absolute lr)',
+    )
+    parser.add_argument(
+        '--blr',
+        type=float,
+        default=5e-4,
+        metavar='LR',
+        help='base learning rate: absolute_lr = base_lr * total_batch_size / 256',
+    )
+    parser.add_argument('--layer_decay', type=float, default=1.0)
+    parser.add_argument(
+        '--min_lr',
+        type=float,
+        default=1e-6,
+        metavar='LR',
+        help='lower lr bound for cyclic schedulers that hit 0 (1e-6)',
+    )
+    parser.add_argument(
+        '--warmup_epochs',
+        type=int,
+        default=0,
+        metavar='N',
+        help='epochs to warmup LR, if scheduler supports',
+    )
+
+    parser.add_argument(
+        '--warmup_steps',
+        type=int,
+        default=-1,
+        metavar='N',
+        help='num of steps to warmup LR, will overload warmup_epochs if set > 0',
+    )
+    parser.add_argument(
+        '--opt',
+        default='adamw',
+        type=str,
+        metavar='OPTIMIZER',
+        help='Optimizer (default: "adamw"',
+    )
+    parser.add_argument(
+        '--opt_eps',
+        default=1e-8,
+        type=float,
+        metavar='EPSILON',
+        help='Optimizer Epsilon (default: 1e-8)',
+    )
+    parser.add_argument(
+        '--opt_betas',
+        default=None,
+        type=float,
+        nargs='+',
+        metavar='BETA',
+        help='Optimizer Betas (default: None, use opt default)',
+    )
+    parser.add_argument(
+        '--momentum',
+        type=float,
+        default=0.9,
+        metavar='M',
+        help='SGD momentum (default: 0.9)',
+    )
+    parser.add_argument(
+        '--weight_decay_end',
+        type=float,
+        default=None,
+        help="""Final value of the
+        weight decay. We use a cosine schedule for WD and using a larger decay by
+        the end of training improves performance for ViTs.""",
+    )
+
+    # Augmentation parameters
+    parser.add_argument(
+        '--color_jitter',
+        type=float,
+        default=None,
+        metavar='PCT',
+        help='Color jitter factor (enabled only when not using Auto/RandAug)',
+    )
+    parser.add_argument(
+        '--aa',
+        type=str,
+        default='rand-m9-mstd0.5-inc1',
+        metavar='NAME',
+        help='Use AutoAugment policy. "v0" or "original". " + "(default: rand-m9-mstd0.5-inc1)',
+    )
+    parser.add_argument(
+        '--smoothing', type=float, default=0.1, help='Label smoothing (default: 0.1)'
+    )
+
+    parser.add_argument(
+        '--train_interpolation',
+        type=str,
+        default='bicubic',
+        help='Training interpolation (random, bilinear, bicubic default: "bicubic")',
+    )
+
+    # * Random Erase params
+    parser.add_argument(
+        '--reprob',
+        type=float,
+        default=0.25,
+        metavar='PCT',
+        help='Random erase prob (default: 0.25)',
+    )
+    parser.add_argument(
+        '--remode',
+        type=str,
+        default='pixel',
+        help='Random erase mode (default: "pixel")',
+    )
+    parser.add_argument(
+        '--recount', type=int, default=1, help='Random erase count (default: 1)'
+    )
+    parser.add_argument(
+        '--resplit',
+        type=str2bool,
+        default=False,
+        help='Do not random erase first (clean) augmentation split',
+    )
+
+    # * Mixup params
+    parser.add_argument(
+        '--mixup', type=float, default=0.0, help='mixup alpha, mixup enabled if > 0.'
+    )
+    parser.add_argument(
+        '--cutmix', type=float, default=0.0, help='cutmix alpha, cutmix enabled if > 0.'
+    )
+    parser.add_argument(
+        '--cutmix_minmax',
+        type=float,
+        nargs='+',
+        default=None,
+        help='cutmix min/max ratio, overrides alpha and enables cutmix if set (default: None)',
+    )
+    parser.add_argument(
+        '--mixup_prob',
+        type=float,
+        default=1.0,
+        help='Probability of performing mixup or cutmix when either/both is enabled',
+    )
+    parser.add_argument(
+        '--mixup_switch_prob',
+        type=float,
+        default=0.5,
+        help='Probability of switching to cutmix when both mixup and cutmix enabled',
+    )
+    parser.add_argument(
+        '--mixup_mode',
+        type=str,
+        default='batch',
+        help='How to apply mixup/cutmix params. Per "batch", "pair", or "elem"',
+    )
+
+    # * Finetuning params
+    parser.add_argument('--finetune', default='', help='finetune from checkpoint')
+    parser.add_argument(
+        '--head_init_scale',
+        default=0.001,
+        type=float,
+        help='classifier head initial scale, typically adjusted in fine-tuning',
+    )
+    parser.add_argument(
+        '--model_key',
+        default='model|module',
+        type=str,
+        help='which key to load from saved state dict, usually model or model_ema',
+    )
+    parser.add_argument('--model_prefix', default='', type=str)
+
+    # Dataset parameters
+    parser.add_argument(
+        '--data_path', default='path/dataset', type=str, help='dataset path'
+    )
+    parser.add_argument(
+        '--nb_classes', default=2, type=int, help='number of the classification types'
+    )
+    parser.add_argument(
+        '--output_dir',
+        default='./5class-checkpoints',
+        help='path where to save, empty for no saving',
+    )
+    parser.add_argument('--log_dir', default=None, help='path where to tensorboard log')
+    parser.add_argument(
+        '--device', default='cuda:0', help='device to use for training / testing'
+    )
+    parser.add_argument('--seed', default=0, type=int)
+    parser.add_argument('--resume', default='', help='resume from checkpoint')
+
+    parser.add_argument(
+        '--eval_data_path', default=None, type=str, help='dataset path for evaluation'
+    )
+    parser.add_argument('--imagenet_default_mean_and_std', type=str2bool, default=True)
+    parser.add_argument(
+        '--data_set',
+        default='IMNET',
+        choices=['CIFAR', 'IMNET', 'image_folder'],
+        type=str,
+        help='ImageNet dataset path',
+    )
+    parser.add_argument('--auto_resume', type=str2bool, default=True)
+    parser.add_argument('--save_ckpt', type=str2bool, default=True)
+    parser.add_argument('--save_ckpt_freq', default=5, type=int)
+    parser.add_argument('--save_ckpt_num', default=100, type=int)
+
+    parser.add_argument(
+        '--start_epoch', default=0, type=int, metavar='N', help='start epoch'
+    )
+    parser.add_argument(
+        '--eval', type=str2bool, default=False, help='Perform evaluation only'
+    )
+    parser.add_argument(
+        '--dist_eval',
+        type=str2bool,
+        default=True,
+        help='Enabling distributed evaluation',
+    )
+    parser.add_argument(
+        '--disable_eval',
+        type=str2bool,
+        default=False,
+        help='Disabling evaluation during training',
+    )
+    parser.add_argument('--num_workers', default=8, type=int)
+    parser.add_argument(
+        '--pin_mem',
+        type=str2bool,
+        default=True,
+        help='Pin CPU memory in DataLoader for more efficient (sometimes) transfer to GPU.',
+    )
+
+    # Evaluation parameters
+    parser.add_argument('--crop_pct', type=float, default=None)
+
+    # GPU selection parameter
+    parser.add_argument(
+        '--gpu_ids',
+        type=str,
+        default='2',
+        help='gpu ids: e.g. 0  0,1,2, 0,2. use -1 for CPU',
+    )
+
+    parser.add_argument(
+        '--use_amp',
+        type=str2bool,
+        default=False,
+        help='Use apex AMP (Automatic Mixed Precision) or not',
+    )
+    return parser
+
+
+def main(args):
+    print(args)
+
+    # Set device based on gpu_ids
+    if args.gpu_ids == '-1':
+        device = torch.device('cpu')
+    else:
+        os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu_ids
+        device = torch.device('cuda')
+
+    # Fix the seed for reproducibility
+    seed = args.seed
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    cudnn.benchmark = True
+
+    dataset_train = TrainDataset(is_train=True, args=args)
+
+    if args.disable_eval:
+        dataset_val = None
+    else:
+        dataset_val = TrainDataset(is_train=False, args=args)
+
+    sampler_train = torch.utils.data.RandomSampler(dataset_train)
+
+    if dataset_val is not None:
+        sampler_val = torch.utils.data.SequentialSampler(dataset_val)
+    else:
+        sampler_val = None
+
+    if args.log_dir is not None:
+        os.makedirs(args.log_dir, exist_ok=True)
+        log_writer = utils.TensorboardLogger(log_dir=args.log_dir)
+    else:
+        log_writer = None
+
+    data_loader_train = torch.utils.data.DataLoader(
+        dataset_train,
+        sampler=sampler_train,
+        batch_size=args.batch_size,
+        num_workers=args.num_workers,
+        pin_memory=args.pin_mem,
+        drop_last=True,
+    )
+    if dataset_val is not None:
+        data_loader_val = torch.utils.data.DataLoader(
+            dataset_val,
+            sampler=sampler_val,
+            batch_size=args.batch_size,
+            num_workers=args.num_workers,
+            pin_memory=args.pin_mem,
+            drop_last=False,
+        )
+    else:
+        data_loader_val = None
+
+    mixup_fn = None
+    mixup_active = args.mixup > 0 or args.cutmix > 0.0 or args.cutmix_minmax is not None
+    if mixup_active:
+        print('Mixup is activated!')
+        mixup_fn = Mixup(
+            mixup_alpha=args.mixup,
+            cutmix_alpha=args.cutmix,
+            cutmix_minmax=args.cutmix_minmax,
+            prob=args.mixup_prob,
+            switch_prob=args.mixup_switch_prob,
+            mode=args.mixup_mode,
+            label_smoothing=args.smoothing,
+            num_classes=args.nb_classes,
+        )
+
+    model = AIDE.__dict__[args.model](
+        resnet_path=args.resnet_path, convnext_path=args.convnext_path
+    )
+
+    model.to(device)
+
+    model_ema = None
+    if args.model_ema:
+        model_ema = ModelEma(
+            model,
+            decay=args.model_ema_decay,
+            device='cpu' if args.model_ema_force_cpu else '',
+            resume='',
+        )
+        print('Using EMA with decay = %.8f' % args.model_ema_decay)
+
+    model_without_ddp = model
+    n_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad)
+
+    print('Model = %s' % str(model_without_ddp))
+    print('number of params:', n_parameters)
+
+    eff_batch_size = args.batch_size * args.update_freq
+    num_training_steps_per_epoch = len(dataset_train) // eff_batch_size
+
+    if args.lr is None:
+        args.lr = args.blr * eff_batch_size / 256
+
+    print('base lr: %.2e' % (args.lr * 256 / eff_batch_size))
+    print('actual lr: %.2e' % args.lr)
+
+    print('accumulate grad iterations: %d' % args.update_freq)
+    print('effective batch size: %d' % eff_batch_size)
+
+    assigner = None
+
+    optimizer = create_optimizer(
+        args,
+        model_without_ddp,
+        skip_list=None,
+        get_num_layer=assigner.get_layer_id if assigner is not None else None,
+        get_layer_scale=assigner.get_scale if assigner is not None else None,
+    )
+    loss_scaler = NativeScaler()
+
+    if mixup_fn is not None:
+        criterion = SoftTargetCrossEntropy()
+    elif args.smoothing > 0.0:
+        criterion = LabelSmoothingCrossEntropy(smoothing=args.smoothing)
+    else:
+        criterion = torch.nn.CrossEntropyLoss()
+
+    print('criterion = %s' % str(criterion))
+
+    utils.auto_load_model(
+        args=args,
+        model=model,
+        model_without_ddp=model_without_ddp,
+        optimizer=optimizer,
+        loss_scaler=loss_scaler,
+        model_ema=model_ema,
+    )
+
+    if args.eval:
+        print('Eval only mode')
+
+        vals = os.listdir(args.eval_data_path)
+        eval_data_path = args.eval_data_path
+
+        rows = [
+            ['{} model testing on...'.format(args.resume)],
+            ['testset', 'accuracy', 'avg precision', 'r_acc', 'f_acc'],
+        ]
+
+        for v_id, val in enumerate(vals):
+            args.eval_data_path = os.path.join(args.eval_data_path, val)
+            dataset_val = TestDataset(is_train=False, args=args)
+            args.eval_data_path = eval_data_path
+
+            sampler_val = torch.utils.data.SequentialSampler(dataset_val)
+
+            data_loader_val = torch.utils.data.DataLoader(
+                dataset_val,
+                sampler=sampler_val,
+                batch_size=args.batch_size,
+                num_workers=args.num_workers,
+                pin_memory=args.pin_mem,
+                drop_last=False,
+            )
+
+            test_stats, acc, ap, r_acc, f_acc = evaluate(data_loader_val, model, device)
+            print(
+                f'Accuracy of the network on {len(dataset_val)} test images: {test_stats["acc1"]:.5f}%'
+            )
+
+            print(
+                f'test dataset is {val} acc: {acc}, ap: {ap}, r_acc: {r_acc}, f_acc: {f_acc}'
+            )
+            print('***********************************')
+
+            rows.append([val, acc, ap, r_acc, f_acc])
+
+        def calculate_column_means(rows):
+            if not rows or len(rows[0]) < 2:
+                raise ValueError(
+                    'The input rows list is empty or lacks numeric columns.'
+                )
+            num_columns = len(rows[0]) - 1
+            means = ['mean'] + [
+                sum(row[i] for row in rows) / len(rows)
+                for i in range(1, num_columns + 1)
+            ]
+            return means
+
+        rows.append(calculate_column_means(rows[2:]))
+        test_dataset_name = args.eval_data_path.split('/')[-2]
+
+        csv_name = os.path.join(
+            args.output_dir, f'{os.path.basename(args.resume)}_{test_dataset_name}.csv'
+        )
+        with open(csv_name, 'w') as f:
+            csv_writer = csv.writer(f, delimiter=',')
+            csv_writer.writerows(rows)
+        return
+
+    max_accuracy = 0.0
+    if args.model_ema and args.model_ema_eval:
+        max_accuracy_ema = 0.0
+
+    print('Start training for %d epochs' % args.epochs)
+    start_time = time.time()
+    for epoch in range(args.start_epoch, args.epochs):
+        if log_writer is not None:
+            log_writer.set_step(epoch * num_training_steps_per_epoch * args.update_freq)
+        train_stats = train_one_epoch(
+            model,
+            criterion,
+            data_loader_train,
+            optimizer,
+            device,
+            epoch,
+            loss_scaler,
+            args.clip_grad,
+            model_ema,
+            mixup_fn,
+            log_writer=log_writer,
+            args=args,
+        )
+        if args.output_dir and args.save_ckpt:
+            if (epoch + 1) % args.save_ckpt_freq == 0 or epoch + 1 == args.epochs:
+                utils.save_model(
+                    args=args,
+                    model=model,
+                    model_without_ddp=model_without_ddp,
+                    optimizer=optimizer,
+                    loss_scaler=loss_scaler,
+                    epoch=epoch,
+                    model_ema=model_ema,
+                )
+        if data_loader_val is not None:
+            test_stats, acc, ap, r_acc, f_acc = evaluate(
+                data_loader_val, model, device, use_amp=args.use_amp
+            )
+            print(
+                f'Accuracy of the model on the {len(dataset_val)} test images: {test_stats["acc1"]:.1f}%, ap: {ap}.'
+            )
+            if max_accuracy < test_stats['acc1']:
+                max_accuracy = test_stats['acc1']
+                if args.output_dir and args.save_ckpt:
+                    utils.save_model(
+                        args=args,
+                        model=model,
+                        model_without_ddp=model_without_ddp,
+                        optimizer=optimizer,
+                        loss_scaler=loss_scaler,
+                        epoch='best',
+                        model_ema=model_ema,
+                    )
+            print(f'Max accuracy: {max_accuracy:.2f}%')
+
+            if log_writer is not None:
+                log_writer.update(test_acc1=test_stats['acc1'], head='perf', step=epoch)
+                log_writer.update(test_acc5=test_stats['acc5'], head='perf', step=epoch)
+                log_writer.update(test_loss=test_stats['loss'], head='perf', step=epoch)
+
+            log_stats = {
+                **{f'train_{k}': v for k, v in train_stats.items()},
+                **{f'test_{k}': v for k, v in test_stats.items()},
+                'epoch': epoch,
+                'n_parameters': n_parameters,
+            }
+
+            if args.model_ema and args.model_ema_eval:
+                test_stats_ema, acc, ap, r_acc, f_acc = evaluate(
+                    data_loader_val, model_ema.ema, device, use_amp=args.use_amp
+                )
+                print(
+                    f'Accuracy of the model EMA on {len(dataset_val)} test images: {test_stats_ema["acc1"]:.1f}%, ap: {ap}'
+                )
+                if max_accuracy_ema < test_stats_ema['acc1']:
+                    max_accuracy_ema = test_stats_ema['acc1']
+                    if args.output_dir and args.save_ckpt:
+                        utils.save_model(
+                            args=args,
+                            model=model,
+                            model_without_ddp=model_without_ddp,
+                            optimizer=optimizer,
+                            loss_scaler=loss_scaler,
+                            epoch='best-ema',
+                            model_ema=model_ema,
+                        )
+                    print(f'Max EMA accuracy: {max_accuracy_ema:.2f}%')
+                if log_writer is not None:
+                    log_writer.update(
+                        test_acc1_ema=test_stats_ema['acc1'], head='perf', step=epoch
+                    )
+                log_stats.update(
+                    {**{f'test_{k}_ema': v for k, v in test_stats_ema.items()}}
+                )
+        else:
+            log_stats = {
+                **{f'train_{k}': v for k, v in train_stats.items()},
+                'epoch': epoch,
+                'n_parameters': n_parameters,
+            }
+
+        if args.output_dir:
+            if log_writer is not None:
+                log_writer.flush()
+            with open(
+                os.path.join(args.output_dir, 'log.txt'), mode='a', encoding='utf-8'
+            ) as f:
+                f.write(json.dumps(log_stats) + '\n')
+
+    total_time = time.time() - start_time
+    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+    print('Training time {}'.format(total_time_str))
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser('AIDE training', parents=[get_args_parser()])
+    args = parser.parse_args()
+    if args.output_dir:
+        Path(args.output_dir).mkdir(parents=True, exist_ok=True)
+    main(args)

detector_codes/AIDE-main/models/AIDE.py

@@ -0,0 +1,298 @@
+import torch.nn as nn
+import torch.utils.model_zoo as model_zoo
+import torch
+import clip
+import open_clip
+from .srm_filter_kernel import all_normalized_hpf_list
+import numpy as np
+
+class HPF(nn.Module):
+  def __init__(self):
+    super(HPF, self).__init__()
+
+    #Load 30 SRM Filters
+    all_hpf_list_5x5 = []
+
+    for hpf_item in all_normalized_hpf_list:
+      if hpf_item.shape[0] == 3:
+        hpf_item = np.pad(hpf_item, pad_width=((1, 1), (1, 1)), mode='constant')
+
+      all_hpf_list_5x5.append(hpf_item)
+
+    hpf_weight = torch.Tensor(all_hpf_list_5x5).view(30, 1, 5, 5).contiguous()
+    hpf_weight = torch.nn.Parameter(hpf_weight.repeat(1, 3, 1, 1), requires_grad=False)
+   
+
+    self.hpf = nn.Conv2d(3, 30, kernel_size=5, padding=2, bias=False)
+    self.hpf.weight = hpf_weight
+
+
+  def forward(self, input):
+
+    output = self.hpf(input)
+
+    return output
+
+
+
+def conv3x3(in_planes, out_planes, stride=1):
+    """3x3 convolution with padding"""
+    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
+                     padding=1, bias=False)
+
+
+def conv1x1(in_planes, out_planes, stride=1):
+    """1x1 convolution"""
+    return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)
+
+
+class BasicBlock(nn.Module):
+    expansion = 1
+
+    def __init__(self, inplanes, planes, stride=1, downsample=None):
+        super(BasicBlock, self).__init__()
+        self.conv1 = conv3x3(inplanes, planes, stride)
+        self.bn1 = nn.BatchNorm2d(planes)
+        self.relu = nn.ReLU(inplace=True)
+        self.conv2 = conv3x3(planes, planes)
+        self.bn2 = nn.BatchNorm2d(planes)
+        self.downsample = downsample
+        self.stride = stride
+
+    def forward(self, x):
+        identity = x
+
+        out = self.conv1(x)
+        out = self.bn1(out)
+        out = self.relu(out)
+
+        out = self.conv2(out)
+        out = self.bn2(out)
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu(out)
+
+        return out
+
+
+class Bottleneck(nn.Module):
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1, downsample=None):
+        super(Bottleneck, self).__init__()
+        self.conv1 = conv1x1(inplanes, planes)
+        self.bn1 = nn.BatchNorm2d(planes)
+        self.conv2 = conv3x3(planes, planes, stride)
+        self.bn2 = nn.BatchNorm2d(planes)
+        self.conv3 = conv1x1(planes, planes * self.expansion)
+        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
+        self.relu = nn.ReLU(inplace=True)
+        self.downsample = downsample
+        self.stride = stride
+
+    def forward(self, x):
+        identity = x
+
+        out = self.conv1(x)
+        out = self.bn1(out)
+        out = self.relu(out)
+
+        out = self.conv2(out)
+        out = self.bn2(out)
+        out = self.relu(out)
+
+        out = self.conv3(out)
+        out = self.bn3(out)
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu(out)
+
+        return out
+
+
+class ResNet(nn.Module):
+
+    def __init__(self, block, layers, num_classes=1000, zero_init_residual=True):
+        super(ResNet, self).__init__()
+
+        self.inplanes = 64
+        self.conv1 = nn.Conv2d(30, 64, kernel_size=7, stride=2, padding=3,
+                               bias=False)
+        self.bn1 = nn.BatchNorm2d(64)
+        self.relu = nn.ReLU(inplace=True)
+        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
+        self.layer1 = self._make_layer(block, 64, layers[0])
+        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
+        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
+        self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.fc = nn.Linear(512 * block.expansion, num_classes)
+
+        for m in self.modules():
+            if isinstance(m, nn.Conv2d):
+                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
+            elif isinstance(m, nn.BatchNorm2d):
+                nn.init.constant_(m.weight, 1)
+                nn.init.constant_(m.bias, 0)
+
+        # Zero-initialize the last BN in each residual branch,
+        # so that the residual branch starts with zeros, and each residual block behaves like an identity.
+        # This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
+        if zero_init_residual:
+            for m in self.modules():
+                if isinstance(m, Bottleneck):
+                    nn.init.constant_(m.bn3.weight, 0)
+                elif isinstance(m, BasicBlock):
+                    nn.init.constant_(m.bn2.weight, 0)
+
+    def _make_layer(self, block, planes, blocks, stride=1):
+        downsample = None
+        if stride != 1 or self.inplanes != planes * block.expansion:
+            downsample = nn.Sequential(
+                conv1x1(self.inplanes, planes * block.expansion, stride),
+                nn.BatchNorm2d(planes * block.expansion),
+            )
+
+        layers = []
+        layers.append(block(self.inplanes, planes, stride, downsample))
+        self.inplanes = planes * block.expansion
+        for _ in range(1, blocks):
+            layers.append(block(self.inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+
+        x = self.conv1(x)
+        x = self.bn1(x)
+        x = self.relu(x)
+        x = self.maxpool(x)
+
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+
+        x = self.avgpool(x)
+        x = x.view(x.size(0), -1)
+
+
+        return x
+
+class Mlp(nn.Module):
+    """ MLP as used in Vision Transformer, MLP-Mixer and related networks
+    """
+
+    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU):
+        super().__init__()
+        out_features = out_features or in_features
+        hidden_features = hidden_features or in_features
+
+        self.fc1 = nn.Linear(in_features, hidden_features)
+        self.act = act_layer()
+        self.fc2 = nn.Linear(hidden_features, out_features)
+
+    def forward(self, x):
+        x = self.fc1(x)
+        x = self.act(x)
+        x = self.fc2(x)
+        return x
+
+class AIDE_Model(nn.Module):
+
+    def __init__(self, resnet_path, convnext_path):
+        super(AIDE_Model, self).__init__()
+        self.hpf = HPF()
+        self.model_min = ResNet(Bottleneck, [3, 4, 6, 3])
+        self.model_max = ResNet(Bottleneck, [3, 4, 6, 3])
+       
+        if resnet_path is not None:
+            pretrained_dict = torch.load(resnet_path, map_location='cpu')
+        
+            model_min_dict = self.model_min.state_dict()
+            model_max_dict = self.model_max.state_dict()
+    
+            for k in pretrained_dict.keys():
+                if k in model_min_dict and pretrained_dict[k].size() == model_min_dict[k].size():
+                    model_min_dict[k] = pretrained_dict[k]
+                    model_max_dict[k] = pretrained_dict[k]
+                else:
+                    print(f"Skipping layer {k} because of size mismatch")
+        
+        self.fc = Mlp(2048 + 256 , 1024, 2)
+
+        print("build model with convnext_xxl")
+        self.openclip_convnext_xxl, _, _ = open_clip.create_model_and_transforms(
+            "convnext_xxlarge", pretrained=convnext_path
+        )
+
+        self.openclip_convnext_xxl = self.openclip_convnext_xxl.visual.trunk
+        self.openclip_convnext_xxl.head.global_pool = nn.Identity()
+        self.openclip_convnext_xxl.head.flatten = nn.Identity()
+
+        self.openclip_convnext_xxl.eval()
+        
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.convnext_proj = nn.Sequential(
+            nn.Linear(3072, 256),
+
+        )
+        for param in self.openclip_convnext_xxl.parameters():
+            param.requires_grad = False
+
+    
+
+    def forward(self, x):
+
+        b, t, c, h, w = x.shape
+
+        x_minmin = x[:, 0] #[b, c, h, w]
+        x_maxmax = x[:, 1]
+        x_minmin1 = x[:, 2]
+        x_maxmax1 = x[:, 3]
+        tokens = x[:, 4]
+
+        x_minmin = self.hpf(x_minmin)
+        x_maxmax = self.hpf(x_maxmax)
+        x_minmin1 = self.hpf(x_minmin1)
+        x_maxmax1 = self.hpf(x_maxmax1)
+
+        with torch.no_grad():
+            
+            clip_mean = torch.Tensor([0.48145466, 0.4578275, 0.40821073])
+            clip_mean = clip_mean.to(tokens, non_blocking=True).view(3, 1, 1)
+            clip_std = torch.Tensor([0.26862954, 0.26130258, 0.27577711])
+            clip_std = clip_std.to(tokens, non_blocking=True).view(3, 1, 1)
+            dinov2_mean = torch.Tensor([0.485, 0.456, 0.406]).to(tokens, non_blocking=True).view(3, 1, 1)
+            dinov2_std = torch.Tensor([0.229, 0.224, 0.225]).to(tokens, non_blocking=True).view(3, 1, 1)
+
+            local_convnext_image_feats = self.openclip_convnext_xxl(
+                tokens * (dinov2_std / clip_std) + (dinov2_mean - clip_mean) / clip_std
+            ) #[b, 3072, 8, 8]
+            assert local_convnext_image_feats.size()[1:] == (3072, 8, 8)
+            local_convnext_image_feats = self.avgpool(local_convnext_image_feats).view(tokens.size(0), -1)
+            x_0 = self.convnext_proj(local_convnext_image_feats)
+
+        x_min = self.model_min(x_minmin)
+        x_max = self.model_max(x_maxmax)
+        x_min1 = self.model_min(x_minmin1)
+        x_max1 = self.model_max(x_maxmax1)
+
+        x_1 = (x_min + x_max + x_min1 + x_max1) / 4
+
+        x = torch.cat([x_0, x_1], dim=1)
+
+        x = self.fc(x)
+
+        return x
+
+def AIDE(resnet_path, convnext_path):
+    model = AIDE_Model(resnet_path, convnext_path)
+    return model
+

detector_codes/AIDE-main/models/clip/__init__.py

@@ -0,0 +1 @@
+from .clip import *

detector_codes/AIDE-main/models/clip/bpe_simple_vocab_16e6.txt.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a
+size 1356917

detector_codes/AIDE-main/models/clip/clip.py

@@ -0,0 +1,237 @@
+import hashlib
+import os
+import urllib
+import warnings
+from typing import Any, Union, List
+from pkg_resources import packaging
+
+import torch
+from PIL import Image
+from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize
+from tqdm import tqdm
+
+from .model import build_model
+from .simple_tokenizer import SimpleTokenizer as _Tokenizer
+
+try:
+    from torchvision.transforms import InterpolationMode
+    BICUBIC = InterpolationMode.BICUBIC
+except ImportError:
+    BICUBIC = Image.BICUBIC
+
+
+if packaging.version.parse(torch.__version__) < packaging.version.parse("1.7.1"):
+    warnings.warn("PyTorch version 1.7.1 or higher is recommended")
+
+
+__all__ = ["available_models", "load", "tokenize"]
+_tokenizer = _Tokenizer()
+
+_MODELS = {
+    "RN50": "https://openaipublic.azureedge.net/clip/models/afeb0e10f9e5a86da6080e35cf09123aca3b358a0c3e3b6c78a7b63bc04b6762/RN50.pt",
+    "RN101": "https://openaipublic.azureedge.net/clip/models/8fa8567bab74a42d41c5915025a8e4538c3bdbe8804a470a72f30b0d94fab599/RN101.pt",
+    "RN50x4": "https://openaipublic.azureedge.net/clip/models/7e526bd135e493cef0776de27d5f42653e6b4c8bf9e0f653bb11773263205fdd/RN50x4.pt",
+    "RN50x16": "https://openaipublic.azureedge.net/clip/models/52378b407f34354e150460fe41077663dd5b39c54cd0bfd2b27167a4a06ec9aa/RN50x16.pt",
+    "RN50x64": "https://openaipublic.azureedge.net/clip/models/be1cfb55d75a9666199fb2206c106743da0f6468c9d327f3e0d0a543a9919d9c/RN50x64.pt",
+    "ViT-B/32": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt",
+    "ViT-B/16": "https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt",
+    "ViT-L/14": "https://openaipublic.azureedge.net/clip/models/b8cca3fd41ae0c99ba7e8951adf17d267cdb84cd88be6f7c2e0eca1737a03836/ViT-L-14.pt",
+    "ViT-L/14@336px": "https://openaipublic.azureedge.net/clip/models/3035c92b350959924f9f00213499208652fc7ea050643e8b385c2dac08641f02/ViT-L-14-336px.pt",
+}
+
+
+def _download(url: str, root: str):
+    os.makedirs(root, exist_ok=True)
+    filename = os.path.basename(url)
+
+    expected_sha256 = url.split("/")[-2]
+    download_target = os.path.join(root, filename)
+
+    if os.path.exists(download_target) and not os.path.isfile(download_target):
+        raise RuntimeError(f"{download_target} exists and is not a regular file")
+
+    if os.path.isfile(download_target):
+        if hashlib.sha256(open(download_target, "rb").read()).hexdigest() == expected_sha256:
+            return download_target
+        else:
+            warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file")
+
+    with urllib.request.urlopen(url) as source, open(download_target, "wb") as output:
+        with tqdm(total=int(source.info().get("Content-Length")), ncols=80, unit='iB', unit_scale=True, unit_divisor=1024) as loop:
+            while True:
+                buffer = source.read(8192)
+                if not buffer:
+                    break
+
+                output.write(buffer)
+                loop.update(len(buffer))
+
+    if hashlib.sha256(open(download_target, "rb").read()).hexdigest() != expected_sha256:
+        raise RuntimeError("Model has been downloaded but the SHA256 checksum does not not match")
+
+    return download_target
+
+
+def _convert_image_to_rgb(image):
+    return image.convert("RGB")
+
+
+def _transform(n_px):
+    return Compose([
+        Resize(n_px, interpolation=BICUBIC),
+        CenterCrop(n_px),
+        _convert_image_to_rgb,
+        ToTensor(),
+        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
+    ])
+
+
+def available_models() -> List[str]:
+    """Returns the names of available CLIP models"""
+    return list(_MODELS.keys())
+
+
+def load(name: str, device: Union[str, torch.device] = "cuda" if torch.cuda.is_available() else "cpu", jit: bool = False, download_root: str = None):
+    """Load a CLIP model
+
+    Parameters
+    ----------
+    name : str
+        A model name listed by `clip.available_models()`, or the path to a model checkpoint containing the state_dict
+
+    device : Union[str, torch.device]
+        The device to put the loaded model
+
+    jit : bool
+        Whether to load the optimized JIT model or more hackable non-JIT model (default).
+
+    download_root: str
+        path to download the model files; by default, it uses "~/.cache/clip"
+
+    Returns
+    -------
+    model : torch.nn.Module
+        The CLIP model
+
+    preprocess : Callable[[PIL.Image], torch.Tensor]
+        A torchvision transform that converts a PIL image into a tensor that the returned model can take as its input
+    """
+    if name in _MODELS:
+        model_path = _download(_MODELS[name], download_root or os.path.expanduser("~/.cache/clip"))
+    elif os.path.isfile(name):
+        model_path = name
+    else:
+        raise RuntimeError(f"Model {name} not found; available models = {available_models()}")
+
+    with open(model_path, 'rb') as opened_file:
+        try:
+            # loading JIT archive
+            model = torch.jit.load(opened_file, map_location=device if jit else "cpu").eval()
+            state_dict = None
+        except RuntimeError:
+            # loading saved state dict
+            if jit:
+                warnings.warn(f"File {model_path} is not a JIT archive. Loading as a state dict instead")
+                jit = False
+            state_dict = torch.load(opened_file, map_location="cpu")
+
+    if not jit:
+        model = build_model(state_dict or model.state_dict()).to(device)
+        if str(device) == "cpu":
+            model.float()
+        return model, _transform(model.visual.input_resolution)
+
+    # patch the device names
+    device_holder = torch.jit.trace(lambda: torch.ones([]).to(torch.device(device)), example_inputs=[])
+    device_node = [n for n in device_holder.graph.findAllNodes("prim::Constant") if "Device" in repr(n)][-1]
+
+    def patch_device(module):
+        try:
+            graphs = [module.graph] if hasattr(module, "graph") else []
+        except RuntimeError:
+            graphs = []
+
+        if hasattr(module, "forward1"):
+            graphs.append(module.forward1.graph)
+
+        for graph in graphs:
+            for node in graph.findAllNodes("prim::Constant"):
+                if "value" in node.attributeNames() and str(node["value"]).startswith("cuda"):
+                    node.copyAttributes(device_node)
+
+    model.apply(patch_device)
+    patch_device(model.encode_image)
+    patch_device(model.encode_text)
+
+    # patch dtype to float32 on CPU
+    if str(device) == "cpu":
+        float_holder = torch.jit.trace(lambda: torch.ones([]).float(), example_inputs=[])
+        float_input = list(float_holder.graph.findNode("aten::to").inputs())[1]
+        float_node = float_input.node()
+
+        def patch_float(module):
+            try:
+                graphs = [module.graph] if hasattr(module, "graph") else []
+            except RuntimeError:
+                graphs = []
+
+            if hasattr(module, "forward1"):
+                graphs.append(module.forward1.graph)
+
+            for graph in graphs:
+                for node in graph.findAllNodes("aten::to"):
+                    inputs = list(node.inputs())
+                    for i in [1, 2]:  # dtype can be the second or third argument to aten::to()
+                        if inputs[i].node()["value"] == 5:
+                            inputs[i].node().copyAttributes(float_node)
+
+        model.apply(patch_float)
+        patch_float(model.encode_image)
+        patch_float(model.encode_text)
+
+        model.float()
+
+    return model, _transform(model.input_resolution.item())
+
+
+def tokenize(texts: Union[str, List[str]], context_length: int = 77, truncate: bool = False) -> Union[torch.IntTensor, torch.LongTensor]:
+    """
+    Returns the tokenized representation of given input string(s)
+
+    Parameters
+    ----------
+    texts : Union[str, List[str]]
+        An input string or a list of input strings to tokenize
+
+    context_length : int
+        The context length to use; all CLIP models use 77 as the context length
+
+    truncate: bool
+        Whether to truncate the text in case its encoding is longer than the context length
+
+    Returns
+    -------
+    A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length].
+    We return LongTensor when torch version is <1.8.0, since older index_select requires indices to be long.
+    """
+    if isinstance(texts, str):
+        texts = [texts]
+
+    sot_token = _tokenizer.encoder["<|startoftext|>"]
+    eot_token = _tokenizer.encoder["<|endoftext|>"]
+    all_tokens = [[sot_token] + _tokenizer.encode(text) + [eot_token] for text in texts]
+    if packaging.version.parse(torch.__version__) < packaging.version.parse("1.8.0"):
+        result = torch.zeros(len(all_tokens), context_length, dtype=torch.long)
+    else:
+        result = torch.zeros(len(all_tokens), context_length, dtype=torch.int)
+
+    for i, tokens in enumerate(all_tokens):
+        if len(tokens) > context_length:
+            if truncate:
+                tokens = tokens[:context_length]
+                tokens[-1] = eot_token
+            else:
+                raise RuntimeError(f"Input {texts[i]} is too long for context length {context_length}")
+        result[i, :len(tokens)] = torch.tensor(tokens)
+
+    return result

detector_codes/AIDE-main/models/clip/lora_clip.py

@@ -0,0 +1,151 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from models.clip import clip
+import math
+import copy
+from collections import OrderedDict
+import numpy as np
+from torch.utils.data import Dataset, DataLoader
+from PIL import Image
+import os
+
+
+class LoRALayer(nn.Module):
+    """
+    LoRA层的实现：添加低秩矩阵以高效微调大型模型
+    """
+
+    def __init__(self, in_dim, out_dim, rank=4, alpha=1.0):
+        super().__init__()
+        # 缩放系数，决定了LoRA部分的权重
+        self.scale = alpha / rank
+
+        # 创建LoRA的低秩矩阵A和B
+        self.lora_down = nn.Linear(in_dim, rank, bias=False)
+        self.lora_up = nn.Linear(rank, out_dim, bias=False)
+
+        # 初始化：下投影随机初始化，上投影初始化为0
+        nn.init.kaiming_uniform_(self.lora_down.weight, a=math.sqrt(5))
+        nn.init.zeros_(self.lora_up.weight)
+
+    def forward(self, x):
+        # 计算LoRA的贡献并返回
+        return self.lora_up(self.lora_down(x)) * self.scale
+
+
+# ---------------------- 2. CLIP的LoRA包装器 ----------------------
+
+class CLIPWithLoRA(nn.Module):
+    """
+    为CLIP的图像编码器添加LoRA适配层的完整实现
+    """
+
+    def __init__(self, base_clip_model, lora_rank=8, lora_alpha=16):
+        super().__init__()
+        # 保存原始的CLIP模型
+        self.clip = base_clip_model
+
+        # 冻结原始CLIP的所有参数
+        for param in self.clip.parameters():
+            param.requires_grad = False
+
+        # 分析CLIP的图像编码器结构
+        # 对于ViT-L/14，每个Transformer块有三个投影矩阵
+        # 我们将为每个投影矩阵添加LoRA层
+        self.image_encoder = self.clip.visual
+
+        # 检查并确定这是一个Vision Transformer结构
+        assert hasattr(self.image_encoder, 'transformer'), "不支持的CLIP视觉编码器类型"
+
+        # 创建LoRA层字典
+        self.lora_layers = nn.ModuleDict()
+
+        # 获取隐藏层维度（对于ViT-L/14，通常是1024）
+        if hasattr(self.image_encoder, 'width'):
+            hidden_dim = self.image_encoder.width
+        else:
+            hidden_dim = self.image_encoder.transformer.width
+
+        # 为每个Transformer块的注意力层添加LoRA
+        for block_idx, block in enumerate(self.image_encoder.transformer.resblocks):
+            # 为QKV矩阵添加LoRA
+            # 注意：根据OpenAI的CLIP实现，通常QKV是在一个单一的投影中
+            attn = block.attn
+
+            # 添加LoRA到Query投影
+            self.lora_layers[f"block_{block_idx}_q"] = LoRALayer(
+                hidden_dim, hidden_dim,
+                rank=lora_rank, alpha=lora_alpha
+            )
+
+            # 添加LoRA到Key投影
+            self.lora_layers[f"block_{block_idx}_k"] = LoRALayer(
+                hidden_dim, hidden_dim,
+                rank=lora_rank, alpha=lora_alpha
+            )
+
+            # 添加LoRA到Value投影
+            self.lora_layers[f"block_{block_idx}_v"] = LoRALayer(
+                hidden_dim, hidden_dim,
+                rank=lora_rank, alpha=lora_alpha
+            )
+
+            # 修改注意力的前向传播方法
+            self._patch_attention_forward(block.attn, block_idx)
+
+    def _patch_attention_forward(self, attn_module, block_idx):
+        """
+        通过钩子技术修改注意力层的前向传播，插入LoRA计算
+        """
+        # 保存原始的前向传播方法
+        original_forward = attn_module.forward
+
+        # 保存self引用以在闭包中使用
+        _self = self
+
+        # 定义新的前向传播方法
+        def lora_forward(self, query, key, value, need_weights=False, **kwargs):
+            # 获取模型尺寸
+            B, N, C = query.shape
+
+            # 原始的QKV投影
+            original_qkv = self.in_proj_weight
+
+            # 将输入分成三部分用于q、k、v计算
+            qkv = F.linear(query, original_qkv, self.in_proj_bias)
+            qkv = qkv.reshape(B, N, 3, C).permute(2, 0, 1, 3)  # [3, B, N, C]
+            q, k, v = qkv[0], qkv[1], qkv[2]
+
+            # 应用LoRA修改到q、k、v
+            q = q + _self.lora_layers[f"block_{block_idx}_q"](query)
+            k = k + _self.lora_layers[f"block_{block_idx}_k"](query)
+            v = v + _self.lora_layers[f"block_{block_idx}_v"](query)
+
+            # 继续原始注意力计算
+            self.scale = q.size(-1) ** -0.5
+            q = q * self.scale
+            attn = (q @ k.transpose(-2, -1))
+            attn = attn.softmax(dim=-1)
+            # attn = self.attn_dropout(attn)
+
+            x = (attn @ v).transpose(1, 2).reshape(B, N, C)
+            x = self.out_proj(x)
+            # x = self.out_dropout(x)
+
+            if need_weights:
+                return x, attn
+            return x
+
+        # 替换原始的前向传播方法
+        attn_module.forward = lora_forward.__get__(attn_module, type(attn_module))
+
+    def forward(self, image):
+        """
+        模型的前向传播，使用LoRA修改的CLIP进行图像编码
+        """
+        # 直接使用修改后的CLIP模型
+        image_features = self.clip.encode_image(image)
+
+        # 返回归一化的图像特征
+        return image_features

detector_codes/AIDE-main/models/clip/model.py

@@ -0,0 +1,452 @@
+from collections import OrderedDict
+from typing import Tuple, Union
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+class Bottleneck(nn.Module):
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1):
+        super().__init__()
+
+        # all conv layers have stride 1. an avgpool is performed after the second convolution when stride > 1
+        self.conv1 = nn.Conv2d(inplanes, planes, 1, bias=False)
+        self.bn1 = nn.BatchNorm2d(planes)
+        self.relu1 = nn.ReLU(inplace=True)
+
+        self.conv2 = nn.Conv2d(planes, planes, 3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(planes)
+        self.relu2 = nn.ReLU(inplace=True)
+
+        self.avgpool = nn.AvgPool2d(stride) if stride > 1 else nn.Identity()
+
+        self.conv3 = nn.Conv2d(planes, planes * self.expansion, 1, bias=False)
+        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
+        self.relu3 = nn.ReLU(inplace=True)
+
+        self.downsample = None
+        self.stride = stride
+
+        if stride > 1 or inplanes != planes * Bottleneck.expansion:
+            # downsampling layer is prepended with an avgpool, and the subsequent convolution has stride 1
+            self.downsample = nn.Sequential(OrderedDict([
+                ("-1", nn.AvgPool2d(stride)),
+                ("0", nn.Conv2d(inplanes, planes * self.expansion, 1, stride=1, bias=False)),
+                ("1", nn.BatchNorm2d(planes * self.expansion))
+            ]))
+
+    def forward(self, x: torch.Tensor):
+        identity = x
+
+        out = self.relu1(self.bn1(self.conv1(x)))
+        out = self.relu2(self.bn2(self.conv2(out)))
+        out = self.avgpool(out)
+        out = self.bn3(self.conv3(out))
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu3(out)
+        return out
+
+
+class AttentionPool2d(nn.Module):
+    def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None):
+        super().__init__()
+        self.positional_embedding = nn.Parameter(torch.randn(spacial_dim ** 2 + 1, embed_dim) / embed_dim ** 0.5)
+        self.k_proj = nn.Linear(embed_dim, embed_dim)
+        self.q_proj = nn.Linear(embed_dim, embed_dim)
+        self.v_proj = nn.Linear(embed_dim, embed_dim)
+        self.c_proj = nn.Linear(embed_dim, output_dim or embed_dim)
+        self.num_heads = num_heads
+
+    def forward(self, x):
+        x = x.flatten(start_dim=2).permute(2, 0, 1)  # NCHW -> (HW)NC
+        x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0)  # (HW+1)NC
+        x = x + self.positional_embedding[:, None, :].to(x.dtype)  # (HW+1)NC
+        x, _ = F.multi_head_attention_forward(
+            query=x[:1], key=x, value=x,
+            embed_dim_to_check=x.shape[-1],
+            num_heads=self.num_heads,
+            q_proj_weight=self.q_proj.weight,
+            k_proj_weight=self.k_proj.weight,
+            v_proj_weight=self.v_proj.weight,
+            in_proj_weight=None,
+            in_proj_bias=torch.cat([self.q_proj.bias, self.k_proj.bias, self.v_proj.bias]),
+            bias_k=None,
+            bias_v=None,
+            add_zero_attn=False,
+            dropout_p=0,
+            out_proj_weight=self.c_proj.weight,
+            out_proj_bias=self.c_proj.bias,
+            use_separate_proj_weight=True,
+            training=self.training,
+            need_weights=False
+        )
+        return x.squeeze(0)
+
+
+class ModifiedResNet(nn.Module):
+    """
+    A ResNet class that is similar to torchvision's but contains the following changes:
+    - There are now 3 "stem" convolutions as opposed to 1, with an average pool instead of a max pool.
+    - Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1
+    - The final pooling layer is a QKV attention instead of an average pool
+    """
+
+    def __init__(self, layers, output_dim, heads, input_resolution=224, width=64):
+        super().__init__()
+        self.output_dim = output_dim
+        self.input_resolution = input_resolution
+
+        # the 3-layer stem
+        self.conv1 = nn.Conv2d(3, width // 2, kernel_size=3, stride=2, padding=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(width // 2)
+        self.relu1 = nn.ReLU(inplace=True)
+        self.conv2 = nn.Conv2d(width // 2, width // 2, kernel_size=3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(width // 2)
+        self.relu2 = nn.ReLU(inplace=True)
+        self.conv3 = nn.Conv2d(width // 2, width, kernel_size=3, padding=1, bias=False)
+        self.bn3 = nn.BatchNorm2d(width)
+        self.relu3 = nn.ReLU(inplace=True)
+        self.avgpool = nn.AvgPool2d(2)
+
+        # residual layers
+        self._inplanes = width  # this is a *mutable* variable used during construction
+        self.layer1 = self._make_layer(width, layers[0])
+        self.layer2 = self._make_layer(width * 2, layers[1], stride=2)
+        self.layer3 = self._make_layer(width * 4, layers[2], stride=2)
+        self.layer4 = self._make_layer(width * 8, layers[3], stride=2)
+
+        embed_dim = width * 32  # the ResNet feature dimension
+        self.attnpool = AttentionPool2d(input_resolution // 32, embed_dim, heads, output_dim)
+
+    def _make_layer(self, planes, blocks, stride=1):
+        layers = [Bottleneck(self._inplanes, planes, stride)]
+
+        self._inplanes = planes * Bottleneck.expansion
+        for _ in range(1, blocks):
+            layers.append(Bottleneck(self._inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        def stem(x):
+            x = self.relu1(self.bn1(self.conv1(x)))
+            x = self.relu2(self.bn2(self.conv2(x)))
+            x = self.relu3(self.bn3(self.conv3(x)))
+            x = self.avgpool(x)
+            return x
+
+        x = x.type(self.conv1.weight.dtype)
+        x = stem(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.attnpool(x)
+
+        return x
+
+
+class LayerNorm(nn.LayerNorm):
+    """Subclass torch's LayerNorm to handle fp16."""
+
+    def forward(self, x: torch.Tensor):
+        orig_type = x.dtype
+        ret = super().forward(x.type(torch.float32))
+        return ret.type(orig_type)
+
+
+class QuickGELU(nn.Module):
+    def forward(self, x: torch.Tensor):
+        return x * torch.sigmoid(1.702 * x)
+
+
+class ResidualAttentionBlock(nn.Module):
+    def __init__(self, d_model: int, n_head: int, attn_mask: torch.Tensor = None):
+        super().__init__()
+
+        self.attn = nn.MultiheadAttention(d_model, n_head)
+        self.ln_1 = LayerNorm(d_model)
+        self.mlp = nn.Sequential(OrderedDict([
+            ("c_fc", nn.Linear(d_model, d_model * 4)),
+            ("gelu", QuickGELU()),
+            ("c_proj", nn.Linear(d_model * 4, d_model))
+        ]))
+        self.ln_2 = LayerNorm(d_model)
+        self.attn_mask = attn_mask
+
+    def attention(self, x: torch.Tensor):
+        self.attn_mask = self.attn_mask.to(dtype=x.dtype, device=x.device) if self.attn_mask is not None else None
+        return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0]
+
+    def forward(self, x: torch.Tensor):
+        x = x + self.attention(self.ln_1(x))
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class Transformer(nn.Module):
+    def __init__(self, width: int, layers: int, heads: int, attn_mask: torch.Tensor = None):
+        super().__init__()
+        self.width = width
+        self.layers = layers
+        self.resblocks = nn.Sequential(*[ResidualAttentionBlock(width, heads, attn_mask) for _ in range(layers)])
+
+    def forward(self, x: torch.Tensor):
+        out = {}
+        for idx, layer in enumerate(self.resblocks.children()):
+            x = layer(x)
+            out['layer'+str(idx)] = x[0] # shape:LND. choose cls token feature
+        return out, x 
+
+        # return self.resblocks(x)  # This is the original code 
+
+
+class VisionTransformer(nn.Module):
+    def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int):
+        super().__init__()
+        self.input_resolution = input_resolution
+        self.output_dim = output_dim
+        self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)
+
+        scale = width ** -0.5
+        self.class_embedding = nn.Parameter(scale * torch.randn(width))
+        self.positional_embedding = nn.Parameter(scale * torch.randn((input_resolution // patch_size) ** 2 + 1, width))
+        self.ln_pre = LayerNorm(width)
+
+        self.transformer = Transformer(width, layers, heads)
+
+        self.ln_post = LayerNorm(width)
+        self.proj = nn.Parameter(scale * torch.randn(width, output_dim))
+
+
+
+    def forward(self, x: torch.Tensor):
+        x = self.conv1(x)  # shape = [*, width, grid, grid]
+        x = x.reshape(x.shape[0], x.shape[1], -1)  # shape = [*, width, grid ** 2]
+        x = x.permute(0, 2, 1)  # shape = [*, grid ** 2, width]
+        x = torch.cat([self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device), x], dim=1)  # shape = [*, grid ** 2 + 1, width]
+        x = x + self.positional_embedding.to(x.dtype)
+        x = self.ln_pre(x)
+
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        out, x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+
+        x = self.ln_post(x[:, 0, :])
+
+
+        out['before_projection'] = x  
+
+        if self.proj is not None:
+            x = x @ self.proj
+        out['after_projection'] = x 
+
+        # Return both intermediate features and final clip feature 
+        # return out
+        
+        # This only returns CLIP features 
+        return x 
+
+
+class CLIP(nn.Module):
+    def __init__(self,
+                 embed_dim: int,
+                 # vision
+                 image_resolution: int,
+                 vision_layers: Union[Tuple[int, int, int, int], int],
+                 vision_width: int,
+                 vision_patch_size: int,
+                 # text
+                 context_length: int,
+                 vocab_size: int,
+                 transformer_width: int,
+                 transformer_heads: int,
+                 transformer_layers: int
+                 ):
+        super().__init__()
+
+        self.context_length = context_length
+
+        if isinstance(vision_layers, (tuple, list)):
+            vision_heads = vision_width * 32 // 64
+            self.visual = ModifiedResNet(
+                layers=vision_layers,
+                output_dim=embed_dim,
+                heads=vision_heads,
+                input_resolution=image_resolution,
+                width=vision_width
+            )
+        else:
+            vision_heads = vision_width // 64
+            self.visual = VisionTransformer(
+                input_resolution=image_resolution,
+                patch_size=vision_patch_size,
+                width=vision_width,
+                layers=vision_layers,
+                heads=vision_heads,
+                output_dim=embed_dim
+            )
+
+        self.transformer = Transformer(
+            width=transformer_width,
+            layers=transformer_layers,
+            heads=transformer_heads,
+            attn_mask=self.build_attention_mask()
+        )
+
+        self.vocab_size = vocab_size
+        self.token_embedding = nn.Embedding(vocab_size, transformer_width)
+        self.positional_embedding = nn.Parameter(torch.empty(self.context_length, transformer_width))
+        self.ln_final = LayerNorm(transformer_width)
+
+        self.text_projection = nn.Parameter(torch.empty(transformer_width, embed_dim))
+        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))
+
+        self.initialize_parameters()
+
+    def initialize_parameters(self):
+        nn.init.normal_(self.token_embedding.weight, std=0.02)
+        nn.init.normal_(self.positional_embedding, std=0.01)
+
+        if isinstance(self.visual, ModifiedResNet):
+            if self.visual.attnpool is not None:
+                std = self.visual.attnpool.c_proj.in_features ** -0.5
+                nn.init.normal_(self.visual.attnpool.q_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.k_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.v_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.c_proj.weight, std=std)
+
+            for resnet_block in [self.visual.layer1, self.visual.layer2, self.visual.layer3, self.visual.layer4]:
+                for name, param in resnet_block.named_parameters():
+                    if name.endswith("bn3.weight"):
+                        nn.init.zeros_(param)
+
+        proj_std = (self.transformer.width ** -0.5) * ((2 * self.transformer.layers) ** -0.5)
+        attn_std = self.transformer.width ** -0.5
+        fc_std = (2 * self.transformer.width) ** -0.5
+        for block in self.transformer.resblocks:
+            nn.init.normal_(block.attn.in_proj_weight, std=attn_std)
+            nn.init.normal_(block.attn.out_proj.weight, std=proj_std)
+            nn.init.normal_(block.mlp.c_fc.weight, std=fc_std)
+            nn.init.normal_(block.mlp.c_proj.weight, std=proj_std)
+
+        if self.text_projection is not None:
+            nn.init.normal_(self.text_projection, std=self.transformer.width ** -0.5)
+
+    def build_attention_mask(self):
+        # lazily create causal attention mask, with full attention between the vision tokens
+        # pytorch uses additive attention mask; fill with -inf
+        mask = torch.empty(self.context_length, self.context_length)
+        mask.fill_(float("-inf"))
+        mask.triu_(1)  # zero out the lower diagonal
+        return mask
+
+    @property
+    def dtype(self):
+        return self.visual.conv1.weight.dtype
+
+    def encode_image(self, image):
+        return self.visual(image.type(self.dtype))
+
+    def encode_text(self, text):
+        x = self.token_embedding(text).type(self.dtype)  # [batch_size, n_ctx, d_model]
+
+        x = x + self.positional_embedding.type(self.dtype)
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+        x = self.ln_final(x).type(self.dtype)
+
+        # x.shape = [batch_size, n_ctx, transformer.width]
+        # take features from the eot embedding (eot_token is the highest number in each sequence)
+        x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection
+
+        return x
+
+    def forward(self, image, text):
+        image_features = self.encode_image(image)
+        text_features = self.encode_text(text)
+
+        # normalized features
+        image_features = image_features / image_features.norm(dim=1, keepdim=True)
+        text_features = text_features / text_features.norm(dim=1, keepdim=True)
+
+        # cosine similarity as logits
+        logit_scale = self.logit_scale.exp()
+        logits_per_image = logit_scale * image_features @ text_features.t()
+        logits_per_text = logits_per_image.t()
+
+        # shape = [global_batch_size, global_batch_size]
+        return logits_per_image, logits_per_text
+
+
+def convert_weights(model: nn.Module):
+    """Convert applicable model parameters to fp16"""
+
+    def _convert_weights_to_fp16(l):
+        if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Linear)):
+            l.weight.data = l.weight.data.half()
+            if l.bias is not None:
+                l.bias.data = l.bias.data.half()
+
+        if isinstance(l, nn.MultiheadAttention):
+            for attr in [*[f"{s}_proj_weight" for s in ["in", "q", "k", "v"]], "in_proj_bias", "bias_k", "bias_v"]:
+                tensor = getattr(l, attr)
+                if tensor is not None:
+                    tensor.data = tensor.data.half()
+
+        for name in ["text_projection", "proj"]:
+            if hasattr(l, name):
+                attr = getattr(l, name)
+                if attr is not None:
+                    attr.data = attr.data.half()
+
+    model.apply(_convert_weights_to_fp16)
+
+
+def build_model(state_dict: dict):
+    vit = "visual.proj" in state_dict
+
+    if vit:
+        vision_width = state_dict["visual.conv1.weight"].shape[0]
+        vision_layers = len([k for k in state_dict.keys() if k.startswith("visual.") and k.endswith(".attn.in_proj_weight")])
+        vision_patch_size = state_dict["visual.conv1.weight"].shape[-1]
+        grid_size = round((state_dict["visual.positional_embedding"].shape[0] - 1) ** 0.5)
+        image_resolution = vision_patch_size * grid_size
+    else:
+        counts: list = [len(set(k.split(".")[2] for k in state_dict if k.startswith(f"visual.layer{b}"))) for b in [1, 2, 3, 4]]
+        vision_layers = tuple(counts)
+        vision_width = state_dict["visual.layer1.0.conv1.weight"].shape[0]
+        output_width = round((state_dict["visual.attnpool.positional_embedding"].shape[0] - 1) ** 0.5)
+        vision_patch_size = None
+        assert output_width ** 2 + 1 == state_dict["visual.attnpool.positional_embedding"].shape[0]
+        image_resolution = output_width * 32
+
+    embed_dim = state_dict["text_projection"].shape[1]
+    context_length = state_dict["positional_embedding"].shape[0]
+    vocab_size = state_dict["token_embedding.weight"].shape[0]
+    transformer_width = state_dict["ln_final.weight"].shape[0]
+    transformer_heads = transformer_width // 64
+    transformer_layers = len(set(k.split(".")[2] for k in state_dict if k.startswith("transformer.resblocks")))
+
+    model = CLIP(
+        embed_dim,
+        image_resolution, vision_layers, vision_width, vision_patch_size,
+        context_length, vocab_size, transformer_width, transformer_heads, transformer_layers
+    )
+
+    for key in ["input_resolution", "context_length", "vocab_size"]:
+        if key in state_dict:
+            del state_dict[key]
+
+    convert_weights(model)
+    model.load_state_dict(state_dict)
+    return model.eval()

detector_codes/AIDE-main/models/clip/simple_tokenizer.py

@@ -0,0 +1,132 @@
+import gzip
+import html
+import os
+from functools import lru_cache
+
+import ftfy
+import regex as re
+
+
+@lru_cache()
+def default_bpe():
+    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "bpe_simple_vocab_16e6.txt.gz")
+
+
+@lru_cache()
+def bytes_to_unicode():
+    """
+    Returns list of utf-8 byte and a corresponding list of unicode strings.
+    The reversible bpe codes work on unicode strings.
+    This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
+    When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
+    This is a signficant percentage of your normal, say, 32K bpe vocab.
+    To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
+    And avoids mapping to whitespace/control characters the bpe code barfs on.
+    """
+    bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1))
+    cs = bs[:]
+    n = 0
+    for b in range(2**8):
+        if b not in bs:
+            bs.append(b)
+            cs.append(2**8+n)
+            n += 1
+    cs = [chr(n) for n in cs]
+    return dict(zip(bs, cs))
+
+
+def get_pairs(word):
+    """Return set of symbol pairs in a word.
+    Word is represented as tuple of symbols (symbols being variable-length strings).
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+    return pairs
+
+
+def basic_clean(text):
+    text = ftfy.fix_text(text)
+    text = html.unescape(html.unescape(text))
+    return text.strip()
+
+
+def whitespace_clean(text):
+    text = re.sub(r'\s+', ' ', text)
+    text = text.strip()
+    return text
+
+
+class SimpleTokenizer(object):
+    def __init__(self, bpe_path: str = default_bpe()):
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+        merges = gzip.open(bpe_path).read().decode("utf-8").split('\n')
+        merges = merges[1:49152-256-2+1]
+        merges = [tuple(merge.split()) for merge in merges]
+        vocab = list(bytes_to_unicode().values())
+        vocab = vocab + [v+'</w>' for v in vocab]
+        for merge in merges:
+            vocab.append(''.join(merge))
+        vocab.extend(['<|startoftext|>', '<|endoftext|>'])
+        self.encoder = dict(zip(vocab, range(len(vocab))))
+        self.decoder = {v: k for k, v in self.encoder.items()}
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {'<|startoftext|>': '<|startoftext|>', '<|endoftext|>': '<|endoftext|>'}
+        self.pat = re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE)
+
+    def bpe(self, token):
+        if token in self.cache:
+            return self.cache[token]
+        word = tuple(token[:-1]) + ( token[-1] + '</w>',)
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token+'</w>'
+
+        while True:
+            bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf')))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                    new_word.extend(word[i:j])
+                    i = j
+                except:
+                    new_word.extend(word[i:])
+                    break
+
+                if word[i] == first and i < len(word)-1 and word[i+1] == second:
+                    new_word.append(first+second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = ' '.join(word)
+        self.cache[token] = word
+        return word
+
+    def encode(self, text):
+        bpe_tokens = []
+        text = whitespace_clean(basic_clean(text)).lower()
+        for token in re.findall(self.pat, text):
+            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
+            bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
+        return bpe_tokens
+
+    def decode(self, tokens):
+        text = ''.join([self.decoder[token] for token in tokens])
+        text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors="replace").replace('</w>', ' ')
+        return text

detector_codes/AIDE-main/models/srm_filter_kernel.py

@@ -0,0 +1,220 @@
+
+import numpy as np
+
+filter_class_1 = [
+  np.array([
+    [1, 0, 0],
+    [0, -1, 0],
+    [0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 1, 0],
+    [0, -1, 0],
+    [0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 1],
+    [0, -1, 0],
+    [0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [1, -1, 0],
+    [0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [0, -1, 1],
+    [0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [0, -1, 0],
+    [1, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [0, -1, 0],
+    [0, 1, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [0, -1, 0],
+    [0, 0, 1]
+  ], dtype=np.float32)
+]
+
+
+filter_class_2 = [
+  np.array([
+    [1, 0, 0],
+    [0, -2, 0],
+    [0, 0, 1]
+  ], dtype=np.float32),
+  np.array([
+    [0, 1, 0],
+    [0, -2, 0],
+    [0, 1, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 1],
+    [0, -2, 0],
+    [1, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [1, -2, 1],
+    [0, 0, 0]
+  ], dtype=np.float32),
+]
+
+
+filter_class_3 = [
+  np.array([
+    [-1, 0, 0, 0, 0],
+    [0, 3, 0, 0, 0],
+    [0, 0, -3, 0, 0],
+    [0, 0, 0, 1, 0],
+    [0, 0, 0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, -1, 0, 0],
+    [0, 0, 3, 0, 0],
+    [0, 0, -3, 0, 0],
+    [0, 0, 1, 0, 0],
+    [0, 0, 0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, -1],
+    [0, 0, 0, 3, 0],
+    [0, 0, -3, 0, 0],
+    [0, 1, 0, 0, 0],
+    [0, 0, 0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 0, 0],
+    [0, 1, -3, 3, -1],
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, 0],
+    [0, 1, 0, 0, 0],
+    [0, 0, -3, 0, 0],
+    [0, 0, 0, 3, 0],
+    [0, 0, 0, 0, -1]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, 0],
+    [0, 0, 1, 0, 0],
+    [0, 0, -3, 0, 0],
+    [0, 0, 3, 0, 0],
+    [0, 0, -1, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 1, 0],
+    [0, 0, -3, 0, 0],
+    [0, 3, 0, 0, 0],
+    [-1, 0, 0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 0, 0],
+    [-1, 3, -3, 1, 0],
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 0, 0]
+  ], dtype=np.float32)
+]
+
+
+filter_edge_3x3 = [
+  np.array([
+    [-1, 2, -1],
+    [2, -4, 2],
+    [0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 2, -1],
+    [0, -4, 2],
+    [0, 2, -1]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0],
+    [2, -4, 2],
+    [-1, 2, -1]
+  ], dtype=np.float32),
+  np.array([
+    [-1, 2, 0],
+    [2, -4, 0],
+    [-1, 2, 0]
+  ], dtype=np.float32),
+]
+
+filter_edge_5x5 = [
+  np.array([
+    [-1, 2, -2, 2, -1],
+    [2, -6, 8, -6, 2],
+    [-2, 8, -12, 8, -2],
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 0, 0]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, -2, 2, -1],
+    [0, 0, 8, -6, 2],
+    [0, 0, -12, 8, -2],
+    [0, 0, 8, -6, 2],
+    [0, 0, -2, 2, -1]
+  ], dtype=np.float32),
+  np.array([
+    [0, 0, 0, 0, 0],
+    [0, 0, 0, 0, 0],
+    [-2, 8, -12, 8, -2],
+    [2, -6, 8, -6, 2],
+    [-1, 2, -2, 2, -1]
+  ], dtype=np.float32),
+  np.array([
+    [-1, 2, -2, 0, 0],
+    [2, -6, 8, 0, 0],
+    [-2, 8, -12, 0, 0],
+    [2, -6, 8, 0, 0],
+    [-1, 2, -2, 0, 0]
+  ], dtype=np.float32),
+]
+
+square_3x3 = np.array([
+  [-1, 2, -1],
+  [2, -4, 2],
+  [-1, 2, -1]
+], dtype=np.float32)
+
+square_5x5 = np.array([
+  [-1, 2, -2, 2, -1],
+  [2, -6, 8, -6, 2],
+  [-2, 8, -12, 8, -2],
+  [2, -6, 8, -6, 2],
+  [-1, 2, -2, 2, -1]
+], dtype=np.float32)
+
+
+all_hpf_list = filter_class_1 + filter_class_2 + filter_class_3 + filter_edge_3x3 + filter_edge_5x5 + [square_3x3, square_5x5]
+
+hpf_3x3_list = filter_class_1 + filter_class_2 + filter_edge_3x3 + [square_3x3]
+hpf_5x5_list = filter_class_3 + filter_edge_5x5 + [square_5x5]
+
+normalized_filter_class_2 = [hpf / 2 for hpf in filter_class_2]
+normalized_filter_class_3 = [hpf / 3 for hpf in filter_class_3]
+normalized_filter_edge_3x3 = [hpf / 4 for hpf in filter_edge_3x3]
+normalized_square_3x3 = square_3x3 / 4
+normalized_filter_edge_5x5 = [hpf / 12 for hpf in filter_edge_5x5]
+normalized_square_5x5 = square_5x5 / 12
+
+all_normalized_hpf_list = filter_class_1 + normalized_filter_class_2 + normalized_filter_class_3 + \
+ normalized_filter_edge_3x3 + normalized_filter_edge_5x5 + [normalized_square_3x3, normalized_square_5x5]
+
+normalized_hpf_3x3_list = filter_class_1 + normalized_filter_class_2 + normalized_filter_edge_3x3 + [normalized_square_3x3]
+normalized_hpf_5x5_list = normalized_filter_class_3 + normalized_filter_edge_5x5 + [normalized_square_5x5]
+
+normalized_3x3_list = normalized_filter_edge_3x3 + [normalized_square_3x3]
+normalized_5x5_list = normalized_filter_edge_5x5 + [normalized_square_5x5]

detector_codes/AIDE-main/models/utils.py

@@ -0,0 +1,116 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+
+# All rights reserved.
+
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+import numpy.random as random
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+# from MinkowskiEngine import SparseTensor
+
+# class MinkowskiGRN(nn.Module):
+#     """ GRN layer for sparse tensors.
+#     """
+#     def __init__(self, dim):
+#         super().__init__()
+#         self.gamma = nn.Parameter(torch.zeros(1, dim))
+#         self.beta = nn.Parameter(torch.zeros(1, dim))
+
+#     def forward(self, x):
+#         cm = x.coordinate_manager
+#         in_key = x.coordinate_map_key
+
+#         Gx = torch.norm(x.F, p=2, dim=0, keepdim=True)
+#         Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6)
+#         return SparseTensor(
+#                 self.gamma * (x.F * Nx) + self.beta + x.F,
+#                 coordinate_map_key=in_key,
+#                 coordinate_manager=cm)
+
+# class MinkowskiDropPath(nn.Module):
+#     """ Drop Path for sparse tensors.
+#     """
+
+#     def __init__(self, drop_prob: float = 0., scale_by_keep: bool = True):
+#         super(MinkowskiDropPath, self).__init__()
+#         self.drop_prob = drop_prob
+#         self.scale_by_keep = scale_by_keep
+    
+#     def forward(self, x):
+#         if self.drop_prob == 0. or not self.training:
+#             return x
+#         cm = x.coordinate_manager
+#         in_key = x.coordinate_map_key
+#         keep_prob = 1 - self.drop_prob
+#         mask = torch.cat([
+#             torch.ones(len(_)) if random.uniform(0, 1) > self.drop_prob
+#             else torch.zeros(len(_)) for _ in x.decomposed_coordinates
+#         ]).view(-1, 1).to(x.device)
+#         if keep_prob > 0.0 and self.scale_by_keep:
+#             mask.div_(keep_prob)
+#         return SparseTensor(
+#                 x.F * mask,
+#                 coordinate_map_key=in_key,
+#                 coordinate_manager=cm)
+
+# class MinkowskiLayerNorm(nn.Module):
+#     """ Channel-wise layer normalization for sparse tensors.
+#     """
+
+#     def __init__(
+#         self,
+#         normalized_shape,
+#         eps=1e-6,
+#     ):
+#         super(MinkowskiLayerNorm, self).__init__()
+#         self.ln = nn.LayerNorm(normalized_shape, eps=eps)
+#     def forward(self, input):
+#         output = self.ln(input.F)
+#         return SparseTensor(
+#             output,
+#             coordinate_map_key=input.coordinate_map_key,
+#             coordinate_manager=input.coordinate_manager)
+            
+class LayerNorm(nn.Module):
+    """ LayerNorm that supports two data formats: channels_last (default) or channels_first. 
+    The ordering of the dimensions in the inputs. channels_last corresponds to inputs with 
+    shape (batch_size, height, width, channels) while channels_first corresponds to inputs 
+    with shape (batch_size, channels, height, width).
+    """
+    def __init__(self, normalized_shape, eps=1e-6, data_format="channels_last"):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(normalized_shape))
+        self.bias = nn.Parameter(torch.zeros(normalized_shape))
+        self.eps = eps
+        self.data_format = data_format
+        if self.data_format not in ["channels_last", "channels_first"]:
+            raise NotImplementedError 
+        self.normalized_shape = (normalized_shape, )
+    
+    def forward(self, x):
+        if self.data_format == "channels_last":
+            return F.layer_norm(x, self.normalized_shape, self.weight, self.bias, self.eps)
+        elif self.data_format == "channels_first":
+            u = x.mean(1, keepdim=True)
+            s = (x - u).pow(2).mean(1, keepdim=True)
+            x = (x - u) / torch.sqrt(s + self.eps)
+            x = self.weight[:, None, None] * x + self.bias[:, None, None]
+            return x
+
+class GRN(nn.Module):
+    """ GRN (Global Response Normalization) layer
+    """
+    def __init__(self, dim):
+        super().__init__()
+        self.gamma = nn.Parameter(torch.zeros(1, 1, 1, dim))
+        self.beta = nn.Parameter(torch.zeros(1, 1, 1, dim))
+
+    def forward(self, x):
+        Gx = torch.norm(x, p=2, dim=(1,2), keepdim=True)
+        Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6)
+        return self.gamma * (x * Nx) + self.beta + x

detector_codes/AIDE-main/optim_factory.py

@@ -0,0 +1,222 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+
+# All rights reserved.
+
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+import torch
+from torch import optim as optim
+
+from timm.optim.adafactor import Adafactor
+from timm.optim.adahessian import Adahessian
+from timm.optim.adamp import AdamP
+from timm.optim.lookahead import Lookahead
+# from timm.optim.nadam import Nadam
+# from timm.optim.novograd import NovoGrad
+# from timm.optim.nvnovograd import NvNovoGrad
+# from timm.optim.radam import RAdam
+from timm.optim.rmsprop_tf import RMSpropTF
+from timm.optim.sgdp import SGDP
+
+import json
+
+try:
+    from apex.optimizers import FusedNovoGrad, FusedAdam, FusedLAMB, FusedSGD
+    has_apex = True
+except ImportError:
+    has_apex = False
+
+
+def get_num_layer_for_convnext_single(var_name, depths):
+    """
+    Each layer is assigned distinctive layer ids
+    """
+    if var_name.startswith("downsample_layers"):
+        stage_id = int(var_name.split('.')[1])
+        layer_id = sum(depths[:stage_id]) + 1
+        return layer_id
+
+    elif var_name.startswith("stages"):
+        stage_id = int(var_name.split('.')[1])
+        block_id = int(var_name.split('.')[2])
+        layer_id = sum(depths[:stage_id]) + block_id + 1
+        return layer_id
+
+    else:
+        return sum(depths) + 1
+
+
+def get_num_layer_for_convnext(var_name):
+    """
+    Divide [3, 3, 27, 3] layers into 12 groups; each group is three
+    consecutive blocks, including possible neighboring downsample layers;
+    adapted from https://github.com/microsoft/unilm/blob/master/beit/optim_factory.py
+    """
+    num_max_layer = 12
+    if var_name.startswith("downsample_layers"):
+        stage_id = int(var_name.split('.')[1])
+        if stage_id == 0:
+            layer_id = 0
+        elif stage_id == 1 or stage_id == 2:
+            layer_id = stage_id + 1
+        elif stage_id == 3:
+            layer_id = 12
+        return layer_id
+
+    elif var_name.startswith("stages"):
+        stage_id = int(var_name.split('.')[1])
+        block_id = int(var_name.split('.')[2])
+        if stage_id == 0 or stage_id == 1:
+            layer_id = stage_id + 1
+        elif stage_id == 2:
+            layer_id = 3 + block_id // 3
+        elif stage_id == 3:
+            layer_id = 12
+        return layer_id
+    else:
+        return num_max_layer + 1
+
+class LayerDecayValueAssigner(object):
+    def __init__(self, values, depths=[3,3,27,3], layer_decay_type='single'):
+        self.values = values
+        self.depths = depths
+        self.layer_decay_type = layer_decay_type
+
+    def get_scale(self, layer_id):
+        return self.values[layer_id]
+
+    def get_layer_id(self, var_name):
+        if self.layer_decay_type == 'single':
+            return get_num_layer_for_convnext_single(var_name, self.depths)
+        else:
+            return get_num_layer_for_convnext(var_name)
+
+
+def get_parameter_groups(model, weight_decay=1e-5, skip_list=(), get_num_layer=None, get_layer_scale=None):
+    parameter_group_names = {}
+    parameter_group_vars = {}
+
+    for name, param in model.named_parameters():
+        if not param.requires_grad:
+            continue  # frozen weights
+        if len(param.shape) == 1 or name.endswith(".bias") or name in skip_list or \
+            name.endswith(".gamma") or name.endswith(".beta"):
+            group_name = "no_decay"
+            this_weight_decay = 0.
+        else:
+            group_name = "decay"
+            this_weight_decay = weight_decay
+        if get_num_layer is not None:
+            layer_id = get_num_layer(name)
+            group_name = "layer_%d_%s" % (layer_id, group_name)
+        else:
+            layer_id = None
+
+        if group_name not in parameter_group_names:
+            if get_layer_scale is not None:
+                scale = get_layer_scale(layer_id)
+            else:
+                scale = 1.
+
+            parameter_group_names[group_name] = {
+                "weight_decay": this_weight_decay,
+                "params": [],
+                "lr_scale": scale
+            }
+            parameter_group_vars[group_name] = {
+                "weight_decay": this_weight_decay,
+                "params": [],
+                "lr_scale": scale
+            }
+
+        parameter_group_vars[group_name]["params"].append(param)
+        parameter_group_names[group_name]["params"].append(name)
+    print("Param groups = %s" % json.dumps(parameter_group_names, indent=2))
+    return list(parameter_group_vars.values())
+
+
+def create_optimizer(args, model, get_num_layer=None, get_layer_scale=None, filter_bias_and_bn=True, skip_list=None):
+    opt_lower = args.opt.lower()
+    weight_decay = args.weight_decay
+    # if weight_decay and filter_bias_and_bn:
+    if filter_bias_and_bn:
+        skip = {}
+        if skip_list is not None:
+            skip = skip_list
+        elif hasattr(model, 'no_weight_decay'):
+            skip = model.no_weight_decay()
+        parameters = get_parameter_groups(model, weight_decay, skip, get_num_layer, get_layer_scale)
+        weight_decay = 0.
+    else:
+        parameters = model.parameters()
+
+    if 'fused' in opt_lower:
+        assert has_apex and torch.cuda.is_available(), 'APEX and CUDA required for fused optimizers'
+
+    opt_args = dict(lr=args.lr, weight_decay=weight_decay)
+    if hasattr(args, 'opt_eps') and args.opt_eps is not None:
+        opt_args['eps'] = args.opt_eps
+    if hasattr(args, 'opt_betas') and args.opt_betas is not None:
+        opt_args['betas'] = args.opt_betas
+
+    opt_split = opt_lower.split('_')
+    opt_lower = opt_split[-1]
+    if opt_lower == 'sgd' or opt_lower == 'nesterov':
+        opt_args.pop('eps', None)
+        optimizer = optim.SGD(parameters, momentum=args.momentum, nesterov=True, **opt_args)
+    elif opt_lower == 'momentum':
+        opt_args.pop('eps', None)
+        optimizer = optim.SGD(parameters, momentum=args.momentum, nesterov=False, **opt_args)
+    elif opt_lower == 'adam':
+        optimizer = optim.Adam(parameters, **opt_args)
+    elif opt_lower == 'adamw':
+        optimizer = optim.AdamW(parameters, **opt_args)
+    # elif opt_lower == 'nadam':
+    #     optimizer = Nadam(parameters, **opt_args)
+    # elif opt_lower == 'radam':
+    #     optimizer = RAdam(parameters, **opt_args)
+    elif opt_lower == 'adamp':
+        optimizer = AdamP(parameters, wd_ratio=0.01, nesterov=True, **opt_args)
+    elif opt_lower == 'sgdp':
+        optimizer = SGDP(parameters, momentum=args.momentum, nesterov=True, **opt_args)
+    elif opt_lower == 'adadelta':
+        optimizer = optim.Adadelta(parameters, **opt_args)
+    elif opt_lower == 'adafactor':
+        if not args.lr:
+            opt_args['lr'] = None
+        optimizer = Adafactor(parameters, **opt_args)
+    elif opt_lower == 'adahessian':
+        optimizer = Adahessian(parameters, **opt_args)
+    elif opt_lower == 'rmsprop':
+        optimizer = optim.RMSprop(parameters, alpha=0.9, momentum=args.momentum, **opt_args)
+    elif opt_lower == 'rmsproptf':
+        optimizer = RMSpropTF(parameters, alpha=0.9, momentum=args.momentum, **opt_args)
+    elif opt_lower == 'novograd':
+        optimizer = NovoGrad(parameters, **opt_args)
+    elif opt_lower == 'nvnovograd':
+        optimizer = NvNovoGrad(parameters, **opt_args)
+    elif opt_lower == 'fusedsgd':
+        opt_args.pop('eps', None)
+        optimizer = FusedSGD(parameters, momentum=args.momentum, nesterov=True, **opt_args)
+    elif opt_lower == 'fusedmomentum':
+        opt_args.pop('eps', None)
+        optimizer = FusedSGD(parameters, momentum=args.momentum, nesterov=False, **opt_args)
+    elif opt_lower == 'fusedadam':
+        optimizer = FusedAdam(parameters, adam_w_mode=False, **opt_args)
+    elif opt_lower == 'fusedadamw':
+        optimizer = FusedAdam(parameters, adam_w_mode=True, **opt_args)
+    elif opt_lower == 'fusedlamb':
+        optimizer = FusedLAMB(parameters, **opt_args)
+    elif opt_lower == 'fusednovograd':
+        opt_args.setdefault('betas', (0.95, 0.98))
+        optimizer = FusedNovoGrad(parameters, **opt_args)
+    else:
+        assert False and "Invalid optimizer"
+
+    if len(opt_split) > 1:
+        if opt_split[0] == 'lookahead':
+            optimizer = Lookahead(optimizer)
+
+    return optimizer

detector_codes/AIDE-main/requirements.txt

@@ -0,0 +1,36 @@
+einops==0.6.1
+fairscale==0.4.13
+filelock==3.13.1
+ftfy==6.1.3
+h5py==3.10.0
+imgaug==0.2.6
+keras==2.11.0
+kornia==0.7.2
+kornia_rs==0.1.2
+lmdb==1.4.1
+matplotlib==3.7.4
+matplotlib-inline==0.1.6
+numpy==1.24.3
+omegaconf==2.3.0
+open-clip-torch==2.24.0
+openai-clip==1.0.1
+openpyxl==3.1.2
+pandas==2.0.3
+Pillow==9.5.0
+safetensors==0.4.1
+scikit-image==0.20.0
+scikit-learn==1.3.2
+scipy==1.9.1
+sentencepiece==0.2.0
+streamlit==1.30.0
+tenacity==8.2.3
+tensorboard==2.11.2
+tensorboard-data-server==0.6.1
+tensorboard-plugin-wit==1.8.1
+tensorboardX==2.6.2.2
+torch==1.11.0
+torch-fidelity==0.3.0
+torchmetrics==0.6.0
+torchsummary==1.5.1
+torchvision==0.12.0
+tqdm==4.66.1

detector_codes/AIDE-main/scripts/eval.sh

@@ -0,0 +1,21 @@
+MODEL="AIDE"
+RESUME_PATH="./4class-checkpoints"
+
+eval_datasets=(
+    # "/data/ziqiang/Benchmark" \
+    # "/data/ziqiang/jpeg" \
+    # "/data/ziqiang/noise" \
+    "/data/ziqiang/sample" \
+)
+for eval_dataset in "${eval_datasets[@]}"
+do
+    python  main_finetune.py \
+        --model $MODEL \
+        --data_path /data/ziqiang/yjz/dataset/ForenSynths/train \
+        --eval_data_path $eval_dataset \
+        --batch_size 64 \
+        --output_dir $RESUME_PATH \
+        --resume $RESUME_PATH/checkpoint-best.pth \
+        --eval True
+done
+

detector_codes/AIDE-main/scripts/train.sh

@@ -0,0 +1,24 @@
+
+GPU_NUM=8
+WORLD_SIZE=1
+RANK=0
+MASTER_ADDR=localhost
+MASTER_PORT=29512
+
+
+DISTRIBUTED_ARGS="
+    --nproc_per_node $GPU_NUM \
+    --nnodes $WORLD_SIZE \
+    --node_rank $RANK \
+    --master_addr $MASTER_ADDR \
+    --master_port $MASTER_PORT
+"
+
+PY_ARGS=${@:1}  # Any other arguments 
+
+python -m torch.distributed.launch $DISTRIBUTED_ARGS main_finetune.py \
+    --model AIDE \
+    --batch_size 32 \
+    --blr 1e-4 \
+    --epochs 20 \
+    ${PY_ARGS}

detector_codes/AIDE-main/utils.py

@@ -0,0 +1,700 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+
+# All rights reserved.
+
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+import datetime
+import math
+import os
+import time
+from collections import OrderedDict, defaultdict, deque
+from pathlib import Path
+
+import numpy as np
+import torch
+import torch.distributed as dist
+from tensorboardX import SummaryWriter
+from timm.utils import get_state_dict
+from torch import inf
+
+
+def str2bool(v):
+    """
+    Converts string to bool type; enables command line
+    arguments in the format of '--arg1 true --arg2 false'
+    """
+    if isinstance(v, bool):
+        return v
+    if v.lower() in ('yes', 'true', 't', 'y', '1'):
+        return True
+    elif v.lower() in ('no', 'false', 'f', 'n', '0'):
+        return False
+    else:
+        raise argparse.ArgumentTypeError('Boolean value expected.')
+
+
+class SmoothedValue(object):
+    """Track a series of values and provide access to smoothed values over a
+    window or the global series average.
+    """
+
+    def __init__(self, window_size=20, fmt=None):
+        if fmt is None:
+            fmt = '{median:.4f} ({global_avg:.4f})'
+        self.deque = deque(maxlen=window_size)
+        self.total = 0.0
+        self.count = 0
+        self.fmt = fmt
+
+    def update(self, value, n=1):
+        self.deque.append(value)
+        self.count += n
+        self.total += value * n
+
+    def synchronize_between_processes(self):
+        """
+        Warning: does not synchronize the deque!
+        """
+        if not is_dist_avail_and_initialized():
+            return
+        t = torch.tensor([self.count, self.total], dtype=torch.float64, device='cuda')
+        dist.barrier()
+        dist.all_reduce(t)
+        t = t.tolist()
+        self.count = int(t[0])
+        self.total = t[1]
+
+    @property
+    def median(self):
+        d = torch.tensor(list(self.deque))
+        return d.median().item()
+
+    @property
+    def avg(self):
+        d = torch.tensor(list(self.deque), dtype=torch.float32)
+        return d.mean().item()
+
+    @property
+    def global_avg(self):
+        return self.total / self.count
+
+    @property
+    def max(self):
+        return max(self.deque)
+
+    @property
+    def value(self):
+        return self.deque[-1]
+
+    def __str__(self):
+        return self.fmt.format(
+            median=self.median,
+            avg=self.avg,
+            global_avg=self.global_avg,
+            max=self.max,
+            value=self.value,
+        )
+
+
+class MetricLogger(object):
+    def __init__(self, delimiter='\t'):
+        self.meters = defaultdict(SmoothedValue)
+        self.delimiter = delimiter
+
+    def update(self, **kwargs):
+        for k, v in kwargs.items():
+            if v is None:
+                continue
+            if isinstance(v, torch.Tensor):
+                v = v.item()
+            assert isinstance(v, (float, int))
+            self.meters[k].update(v)
+
+    def __getattr__(self, attr):
+        if attr in self.meters:
+            return self.meters[attr]
+        if attr in self.__dict__:
+            return self.__dict__[attr]
+        raise AttributeError(
+            "'{}' object has no attribute '{}'".format(type(self).__name__, attr)
+        )
+
+    def __str__(self):
+        loss_str = []
+        for name, meter in self.meters.items():
+            loss_str.append('{}: {}'.format(name, str(meter)))
+        return self.delimiter.join(loss_str)
+
+    def synchronize_between_processes(self):
+        for meter in self.meters.values():
+            meter.synchronize_between_processes()
+
+    def add_meter(self, name, meter):
+        self.meters[name] = meter
+
+    def log_every(self, iterable, print_freq, header=None):
+        i = 0
+        if not header:
+            header = ''
+        start_time = time.time()
+        end = time.time()
+        iter_time = SmoothedValue(fmt='{avg:.4f}')
+        data_time = SmoothedValue(fmt='{avg:.4f}')
+        space_fmt = ':' + str(len(str(len(iterable)))) + 'd'
+        log_msg = [
+            header,
+            '[{0' + space_fmt + '}/{1}]',
+            'eta: {eta}',
+            '{meters}',
+            'time: {time}',
+            'data: {data}',
+        ]
+        if torch.cuda.is_available():
+            log_msg.append('max mem: {memory:.0f}')
+        log_msg = self.delimiter.join(log_msg)
+        MB = 1024.0 * 1024.0
+        for obj in iterable:
+            data_time.update(time.time() - end)
+            yield obj
+            iter_time.update(time.time() - end)
+            if i % print_freq == 0 or i == len(iterable) - 1:
+                eta_seconds = iter_time.global_avg * (len(iterable) - i)
+                eta_string = str(datetime.timedelta(seconds=int(eta_seconds)))
+                if torch.cuda.is_available():
+                    print(
+                        log_msg.format(
+                            i,
+                            len(iterable),
+                            eta=eta_string,
+                            meters=str(self),
+                            time=str(iter_time),
+                            data=str(data_time),
+                            memory=torch.cuda.max_memory_allocated() / MB,
+                        )
+                    )
+                else:
+                    print(
+                        log_msg.format(
+                            i,
+                            len(iterable),
+                            eta=eta_string,
+                            meters=str(self),
+                            time=str(iter_time),
+                            data=str(data_time),
+                        )
+                    )
+            i += 1
+            end = time.time()
+        total_time = time.time() - start_time
+        total_time_str = str(datetime.timedelta(seconds=int(total_time)))
+        print(
+            '{} Total time: {} ({:.4f} s / it)'.format(
+                header, total_time_str, total_time / len(iterable)
+            )
+        )
+
+
+class TensorboardLogger(object):
+    def __init__(self, log_dir):
+        self.writer = SummaryWriter(logdir=log_dir)
+        self.step = 0
+
+    def set_step(self, step=None):
+        if step is not None:
+            self.step = step
+        else:
+            self.step += 1
+
+    def update(self, head='scalar', step=None, **kwargs):
+        for k, v in kwargs.items():
+            if v is None:
+                continue
+            if isinstance(v, torch.Tensor):
+                v = v.item()
+            assert isinstance(v, (float, int))
+            self.writer.add_scalar(
+                head + '/' + k, v, self.step if step is None else step
+            )
+
+    def flush(self):
+        self.writer.flush()
+
+
+class WandbLogger(object):
+    def __init__(self, args):
+        self.args = args
+
+        try:
+            import wandb
+
+            self._wandb = wandb
+        except ImportError:
+            raise ImportError(
+                'To use the Weights and Biases Logger please install wandb.'
+                'Run `pip install wandb` to install it.'
+            )
+
+        # Initialize a W&B run
+        if self._wandb.run is None:
+            self._wandb.init(project=args.project, config=args)
+
+    def log_epoch_metrics(self, metrics, commit=True):
+        """
+        Log train/test metrics onto W&B.
+        """
+        # Log number of model parameters as W&B summary
+        self._wandb.summary['n_parameters'] = metrics.get('n_parameters', None)
+        metrics.pop('n_parameters', None)
+
+        # Log current epoch
+        self._wandb.log({'epoch': metrics.get('epoch')}, commit=False)
+        metrics.pop('epoch')
+
+        for k, v in metrics.items():
+            if 'train' in k:
+                self._wandb.log({f'Global Train/{k}': v}, commit=False)
+            elif 'test' in k:
+                self._wandb.log({f'Global Test/{k}': v}, commit=False)
+
+        self._wandb.log({})
+
+    def log_checkpoints(self):
+        output_dir = self.args.output_dir
+        model_artifact = self._wandb.Artifact(
+            self._wandb.run.id + '_model', type='model'
+        )
+
+        model_artifact.add_dir(output_dir)
+        self._wandb.log_artifact(model_artifact, aliases=['latest', 'best'])
+
+    def set_steps(self):
+        # Set global training step
+        self._wandb.define_metric(
+            'Rank-0 Batch Wise/*', step_metric='Rank-0 Batch Wise/global_train_step'
+        )
+        # Set epoch-wise step
+        self._wandb.define_metric('Global Train/*', step_metric='epoch')
+        self._wandb.define_metric('Global Test/*', step_metric='epoch')
+
+
+def setup_for_distributed(is_master):
+    """
+    This function disables printing when not in master process
+    """
+    import builtins as __builtin__
+
+    builtin_print = __builtin__.print
+
+    def print(*args, **kwargs):
+        force = kwargs.pop('force', False)
+        if is_master or force:
+            builtin_print(*args, **kwargs)
+
+    __builtin__.print = print
+
+
+def is_dist_avail_and_initialized():
+    if not dist.is_available():
+        return False
+    if not dist.is_initialized():
+        return False
+    return True
+
+
+def get_world_size():
+    if not is_dist_avail_and_initialized():
+        return 1
+    return dist.get_world_size()
+
+
+def get_rank():
+    if not is_dist_avail_and_initialized():
+        return 0
+    return dist.get_rank()
+
+
+def is_main_process():
+    return get_rank() == 0
+
+
+def save_on_master(*args, **kwargs):
+    if is_main_process():
+        torch.save(*args, **kwargs)
+
+
+def init_distributed_mode(args):
+
+    if args.dist_on_itp:
+        args.rank = int(os.environ['OMPI_COMM_WORLD_RANK'])
+        args.world_size = int(os.environ['OMPI_COMM_WORLD_SIZE'])
+        args.gpu = int(os.environ['OMPI_COMM_WORLD_LOCAL_RANK'])
+        args.dist_url = 'tcp://%s:%s' % (
+            os.environ['MASTER_ADDR'],
+            os.environ['MASTER_PORT'],
+        )
+        os.environ['LOCAL_RANK'] = str(args.gpu)
+        os.environ['RANK'] = str(args.rank)
+        os.environ['WORLD_SIZE'] = str(args.world_size)
+        # ["RANK", "WORLD_SIZE", "MASTER_ADDR", "MASTER_PORT", "LOCAL_RANK"]
+    elif 'RANK' in os.environ and 'WORLD_SIZE' in os.environ:
+        args.rank = int(os.environ['RANK'])
+        args.world_size = int(os.environ['WORLD_SIZE'])
+        args.gpu = int(os.environ['LOCAL_RANK'])
+    elif 'SLURM_PROCID' in os.environ:
+        args.rank = int(os.environ['SLURM_PROCID'])
+        args.gpu = args.rank % torch.cuda.device_count()
+
+        os.environ['RANK'] = str(args.rank)
+        os.environ['LOCAL_RANK'] = str(args.gpu)
+        os.environ['WORLD_SIZE'] = str(args.world_size)
+    else:
+        print('Not using distributed mode')
+        args.distributed = False
+        return
+
+    args.distributed = True
+
+    torch.cuda.set_device(args.gpu)
+    args.dist_backend = 'nccl'
+    print(
+        '| distributed init (rank {}): {}, gpu {}'.format(
+            args.rank, args.dist_url, args.gpu
+        ),
+        flush=True,
+    )
+    torch.distributed.init_process_group(
+        backend=args.dist_backend,
+        init_method=args.dist_url,
+        world_size=args.world_size,
+        rank=args.rank,
+    )
+    torch.distributed.barrier()
+    setup_for_distributed(args.rank == 0)
+
+
+def all_reduce_mean(x):
+    world_size = get_world_size()
+    if world_size > 1:
+        x_reduce = torch.tensor(x).cuda()
+        dist.all_reduce(x_reduce)
+        x_reduce /= world_size
+        return x_reduce.item()
+    else:
+        return x
+
+
+def load_state_dict(
+    model, state_dict, prefix='', ignore_missing='relative_position_index'
+):
+    missing_keys = []
+    unexpected_keys = []
+    error_msgs = []
+    # copy state_dict so _load_from_state_dict can modify it
+    metadata = getattr(state_dict, '_metadata', None)
+    state_dict = state_dict.copy()
+    if metadata is not None:
+        state_dict._metadata = metadata
+
+    def load(module, prefix=''):
+        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
+        module._load_from_state_dict(
+            state_dict,
+            prefix,
+            local_metadata,
+            True,
+            missing_keys,
+            unexpected_keys,
+            error_msgs,
+        )
+        for name, child in module._modules.items():
+            if child is not None:
+                load(child, prefix + name + '.')
+
+    load(model, prefix=prefix)
+
+    warn_missing_keys = []
+    ignore_missing_keys = []
+    for key in missing_keys:
+        keep_flag = True
+        for ignore_key in ignore_missing.split('|'):
+            if ignore_key in key:
+                keep_flag = False
+                break
+        if keep_flag:
+            warn_missing_keys.append(key)
+        else:
+            ignore_missing_keys.append(key)
+
+    missing_keys = warn_missing_keys
+
+    if len(missing_keys) > 0:
+        print(
+            'Weights of {} not initialized from pretrained model: {}'.format(
+                model.__class__.__name__, missing_keys
+            )
+        )
+    if len(unexpected_keys) > 0:
+        print(
+            'Weights from pretrained model not used in {}: {}'.format(
+                model.__class__.__name__, unexpected_keys
+            )
+        )
+    if len(ignore_missing_keys) > 0:
+        print(
+            'Ignored weights of {} not initialized from pretrained model: {}'.format(
+                model.__class__.__name__, ignore_missing_keys
+            )
+        )
+    if len(error_msgs) > 0:
+        print('\n'.join(error_msgs))
+
+
+class NativeScalerWithGradNormCount:
+    state_dict_key = 'amp_scaler'
+
+    def __init__(self):
+        self._scaler = torch.cuda.amp.GradScaler()
+
+    def __call__(
+        self,
+        loss,
+        optimizer,
+        clip_grad=None,
+        parameters=None,
+        create_graph=False,
+        update_grad=True,
+    ):
+        self._scaler.scale(loss).backward(create_graph=create_graph)
+        if update_grad:
+            if clip_grad is not None:
+                assert parameters is not None
+                self._scaler.unscale_(
+                    optimizer
+                )  # unscale the gradients of optimizer's assigned params in-place
+                norm = torch.nn.utils.clip_grad_norm_(parameters, clip_grad)
+            else:
+                self._scaler.unscale_(optimizer)
+                norm = get_grad_norm_(parameters)
+            self._scaler.step(optimizer)
+            self._scaler.update()
+        else:
+            norm = None
+        return norm
+
+    def state_dict(self):
+        return self._scaler.state_dict()
+
+    def load_state_dict(self, state_dict):
+        self._scaler.load_state_dict(state_dict)
+
+
+def get_grad_norm_(parameters, norm_type: float = 2.0) -> torch.Tensor:
+    if isinstance(parameters, torch.Tensor):
+        parameters = [parameters]
+    parameters = [p for p in parameters if p.grad is not None]
+    norm_type = float(norm_type)
+    if len(parameters) == 0:
+        return torch.tensor(0.0)
+    device = parameters[0].grad.device
+    if norm_type == inf:
+        total_norm = max(p.grad.detach().abs().max().to(device) for p in parameters)
+    else:
+        total_norm = torch.norm(
+            torch.stack(
+                [torch.norm(p.grad.detach(), norm_type).to(device) for p in parameters]
+            ),
+            norm_type,
+        )
+    return total_norm
+
+
+def save_model(
+    args, epoch, model, model_without_ddp, optimizer, loss_scaler, model_ema=None
+):
+    output_dir = Path(args.output_dir)
+    epoch_name = str(epoch)
+    checkpoint_paths = [output_dir / ('checkpoint-%s.pth' % epoch_name)]
+    for checkpoint_path in checkpoint_paths:
+        to_save = {
+            'model': model_without_ddp.state_dict(),
+            'optimizer': optimizer.state_dict(),
+            'epoch': epoch,
+            'scaler': loss_scaler.state_dict(),
+            'args': args,
+        }
+
+        if model_ema is not None:
+            to_save['model_ema'] = get_state_dict(model_ema)
+
+        save_on_master(to_save, checkpoint_path)
+
+    if is_main_process() and isinstance(epoch, int):
+        to_del = epoch - args.save_ckpt_num * args.save_ckpt_freq
+        old_ckpt = output_dir / ('checkpoint-%s.pth' % to_del)
+        if os.path.exists(old_ckpt):
+            os.remove(old_ckpt)
+
+
+def auto_load_model(
+    args, model, model_without_ddp, optimizer, loss_scaler, model_ema=None
+):
+    output_dir = Path(args.output_dir)
+    if args.auto_resume and len(args.resume) == 0:
+        import glob
+
+        all_checkpoints = glob.glob(os.path.join(output_dir, 'checkpoint-*.pth'))
+        latest_ckpt = -1
+        for ckpt in all_checkpoints:
+            t = ckpt.split('-')[-1].split('.')[0]
+            if t.isdigit():
+                latest_ckpt = max(int(t), latest_ckpt)
+        if latest_ckpt >= 0:
+            args.resume = os.path.join(output_dir, 'checkpoint-%d.pth' % latest_ckpt)
+        print('Auto resume checkpoint: %s' % args.resume)
+
+    if args.resume:
+        if args.resume.startswith('https'):
+            checkpoint = torch.hub.load_state_dict_from_url(
+                args.resume, map_location='cpu', check_hash=True
+            )
+        else:
+            checkpoint = torch.load(args.resume, map_location='cpu')
+
+        model_without_ddp.load_state_dict(checkpoint['model'])
+        print('Resume checkpoint %s' % args.resume)
+        if 'optimizer' in checkpoint and 'epoch' in checkpoint:
+            optimizer.load_state_dict(checkpoint['optimizer'])
+            if not isinstance(
+                checkpoint['epoch'], str
+            ):  # does not support resuming with 'best', 'best-ema'
+                args.start_epoch = checkpoint['epoch'] + 1
+            else:
+                assert args.eval, 'Does not support resuming with checkpoint-best'
+            if hasattr(args, 'model_ema') and args.model_ema:
+                if 'model_ema' in checkpoint.keys():
+                    model_ema.ema.load_state_dict(checkpoint['model_ema'])
+                else:
+                    model_ema.ema.load_state_dict(checkpoint['model'])
+            if 'scaler' in checkpoint:
+                loss_scaler.load_state_dict(checkpoint['scaler'])
+            print('With optim & sched!')
+
+
+def cosine_scheduler(
+    base_value,
+    final_value,
+    epochs,
+    niter_per_ep,
+    warmup_epochs=0,
+    start_warmup_value=0,
+    warmup_steps=-1,
+):
+    warmup_schedule = np.array([])
+    warmup_iters = warmup_epochs * niter_per_ep
+    if warmup_steps > 0:
+        warmup_iters = warmup_steps
+    print('Set warmup steps = %d' % warmup_iters)
+    if warmup_epochs > 0:
+        warmup_schedule = np.linspace(start_warmup_value, base_value, warmup_iters)
+
+    iters = np.arange(epochs * niter_per_ep - warmup_iters)
+    schedule = np.array(
+        [
+            final_value
+            + 0.5
+            * (base_value - final_value)
+            * (1 + math.cos(math.pi * i / (len(iters))))
+            for i in iters
+        ]
+    )
+
+    schedule = np.concatenate((warmup_schedule, schedule))
+
+    assert len(schedule) == epochs * niter_per_ep
+    return schedule
+
+
+def adjust_learning_rate(optimizer, epoch, args):
+    """Decay the learning rate with half-cycle cosine after warmup"""
+    if epoch < args.warmup_epochs:
+        lr = args.lr * epoch / args.warmup_epochs
+    else:
+        lr = args.min_lr + (args.lr - args.min_lr) * 0.5 * (
+            1.0
+            + math.cos(
+                math.pi
+                * (epoch - args.warmup_epochs)
+                / (args.epochs - args.warmup_epochs)
+            )
+        )
+    for param_group in optimizer.param_groups:
+        if 'lr_scale' in param_group:
+            param_group['lr'] = lr * param_group['lr_scale']
+        else:
+            param_group['lr'] = lr
+    return lr
+
+
+def remap_checkpoint_keys(ckpt):
+    new_ckpt = OrderedDict()
+    for k, v in ckpt.items():
+        if k.startswith('encoder'):
+            k = '.'.join(k.split('.')[1:])  # remove encoder in the name
+        if k.endswith('kernel'):
+            k = '.'.join(k.split('.')[:-1])  # remove kernel in the name
+            new_k = k + '.weight'
+            if len(v.shape) == 3:  # resahpe standard convolution
+                kv, in_dim, out_dim = v.shape
+                ks = int(math.sqrt(kv))
+                new_ckpt[new_k] = (
+                    v.permute(2, 1, 0).reshape(out_dim, in_dim, ks, ks).transpose(3, 2)
+                )
+            elif len(v.shape) == 2:  # reshape depthwise convolution
+                kv, dim = v.shape
+                ks = int(math.sqrt(kv))
+                new_ckpt[new_k] = (
+                    v.permute(1, 0).reshape(dim, 1, ks, ks).transpose(3, 2)
+                )
+            continue
+        elif 'ln' in k or 'linear' in k:
+            k = k.split('.')
+            k.pop(-2)  # remove ln and linear in the name
+            new_k = '.'.join(k)
+        else:
+            new_k = k
+        new_ckpt[new_k] = v
+
+    # reshape grn affine parameters and biases
+    for k, v in new_ckpt.items():
+        if k.endswith('bias') and len(v.shape) != 1:
+            new_ckpt[k] = v.reshape(-1)
+        # elif 'grn' in k:
+        #     new_ckpt[k] = v.unsqueeze(0).unsqueeze(1)
+    return new_ckpt
+
+
+class Logger(object):
+    """Log stdout messages."""
+
+    def __init__(self, outfile):
+        self.terminal = sys.stdout
+        self.log = open(outfile, 'a')
+        sys.stdout = self
+
+    def write(self, message):
+        self.terminal.write(message)
+        self.log.write(message)
+
+    def flush(self):
+        self.terminal.flush()
+
+
+def printSet(set_str):
+    set_str = str(set_str)
+    num = len(set_str)
+    print('=' * num * 3)
+    print(' ' * num + set_str)
+    print('=' * num * 3)

detector_codes/C2P-CLIP-DeepfakeDetection-main/Word_Frequency_Analysis.py

@@ -0,0 +1,147 @@
+import argparse
+import os
+import re
+from collections import Counter, OrderedDict
+
+import matplotlib.pyplot as plt
+import nltk
+import numpy as np
+from nltk.corpus import stopwords
+
+# os.environ['CURL_CA_BUNDLE'] = ''
+# os.environ['HTTP_PROXY'] = "http://*:7890"
+# os.environ['HTTPS_PROXY'] = "http://*:7890"
+# os.environ['ALL_PROXY'] = "socks://*:7891"
+nltk.download('stopwords')
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='Word_Frequency_Analysis')
+    parser.add_argument('--root_path', type=str, help='')
+    parser.add_argument('--save_path', type=str, help='save_path', default='')
+    args = parser.parse_args()
+
+    def print_options(parser, args):
+        message = ''
+        message += '----------------- Options ---------------\n'
+        for k, v in sorted(vars(args).items()):
+            comment = ''
+            default = parser.get_default(k)
+            if v != default:
+                comment = '\t[default: %s]' % str(default)
+            message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
+        message += '----------------- End -------------------'
+        print(message)
+
+    print_options(parser, args)
+    return args
+
+
+def get_list(folder_path):
+    image_paths_real = []
+    image_paths_fake = []
+    for root, dirs, files in os.walk(folder_path):
+        for file in files:
+            if file.lower().endswith(('.txt')):
+                abspath_tmp = os.path.abspath(os.path.join(root, file))
+                if '/0_real/' in abspath_tmp:
+                    image_paths_real.append(abspath_tmp)
+                if '/1_fake/' in abspath_tmp:
+                    image_paths_fake.append(abspath_tmp)
+    return image_paths_real, image_paths_fake
+
+
+def get_words_counts(image_paths):
+    all_text = []
+    for tpath in image_paths:
+        with open(tpath, 'r') as file:
+            all_text.append(file.read())
+    content = ' '.join(all_text)
+    words = re.findall(r'\b\w+\b', content.lower())
+
+    stop_words = set(stopwords.words('english'))
+    filtered_words = [word for word in words if word not in stop_words]
+
+    word_counts = Counter(filtered_words)
+
+    common_words = word_counts.most_common(20)
+    common_words_dict = {}
+    for common_word in common_words:
+        common_words_dict[common_word[0]] = common_word[1]
+    return common_words_dict, dict(word_counts)
+
+
+if __name__ == '__main__':
+    opt = parse_args()
+    os.makedirs(os.path.dirname(opt.save_path), mode=0o777, exist_ok=True)
+    if opt.root_path[-1] == '/':
+        opt.root_path = opt.root_path[:-1]
+    if isinstance(opt.root_path, list):
+        image_paths_real, image_paths_fake = [], []
+        for textp in opt.root_path:
+            tmp_real, tmp_fake = get_list(textp)
+            image_paths_real.extend(tmp_real)
+            image_paths_fake.extend(tmp_fake)
+    else:
+        image_paths_real, image_paths_fake = get_list(opt.root_path)
+
+    words_counts_real, words_counts_real_all = get_words_counts(image_paths_real)
+    words_counts_fake, words_counts_fake_all = get_words_counts(image_paths_fake)
+
+    all_words = set(list(words_counts_real.keys()) + list(words_counts_fake.keys()))
+    for word in all_words:
+        if word not in words_counts_real.keys():
+            words_counts_real[word] = (
+                words_counts_real_all[word]
+                if word in list(words_counts_real_all.keys())
+                else 0
+            )
+        if word not in words_counts_fake.keys():
+            words_counts_fake[word] = (
+                words_counts_fake_all[word]
+                if word in list(words_counts_fake_all.keys())
+                else 0
+            )
+
+    words_counts_fake_sorted = OrderedDict(
+        (key, words_counts_fake[key])
+        for key in words_counts_real
+        if key in words_counts_fake
+    )
+
+    words_counts_real = [(k, v) for k, v in words_counts_real.items()]
+    words_counts_fake_sorted = [(k, v) for k, v in words_counts_fake_sorted.items()]
+
+    words_real, counts_real = zip(*words_counts_real)
+    words_fake, counts_fake = zip(*words_counts_fake_sorted)
+    assert words_real == words_fake
+
+    print(f'words_real: {words_real}')
+    print(f'counts_real: {counts_real}')
+
+    print(f'words_fake: {words_fake}')
+    print(f'counts_fake: {counts_fake}')
+    # exit()
+    words_real, counts_real = words_real[:15], counts_real[:15]
+    words_fake, counts_fake = words_fake[:15], counts_fake[:15]
+    plt.figure(figsize=(16, 8))
+    width = 0.35
+    x = np.arange(len(words_real))
+    plt.bar(
+        x - width / 2,
+        counts_real,
+        width,
+        label=f'{" ".join(opt.root_path.split("/")[-2:])} real',
+    )
+    plt.bar(
+        x + width / 2,
+        counts_fake,
+        width,
+        label=f'{" ".join(opt.root_path.split("/")[-2:])} fake',
+    )
+    # plt.ylabel('Frequency')
+    # plt.title('Top 15 Most Common Words')
+    plt.xticks(x, labels=words_fake, rotation=90, fontsize=35)
+    plt.legend(prop={'size': 20})
+    plt.show()
+    plt.savefig(f'{opt.save_path}', bbox_inches='tight', pad_inches=0.1)

detector_codes/C2P-CLIP-DeepfakeDetection-main/data/__init__.py

@@ -0,0 +1,43 @@
+import torch
+import numpy as np
+from torch.utils.data.sampler import WeightedRandomSampler
+
+from .datasets import dataset_folder
+
+import os
+def get_dataset(opt):
+    classes = os.listdir(opt.dataroot) if len(opt.classes) == 0 else opt.classes
+    if '0_real' not in classes or '1_fake' not in classes:
+        dset_lst = []
+        for cls in classes:
+            root = opt.dataroot + '/' + cls
+            dset = dataset_folder(opt, root)
+            dset_lst.append(dset)
+        return torch.utils.data.ConcatDataset(dset_lst)
+    return dataset_folder(opt, opt.dataroot)
+
+def get_bal_sampler(dataset):
+    targets = []
+    for d in dataset.datasets:
+        targets.extend(d.targets)
+
+    ratio = np.bincount(targets)
+    w = 1. / torch.tensor(ratio, dtype=torch.float)
+    sample_weights = w[targets]
+    sampler = WeightedRandomSampler(weights=sample_weights,
+                                    num_samples=len(sample_weights))
+    return sampler
+
+
+def create_dataloader(opt):
+    shuffle = not opt.serial_batches if (opt.isTrain and not opt.class_bal) else False
+    dataset = get_dataset(opt)
+    sampler = get_bal_sampler(dataset) if opt.class_bal else None
+
+    data_loader = torch.utils.data.DataLoader(dataset,
+                                              batch_size=opt.batch_size,
+                                              shuffle=shuffle,
+                                              sampler=sampler,
+                                              drop_last=True if opt.isTrain else False,
+                                              num_workers=int(opt.num_threads))
+    return data_loader

detector_codes/C2P-CLIP-DeepfakeDetection-main/data/datasets.py

@@ -0,0 +1,213 @@
+import cv2
+import numpy as np
+import torchvision.datasets as datasets
+import torchvision.transforms as transforms
+import torchvision.transforms.functional as TF
+from random import random, choice
+from io import BytesIO
+from PIL import Image
+from PIL import ImageFile
+from scipy.ndimage.filters import gaussian_filter
+from torchvision.transforms import InterpolationMode
+from typing import Any, Callable, cast, Dict, List, Optional, Tuple
+import os
+from transformers import AutoTokenizer
+
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+IMG_EXTENSIONS = (".jpg", ".jpeg", ".png", ".ppm", ".bmp", ".pgm", ".tif", ".tiff", ".webp")
+
+def pil_loader(path: str) -> Image.Image:
+    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
+    with open(path, "rb") as f:
+        img = Image.open(f)
+        return img.convert("RGB")
+        
+class ImageFolder2(datasets.DatasetFolder):
+    def __init__(
+        self,
+        root: str,
+        opt,
+        transform: Optional[Callable] = None,
+    ):
+        super().__init__(
+            root,
+            transform=transform,
+            extensions=IMG_EXTENSIONS,
+            loader = pil_loader
+        )
+        self.opt = opt
+        self.tokenizer    = AutoTokenizer.from_pretrained(self.opt.clip, model_max_length=77, padding_side="right", use_fast=False)
+        self.tokenizer.pad_token_id = 0
+
+    def __getitem__(self, index: int) -> Tuple[Any, Any]:
+        """
+        Args:
+            index (int): Index
+
+        Returns:
+            tuple: (sample, target) where target is class_index of the target class.
+        """
+        path, target = self.samples[index]
+
+        
+        textpath = path.replace(self.opt.imgroot, self.opt.textroot)
+        textpath = os.path.splitext(textpath)[0] + '.txt'
+
+        sample = self.loader(path)
+        try:
+            with open(textpath, 'r') as file:
+                text = file.read()
+            cates_len = len(self.opt.cates)//2
+            if target == 1: text = f'{" ".join(self.opt.cates[:cates_len])}. {text} {" ".join(self.opt.cates[:cates_len])}.'
+            if target == 0: text = f'{" ".join(self.opt.cates[cates_len:])}. {text} {" ".join(self.opt.cates[cates_len:])}.'
+            inputs = self.tokenizer([text], padding="max_length", max_length=self.tokenizer.model_max_length, truncation=True, return_tensors="pt")
+            input_ids=inputs['input_ids'][0]
+            attention_mask=inputs['attention_mask'][0]
+        except:
+            text, input_ids, attention_mask = ' ', ' ', ' '
+
+        if self.transform is not None:
+            sample = self.transform(sample)
+        if self.target_transform is not None:
+            target = self.target_transform(target)
+
+        return path, sample, text, input_ids, attention_mask, target
+
+        
+def dataset_folder(opt, root):
+    if opt.mode == 'binary':
+        return binary_dataset(opt, root)
+    if opt.mode == 'filename':
+        return FileNameDataset(opt, root)
+    raise ValueError('opt.mode needs to be binary or filename.')
+
+
+def binary_dataset(opt, root):
+    if opt.isTrain:
+        crop_func = transforms.RandomCrop(opt.cropSize)
+    elif opt.no_crop:
+        crop_func = transforms.Lambda(lambda img: img)
+    else:
+        crop_func = transforms.CenterCrop(opt.cropSize)
+
+    if opt.isTrain and not opt.no_flip:
+        flip_func = transforms.RandomHorizontalFlip()
+    else:
+        flip_func = transforms.Lambda(lambda img: img)
+        
+    if not opt.isTrain and opt.no_resize:
+        rz_func = transforms.Lambda(lambda img: img)
+    else:
+        rz_func = transforms.Lambda(lambda img: translate_duplicate(img, opt.cropSize))
+
+    dset = ImageFolder2(
+            root,
+            opt,
+            transforms.Compose([
+                rz_func,
+                crop_func,
+                flip_func,
+                transforms.ToTensor(),
+                transforms.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711]),
+            ]))
+    return dset
+
+
+class FileNameDataset(datasets.ImageFolder):
+    def name(self):
+        return 'FileNameDataset'
+
+    def __init__(self, opt, root):
+        self.opt = opt
+        super().__init__(root)
+
+    def __getitem__(self, index):
+        # Loading sample
+        path, target = self.samples[index]
+        return path
+
+
+import math
+def translate_duplicate(img, cropSize):
+    if min(img.size) < cropSize:
+        width, height = img.size
+        
+        new_width = width * math.ceil(cropSize/width)
+        new_height = height * math.ceil(cropSize/height)
+        
+        new_img = Image.new('RGB', (new_width, new_height))
+        for i in range(0, new_width, width):
+            for j in range(0, new_height, height):
+                new_img.paste(img, (i, j))
+        return new_img
+    else:
+        return img
+
+def data_augment(img, opt):
+    img = np.array(img)
+
+    if random() < opt.blur_prob:
+        sig = sample_continuous(opt.blur_sig)
+        gaussian_blur(img, sig)
+
+    if random() < opt.jpg_prob:
+        method = sample_discrete(opt.jpg_method)
+        qual = sample_discrete(opt.jpg_qual)
+        img = jpeg_from_key(img, qual, method)
+
+    return Image.fromarray(img)
+
+
+def sample_continuous(s):
+    if len(s) == 1:
+        return s[0]
+    if len(s) == 2:
+        rg = s[1] - s[0]
+        return random() * rg + s[0]
+    raise ValueError("Length of iterable s should be 1 or 2.")
+
+
+def sample_discrete(s):
+    if len(s) == 1:
+        return s[0]
+    return choice(s)
+
+
+def gaussian_blur(img, sigma):
+    gaussian_filter(img[:,:,0], output=img[:,:,0], sigma=sigma)
+    gaussian_filter(img[:,:,1], output=img[:,:,1], sigma=sigma)
+    gaussian_filter(img[:,:,2], output=img[:,:,2], sigma=sigma)
+
+
+def cv2_jpg(img, compress_val):
+    img_cv2 = img[:,:,::-1]
+    encode_param = [int(cv2.IMWRITE_JPEG_QUALITY), compress_val]
+    result, encimg = cv2.imencode('.jpg', img_cv2, encode_param)
+    decimg = cv2.imdecode(encimg, 1)
+    return decimg[:,:,::-1]
+
+
+def pil_jpg(img, compress_val):
+    out = BytesIO()
+    img = Image.fromarray(img)
+    img.save(out, format='jpeg', quality=compress_val)
+    img = Image.open(out)
+    # load from memory before ByteIO closes
+    img = np.array(img)
+    out.close()
+    return img
+
+
+jpeg_dict = {'cv2': cv2_jpg, 'pil': pil_jpg}
+def jpeg_from_key(img, compress_val, key):
+    method = jpeg_dict[key]
+    return method(img, compress_val)
+
+
+rz_dict = {'bilinear': InterpolationMode.BILINEAR,
+           'bicubic': InterpolationMode.BICUBIC,
+           'lanczos': InterpolationMode.LANCZOS,
+           'nearest': InterpolationMode.NEAREST}
+def custom_resize(img, opt):
+    interp = sample_discrete(opt.rz_interp)
+    return TF.resize(img, opt.loadSize, interpolation=rz_dict[interp])

detector_codes/C2P-CLIP-DeepfakeDetection-main/decode_clipfeature_dataset.py

@@ -0,0 +1,113 @@
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import argparse
+import os
+import warnings
+
+import torch
+from tqdm import tqdm
+
+warnings.filterwarnings('ignore')
+from networks.decode_clipfeature_oneImage import (
+    get_clip_model,
+    get_clipcap_model,
+    get_image_features,
+    get_text,
+)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='decode detection feature to text')
+    parser.add_argument('--prefix_length', type=int, default=10)
+    parser.add_argument(
+        '--model_path', type=str, default='./ClipCaption_COCO.pt', help='model_path'
+    )
+    parser.add_argument(
+        '--images_root', type=str, default='', help='image_path', required=True
+    )
+    parser.add_argument(
+        '--save_path', type=str, default='', help='image_path', required=True
+    )
+    parser.add_argument(
+        '--fc_path', type=str, default='./fc_parameters.pth', help='fc_path'
+    )
+    parser.add_argument(
+        '--cal_detection_feat', action='store_true', help='cal_detection_feat'
+    )
+    parser.add_argument('--device', type=str, default='cuda:0', help='cuda:n or cpu')
+    args = parser.parse_args()
+
+    def print_options(parser, args):
+        message = ''
+        message += '----------------- Options ---------------\n'
+        for k, v in sorted(vars(args).items()):
+            comment = ''
+            default = parser.get_default(k)
+            if v != default:
+                comment = '\t[default: %s]' % str(default)
+            message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
+        message += '----------------- End -------------------'
+        print(message)
+
+    print_options(parser, args)
+    return args
+
+
+def get_image_files_in_directory(directory):
+    image_extensions = [
+        '.jpg',
+        '.jpeg',
+        '.png',
+        '.ppm',
+        '.bmp',
+        '.pgm',
+        '.tif',
+        '.tiff',
+        '.webp',
+    ]
+    image_files = []
+    for root, dirs, files in os.walk(directory, followlinks=True):
+        for file in files:
+            if os.path.splitext(file)[1].lower() in image_extensions:
+                absolute_path = os.path.join(root, file)
+                image_files.append(absolute_path)
+    return image_files
+
+
+if __name__ == '__main__':
+    opt = parse_args()
+    device = torch.device(opt.device)
+    assert os.path.exists(opt.images_root)
+
+    opt.images_root = os.path.abspath(opt.images_root)
+    opt.save_path = os.path.abspath(opt.save_path)
+    os.makedirs(opt.save_path, mode=0o777, exist_ok=True)
+
+    image_files = get_image_files_in_directory(opt.images_root)
+    print(f'len(image_files): {len(image_files)}')
+
+    clipmodel, processor = get_clip_model(
+        clip_name='openai/clip-vit-large-patch14', device=device
+    )
+    model, tokenizer = get_clipcap_model(opt.model_path, device=device)
+
+    for image_file in tqdm(image_files):
+        image_features = get_image_features(
+            image_file, clipmodel, processor, device=device
+        )
+        text = get_text(
+            image_features,
+            tokenizer,
+            model,
+            opt.fc_path,
+            opt.cal_detection_feat,
+            device=device,
+        )
+        text_save_path = os.path.splitext(
+            image_file.replace(opt.images_root, opt.save_path)
+        )[0]
+        os.makedirs(os.path.dirname(text_save_path), mode=0o777, exist_ok=True)
+        with open(f'{text_save_path}.txt', 'w') as file:
+            file.write(text)

detector_codes/C2P-CLIP-DeepfakeDetection-main/draw_tsne_kmean.py

@@ -0,0 +1,301 @@
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import argparse
+from random import shuffle
+from typing import Any, Callable, Optional, Tuple
+
+import numpy as np
+import PIL.Image
+import skimage.io as io
+import torch
+import torchvision.datasets as datasets
+from matplotlib import pyplot as plt
+from MulticoreTSNE import MulticoreTSNE as TSNE
+from PIL import ImageFile
+from torch.utils.data import DataLoader
+from transformers import CLIPProcessor
+from utils.logger import Progbar
+
+np.random.seed(123)
+# from clipcap import gg_text, get_model
+from decode_clipfeature_image import get_clip_model, get_clipcap_model, get_text
+from kmeans_pytorch import kmeans
+
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+IMG_EXTENSIONS = (
+    '.jpg',
+    '.jpeg',
+    '.png',
+    '.ppm',
+    '.bmp',
+    '.pgm',
+    '.tif',
+    '.tiff',
+    '.webp',
+)
+
+
+class dataset_folder(datasets.DatasetFolder):
+    def __init__(self, root: str, transform: Optional[Callable] = None):
+        super().__init__(root, transform=None, extensions=IMG_EXTENSIONS, loader=None)
+        self.processor = CLIPProcessor.from_pretrained('openai/clip-vit-large-patch14')
+
+    def __getitem__(self, index: int) -> Tuple[Any, Any]:
+        path, target = self.samples[index]
+        image = PIL.Image.fromarray(io.imread(path))  # .to(device)
+        sample = self.processor(images=image, return_tensors='pt')['pixel_values'][0]
+        return sample, target, path
+
+
+def collate_fn(batch):
+    batch = list(filter(lambda x: x is not None, batch))
+    return torch.utils.data.dataloader.default_collate(batch)
+
+
+def binary_dataset(root):
+    return dataset_folder(root)
+
+
+def generate_colors(num_colors):
+    # cmap = plt.cm.get_cmap('tab20')
+    cmap = plt.colormaps.get_cmap('tab20')
+    colors = cmap(range(num_colors))
+    return colors
+
+
+def tsne_vis(features, labels, draw_dir, opt, device):
+
+    clipcap_model, tokenizer = get_clipcap_model(
+        model_path='https://www.now61.com/f/Xljmi0/coco_prefix_latest.pt', device=device
+    )
+    all_labels = list(set(labels.reshape([-1])))
+    num_clusters = 3
+    all_cluster_centers = []
+    all_text = {}
+    for tmp_label in all_labels:
+        feature_ = features[labels == tmp_label]
+        cluster_ids_x, cluster_centers = kmeans(
+            X=torch.from_numpy(feature_).cuda(),
+            num_clusters=num_clusters,
+            distance='euclidean',
+            device=device,
+        )
+        all_cluster_centers.append(cluster_centers)
+        print('=' * 100)
+        print(opt.legend[tmp_label])
+        tmp_text = []
+        for index in range(num_clusters):
+            tmp_text.append(
+                get_text(
+                    cluster_centers[index].to(device),
+                    tokenizer,
+                    clipcap_model,
+                    fc_path='https://www.now61.com/f/qwvoH5/fc_parameters.pth',
+                    cal_detection_feat=False,
+                    device=device,
+                )
+            )
+            print(f'\n{tmp_text[-1]}')
+        all_text[str(tmp_label + len(all_labels))] = tmp_text
+        print('=' * 100)
+
+    all_cluster_centers = torch.cat(all_cluster_centers, 0).cpu().numpy()
+    features = np.concatenate((features, all_cluster_centers), axis=0)
+    new_label = []
+    for index, all_label in enumerate(all_labels):
+        new_label.extend([len(all_labels) + index] * num_clusters)
+    labels = np.concatenate((labels, np.array(new_label)), axis=0)
+
+    embedding_path = os.path.join(draw_dir, '{}_embedding.npy'.format(opt.save_name))
+
+    if opt.do_fit or not os.path.exists(embedding_path):
+        print('>>> t-SNE fitting')
+        tsne_model = TSNE(
+            n_jobs=64, perplexity=opt.perplexity, random_state=1024, learning_rate=1000
+        )
+        embeddings = tsne_model.fit_transform(features)
+        print('<<< fitting over')
+        np.save(embedding_path, embeddings)
+    else:
+        embeddings = np.load(embedding_path)
+
+    index = [i for i in range(len(embeddings))]
+    shuffle(index)
+    embeddings = [embeddings[index[i]] for i in range(len(index))]
+    labels = [labels[index[i]] for i in range(len(index))]
+    embeddings = np.array(embeddings)
+
+    print('>>> draw image')
+    vis_x = embeddings[:, 0]
+    vis_y = embeddings[:, 1]
+    plt.figure(figsize=(35, 35))
+    plt.rcParams['figure.dpi'] = 1000
+    colors = generate_colors(20)
+    num_classes = len(set(labels))
+
+    for i in range(num_classes):
+        s = 1000 if i > 5 else 20
+        marker = '*' if i > 5 else 'o'
+        color = colors[i]
+        class_index = [j for j, v in enumerate(labels) if v == i]
+        plt.scatter(
+            vis_x[class_index], vis_y[class_index], s=s, color=color, marker=marker
+        )
+
+        if i > 5 and opt.draw_text:
+            texthighs = [0] * len(vis_y[class_index])
+            sorted_A = sorted(
+                enumerate(vis_y[class_index]), key=lambda x: x[1], reverse=False
+            )
+            for rank, (index, value) in enumerate(sorted_A):
+                texthighs[index] = (rank + 1) * 40
+            for tx, ty, ttext, texthigh in zip(
+                vis_x[class_index], vis_y[class_index], all_text[str(i)], texthighs
+            ):
+                fc = colors[i - num_classes // 2]
+                plt.annotate(
+                    f' {ttext} ',
+                    xy=(tx, ty),
+                    xycoords='data',
+                    xytext=(100, texthigh),
+                    fontsize=30,
+                    textcoords='offset points',
+                    color='white',
+                    va='center',
+                    ha='center',
+                    weight='bold',
+                    bbox=dict(boxstyle='round', fc=fc, ec='none'),
+                    arrowprops=dict(
+                        connectionstyle='angle3,angleA=0,angleB=90',
+                        arrowstyle='wedge, tail_width=1.',
+                        fc=fc,
+                        ec='none',
+                        patchA=None,
+                    ),
+                )
+    if opt.draw_text:
+        img_path = os.path.join(draw_dir, '{}_draw-text_tsne.png'.format(opt.save_name))
+    else:
+        img_path = os.path.join(draw_dir, '{}_tsne.png'.format(opt.save_name))
+    plt.xticks([])
+    plt.yticks([])
+    legend = plt.legend(opt.legend, prop={'size': 35})
+    for handle in legend.legend_handles:
+        handle.set_sizes([300])
+    plt.show()
+    plt.savefig(img_path, bbox_inches='tight', pad_inches=0.1)
+    print('<<<save image')
+
+
+def extract_feature(model, draw_loader, device):
+
+    fc_path = 'https://www.now61.com/f/qwvoH5/fc_parameters.pth'
+    mod = (
+        torch.hub.load_state_dict_from_url(fc_path, map_location='cpu', progress=True)
+        if fc_path.startswith('http')
+        else torch.load(fc_path, map_location='cpu')
+    )
+    weight, bias = mod['fc.weight'].to(device), mod['fc.bias'].to(device)
+
+    features = None
+    model.eval()
+    progbar = Progbar(len(draw_loader), stateful_metrics=['run-type'])
+    with torch.no_grad():
+        for _, batch in enumerate(draw_loader):
+            input_img_batch, label_batch, path_batch = batch
+            input_img = input_img_batch.to(device)
+            label = label_batch.reshape((-1)).to(device)
+
+            feature = model.get_image_features(input_img)
+            feature = torch.mul(feature, weight) + bias
+            feature /= feature.norm(2, dim=-1, keepdim=True)
+            if features is None:
+                features = feature.cpu().numpy()
+                gt_labels = label
+            else:
+                gt_labels = torch.cat([gt_labels, label])
+                features = np.vstack((features, feature.cpu().numpy()))
+
+            progbar.add(1, values=[('run-type', 'extract feature')])
+
+    gt_labels = gt_labels.cpu().numpy()
+
+    return features, gt_labels
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='draw tsne')
+    parser.add_argument('--draw_data_path', type=str, required=True)
+    parser.add_argument('--image_path', type=str, help='image_path', default='')
+    parser.add_argument('--device', default='cuda:0', type=str, help='cuda:n or cpu')
+    parser.add_argument(
+        '--do_extract',
+        action='store_true',
+        default=False,
+        help='whether to extract features',
+    )
+    parser.add_argument(
+        '--do_fit', action='store_true', default=False, help='whether to fit tsne model'
+    )
+    parser.add_argument('--save_name', default='cross_all', type=str)
+    parser.add_argument('--legend', nargs='+', help='legend')
+    parser.add_argument('--draw_text', default=0, type=int)
+    parser.add_argument('--perplexity', default=20, type=int)
+    args = parser.parse_args()
+
+    def print_options(parser, args):
+        message = ''
+        message += '----------------- Options ---------------\n'
+        for k, v in sorted(vars(args).items()):
+            comment = ''
+            default = parser.get_default(k)
+            if v != default:
+                comment = '\t[default: %s]' % str(default)
+            message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
+        message += '----------------- End -------------------'
+        print(message)
+
+    print_options(parser, args)
+    return args
+
+
+if __name__ == '__main__':
+    opt = parse_args()
+    device = torch.device(opt.device)
+
+    draw_dir = os.path.join(
+        os.path.splitext(opt.draw_data_path)[0], 'tsne-' + opt.save_name
+    )
+    os.makedirs(draw_dir, exist_ok=True)
+    feature_path = os.path.join(draw_dir, '{}_features.npy'.format(opt.save_name))
+    label_path = os.path.join(draw_dir, '{}_labels.npy'.format(opt.save_name))
+    print('draw dir: %s' % draw_dir)
+
+    clipmodel, processor = get_clip_model(
+        clip_name='openai/clip-vit-large-patch14', device=device
+    )
+
+    draw_loader = DataLoader(
+        dataset=binary_dataset(opt.image_path),
+        num_workers=8,
+        batch_size=1,
+        pin_memory=True,
+        shuffle=False,
+        drop_last=False,
+        collate_fn=collate_fn,
+    )
+
+    if opt.do_extract or not os.path.exists(feature_path):
+        features, gt_labels = extract_feature(clipmodel, draw_loader, device)
+        np.save(feature_path, features)
+        np.save(label_path, gt_labels)
+    else:
+        features = np.load(feature_path)
+        gt_labels = np.load(label_path)
+
+    print('labels:', gt_labels.shape, 'features:', features.shape)
+    tsne_vis(features, gt_labels, draw_dir, opt, device)
+
+# CUDA_VISIBLE_DEVICES=1 python draw_tsne_kmean.py --draw_data_path A_tsne_png_20240812  --image_path ../stylegan_tsne_data  --save_name stylegan_test   --legend stylegan-bedroom-real  stylegan-bedroom-fake  stylegan-car-real  stylegan-car-fake  stylegan-cat-real  stylegan-cat-fake  --do_extract --do_fit --draw_text 0

detector_codes/C2P-CLIP-DeepfakeDetection-main/inference.py

@@ -0,0 +1,164 @@
+import argparse
+import os
+import random
+import time
+import warnings
+
+import numpy as np
+import torch
+import torch.nn as nn
+from data import create_dataloader
+from sklearn.metrics import (
+    accuracy_score,
+    average_precision_score,
+)
+from transformers import CLIPModel
+
+warnings.filterwarnings('ignore')
+
+
+def seed_torch(seed=1029):
+    random.seed(seed)
+    os.environ['PYTHONHASHSEED'] = str(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)  # if you are using multi-GPU.
+    torch.backends.cudnn.benchmark = False
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.enabled = False
+
+
+seed_torch(123)
+
+
+class C2P_CLIP(nn.Module):
+    def __init__(self, name='openai/clip-vit-large-patch14', num_classes=1):
+        super(C2P_CLIP, self).__init__()
+        self.model = CLIPModel.from_pretrained(name)
+        del self.model.text_model
+        del self.model.text_projection
+        del self.model.logit_scale
+
+        self.model.vision_model.requires_grad_(False)
+        self.model.visual_projection.requires_grad_(False)
+        self.model.fc = nn.Linear(768, num_classes)
+        torch.nn.init.normal_(self.model.fc.weight.data, 0.0, 0.02)
+
+    def encode_image(self, img):
+        vision_outputs = self.model.vision_model(
+            pixel_values=img,
+            output_attentions=self.model.config.output_attentions,
+            output_hidden_states=self.model.config.output_hidden_states,
+            return_dict=self.model.config.return_dict,
+        )
+        pooled_output = vision_outputs[1]  # pooled_output
+        image_features = self.model.visual_projection(pooled_output)
+        return image_features
+
+    def forward(self, img):
+        # tmp = x; print(f'x: {tmp.shape}, max: {tmp.max()}, min: {tmp.min()}, mean: {tmp.mean()}')
+        image_embeds = self.encode_image(img)
+        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
+        return self.model.fc(image_embeds)
+
+
+def printSet(set_str):
+    set_str = str(set_str)
+    num = len(set_str)
+    print('=' * num * 3)
+    print(' ' * num + set_str)
+    print('=' * num * 3)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='test C2P-CLIP')
+    parser.add_argument('--loadSize', type=int, default=224)
+    parser.add_argument('--cropSize', type=int, default=224)
+    parser.add_argument('--batch_size', type=int, default=64)
+    parser.add_argument(
+        '--dataroot',
+        type=str,
+        default='/opt/data/private/tcc/data/data/DeepfakeDetection/datasets/ForenSynths_train_val_19test/test/',
+    )
+    parser.add_argument(
+        '--model_path',
+        type=str,
+        default='https://www.now61.com/f/95OefW/C2P_CLIP_release_20240901.zip',
+    )
+    args = parser.parse_args()
+
+    def print_options(parser, args):
+        message = ''
+        message += '----------------- Options ---------------\n'
+        for k, v in sorted(vars(args).items()):
+            comment = ''
+            default = parser.get_default(k)
+            if v != default:
+                comment = '\t[default: %s]' % str(default)
+            message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
+        message += '----------------- End -------------------'
+        print(message)
+
+    print_options(parser, args)
+    return args
+
+
+if __name__ == '__main__':
+    opt = parse_args()
+    DetectionTests = {
+        '19Test': {
+            'dataroot': opt.dataroot,
+            'no_resize': False,
+            'no_crop': False,
+        },
+    }
+
+    # state_dict = torch.hub._legacy_zip_load( 'C2P_CLIP_release_20240901.zip', './', map_location = "cpu", weights_only= False)
+    state_dict = torch.hub.load_state_dict_from_url(
+        opt.model_path, map_location='cpu', progress=True
+    )
+    model = C2P_CLIP(name='openai/clip-vit-large-patch14', num_classes=1)
+    model.load_state_dict(state_dict, strict=True)
+    model.cuda()
+    model.eval()
+
+    for testSet in DetectionTests.keys():
+        dataroot = DetectionTests[testSet]['dataroot']
+        printSet(testSet)
+        accs = []
+        aps = []
+        print(time.strftime('%Y_%m_%d_%H_%M_%S', time.localtime()))
+        for v_id, val in enumerate(os.listdir(dataroot)):
+            opt.dataroot = '{}/{}'.format(dataroot, val)
+            opt.classes = ''  # os.listdir(opt.dataroot) if multiclass[v_id] else ['']
+            opt.no_resize = DetectionTests[testSet]['no_resize']
+            opt.no_crop = DetectionTests[testSet]['no_crop']
+            data_loader = create_dataloader(opt)
+            with torch.no_grad():
+                y_true, y_pred = [], []
+                for img, label, path in data_loader:
+                    y_pred.extend(model(img.cuda()).sigmoid().flatten().tolist())
+                    y_true.extend(label.flatten().tolist())
+            y_true, y_pred = np.array(y_true), np.array(y_pred)
+            r_acc = accuracy_score(y_true[y_true == 0], y_pred[y_true == 0] > 0.5)
+            f_acc = accuracy_score(y_true[y_true == 1], y_pred[y_true == 1] > 0.5)
+            acc = accuracy_score(y_true, y_pred > 0.5)
+            ap = average_precision_score(y_true, y_pred)
+
+            accs.append(acc)
+            aps.append(ap)
+            print(
+                '({} {:12}) acc: {:.2f}; ap: {:.2f}'.format(
+                    v_id, val, acc * 100, ap * 100
+                )
+            )
+        print(
+            '({} {:10}) acc: {:.2f}; ap: {:.2f}'.format(
+                v_id + 1,
+                'Mean',
+                np.array(accs).mean() * 100,
+                np.array(aps).mean() * 100,
+            )
+        )
+        print('*' * 25)

detector_codes/C2P-CLIP-DeepfakeDetection-main/networks/base_model.py

@@ -0,0 +1,108 @@
+# from pix2pix
+import os
+
+import torch
+import torch.nn as nn
+from torch.nn import init
+
+
+class BaseModel(nn.Module):
+    def __init__(self, opt):
+        super(BaseModel, self).__init__()
+        self.opt = opt
+        self.total_steps = 0
+        self.isTrain = opt.isTrain
+        self.lr = opt.lr
+        self.save_dir = os.path.join(opt.checkpoints_dir, opt.name)
+        self.device = (
+            torch.device('cuda:{}'.format(opt.gpu_ids[0]))
+            if opt.gpu_ids
+            else torch.device('cpu')
+        )
+
+    def save_networks(self, epoch):
+        save_filename = 'model_epoch_%s.pth' % epoch
+        save_path = os.path.join(self.save_dir, save_filename)
+
+        # serialize model and optimizer to dict
+        state_dict = {
+            'model': self.model.state_dict(),
+            # 'optimizer' : self.optimizer.state_dict(),
+            'total_steps': self.total_steps,
+        }
+        torch.save(state_dict, save_path)
+        try:
+            try:
+                savemodel = self.model.module
+            except:
+                savemodel = self.model
+            #            savemodel.model.vision_model = savemodel.vision_tower_lora.merge_and_unload()
+            save_path2 = os.path.join(self.save_dir, f'model_epoch_{epoch}')
+            os.makedirs(save_path2, mode=0o777, exist_ok=True)
+            savemodel.model.save_pretrained(save_path2, safe_serialization=False)
+        except:
+            pass
+
+    # load models from the disk
+    def load_networks(self, epoch):
+        load_filename = 'model_epoch_%s.pth' % epoch
+        load_path = os.path.join(self.save_dir, load_filename)
+
+        print('loading the model from %s' % load_path)
+        # if you are using PyTorch newer than 0.4 (e.g., built from
+        # GitHub source), you can remove str() on self.device
+        state_dict = torch.load(load_path, map_location=self.device)
+        if hasattr(state_dict, '_metadata'):
+            del state_dict._metadata
+
+        self.model.load_state_dict(state_dict['model'])
+        self.total_steps = state_dict['total_steps']
+
+        if self.isTrain and not self.opt.new_optim:
+            self.optimizer.load_state_dict(state_dict['optimizer'])
+            ### move optimizer state to GPU
+            for state in self.optimizer.state.values():
+                for k, v in state.items():
+                    if torch.is_tensor(v):
+                        state[k] = v.to(self.device)
+
+            for g in self.optimizer.param_groups:
+                g['lr'] = self.opt.lr
+
+    def eval(self):
+        self.model.eval()
+
+    def train(self):
+        self.model.train()
+
+    def test(self):
+        with torch.no_grad():
+            self.forward()
+
+
+def init_weights(net, init_type='normal', gain=0.02):
+    def init_func(m):
+        classname = m.__class__.__name__
+        if hasattr(m, 'weight') and (
+            classname.find('Conv') != -1 or classname.find('Linear') != -1
+        ):
+            if init_type == 'normal':
+                init.normal_(m.weight.data, 0.0, gain)
+            elif init_type == 'xavier':
+                init.xavier_normal_(m.weight.data, gain=gain)
+            elif init_type == 'kaiming':
+                init.kaiming_normal_(m.weight.data, a=0, mode='fan_in')
+            elif init_type == 'orthogonal':
+                init.orthogonal_(m.weight.data, gain=gain)
+            else:
+                raise NotImplementedError(
+                    'initialization method [%s] is not implemented' % init_type
+                )
+            if hasattr(m, 'bias') and m.bias is not None:
+                init.constant_(m.bias.data, 0.0)
+        elif classname.find('BatchNorm2d') != -1:
+            init.normal_(m.weight.data, 1.0, gain)
+            init.constant_(m.bias.data, 0.0)
+
+    print('initialize network with %s' % init_type)
+    net.apply(init_func)

detector_codes/C2P-CLIP-DeepfakeDetection-main/networks/c2p_clip.py

@@ -0,0 +1,59 @@
+import torch
+import torch.nn as nn
+from peft import LoraConfig, get_peft_model
+from transformers import CLIPModel
+
+
+class C2P_CLIP_Model(nn.Module):
+    def __init__(
+        self,
+        name='openai/clip-vit-large-patch14',
+        num_classes=1,
+        lora_r=16,
+        lora_alpha=32,
+        lora_dropout=0.05,
+        hf_token=None,
+    ):
+        super(C2P_CLIP_Model, self).__init__()
+
+        self.model = CLIPModel.from_pretrained(name, token=hf_token)
+        del self.model.text_model
+        del self.model.text_projection
+        del self.model.logit_scale
+
+        self.vision_tower = self.model.vision_model
+        self.vision_tower.requires_grad_(False)
+        self.model.visual_projection.requires_grad_(False)
+
+        lora_config = LoraConfig(
+            r=lora_r,
+            lora_alpha=lora_alpha,
+            target_modules=['q_proj', 'k_proj', 'v_proj'],
+            lora_dropout=lora_dropout,
+            bias='none',
+        )
+        self.vision_tower_lora = get_peft_model(self.vision_tower, lora_config)
+
+        self.model.fc = nn.Linear(768, num_classes)
+        torch.nn.init.normal_(self.model.fc.weight.data, 0.0, 0.02)
+
+    def encode_image(self, img):
+        vision_outputs = self.vision_tower_lora(
+            pixel_values=img,
+            output_attentions=self.model.config.output_attentions,
+            output_hidden_states=self.model.config.output_hidden_states,
+            return_dict=self.model.config.return_dict,
+        )
+        pooled_output = vision_outputs[1]  # pooled_output
+        image_features = self.model.visual_projection(pooled_output)
+        return image_features
+
+    def forward(self, img):
+        image_embeds = self.encode_image(img)
+        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
+        return self.model.fc(image_embeds)
+
+    def detect(self, img):
+        with torch.no_grad():
+            output = self.forward(img)
+            return torch.sigmoid(output).squeeze(1)

detector_codes/C2P-CLIP-DeepfakeDetection-main/networks/decode_clipfeature_image.py

@@ -0,0 +1,260 @@
+# conda -n create clip-text-decoder python=3.8.5
+import argparse
+import warnings
+from typing import Optional, Tuple
+
+import PIL.Image
+import skimage.io as io
+import torch  # 2.3.1+cu118 conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia
+import torch.nn.functional as nnf
+from torch import nn
+from transformers import (  # 4.25.0
+    CLIPModel,
+    CLIPProcessor,
+    GPT2LMHeadModel,
+    GPT2Tokenizer,
+)
+
+warnings.filterwarnings('ignore')
+
+N = type(None)
+T = torch.Tensor
+D = torch.device
+
+
+class MLP(nn.Module):
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.model(x)
+
+    def __init__(self, sizes: Tuple[int, ...], bias=True, act=nn.Tanh):
+        super(MLP, self).__init__()
+        layers = []
+        for i in range(len(sizes) - 1):
+            layers.append(nn.Linear(sizes[i], sizes[i + 1], bias=bias))
+            if i < len(sizes) - 2:
+                layers.append(act())
+        self.model = nn.Sequential(*layers)
+
+
+class ClipCaptionModel(nn.Module):
+    # @functools.lru_cache #FIXME
+    def get_dummy_token(self, batch_size: int, device: D) -> T:
+        return torch.zeros(
+            batch_size, self.prefix_length, dtype=torch.int64, device=device
+        )
+
+    def forward(
+        self, tokens: T, prefix: T, mask: Optional[T] = None, labels: Optional[T] = None
+    ):
+        embedding_text = self.gpt.transformer.wte(tokens)
+        prefix_projections = self.clip_project(prefix).view(
+            -1, self.prefix_length, self.gpt_embedding_size
+        )
+        embedding_cat = torch.cat((prefix_projections, embedding_text), dim=1)
+        if labels is not None:
+            dummy_token = self.get_dummy_token(tokens.shape[0], tokens.device)
+            labels = torch.cat((dummy_token, tokens), dim=1)
+        out = self.gpt(inputs_embeds=embedding_cat, labels=labels, attention_mask=mask)
+        return out
+
+    def __init__(self, prefix_length: int, prefix_size: int = 512):
+        super(ClipCaptionModel, self).__init__()
+        self.prefix_length = prefix_length
+        self.gpt = GPT2LMHeadModel.from_pretrained('gpt2')
+        self.gpt_embedding_size = self.gpt.transformer.wte.weight.shape[1]
+        self.clip_project = MLP(
+            (
+                prefix_size,
+                (self.gpt_embedding_size * prefix_length) // 2,
+                self.gpt_embedding_size * prefix_length,
+            )
+        )
+
+
+def generate2(
+    model,
+    tokenizer,
+    tokens=None,
+    prompt=None,
+    embed=None,
+    entry_count=1,
+    entry_length=67,
+    top_p=0.8,
+    temperature=1.0,
+    stop_token: str = '.',
+):
+    model.eval()
+    generated_num = 0
+    generated_list = []
+    stop_token_index = tokenizer.encode(stop_token)[0]
+    filter_value = -float('Inf')
+    device = next(model.parameters()).device
+
+    with torch.no_grad():
+        for entry_idx in range(entry_count):
+            if embed is not None:
+                generated = embed
+            else:
+                if tokens is None:
+                    tokens = torch.tensor(tokenizer.encode(prompt))
+                    tokens = tokens.unsqueeze(0).to(device)
+                generated = model.gpt.transformer.wte(tokens)
+
+            for i in range(entry_length):
+                outputs = model.gpt(inputs_embeds=generated)
+                logits = outputs.logits
+                logits = logits[:, -1, :] / (temperature if temperature > 0 else 1.0)
+                sorted_logits, sorted_indices = torch.sort(logits, descending=True)
+                cumulative_probs = torch.cumsum(
+                    nnf.softmax(sorted_logits, dim=-1), dim=-1
+                )
+                sorted_indices_to_remove = cumulative_probs > top_p
+                sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[
+                    ..., :-1
+                ].clone()
+                sorted_indices_to_remove[..., 0] = 0
+
+                indices_to_remove = sorted_indices[sorted_indices_to_remove]
+                logits[:, indices_to_remove] = filter_value
+                next_token = torch.argmax(logits, -1).unsqueeze(0)
+                next_token_embed = model.gpt.transformer.wte(next_token)
+                if tokens is None:
+                    tokens = next_token
+                else:
+                    tokens = torch.cat((tokens, next_token), dim=1)
+                generated = torch.cat((generated, next_token_embed), dim=1)
+                if stop_token_index == next_token.item():
+                    break
+
+            output_list = list(tokens.squeeze().cpu().numpy())
+            output_text = tokenizer.decode(output_list)
+            generated_list.append(output_text)
+
+    return generated_list[0]
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='decode detection feature to text')
+    parser.add_argument('--prefix_length', type=int, default=10)
+    parser.add_argument(
+        '--model_path',
+        type=str,
+        default='https://www.now61.com/f/Xljmi0/coco_prefix_latest.pt',
+        help='model_path',
+    )
+    parser.add_argument(
+        '--image_path', type=str, default='', help='image_path', required=True
+    )
+    parser.add_argument(
+        '--fc_path',
+        type=str,
+        default='https://www.now61.com/f/qwvoH5/fc_parameters.pth',
+        help='fc_path',
+    )
+    parser.add_argument(
+        '--cal_detection_feat', action='store_true', help='cal_detection_feat'
+    )
+    parser.add_argument('--device', type=str, default='cuda:0', help='cuda:n or cpu')
+    args = parser.parse_args()
+
+    def print_options(parser, args):
+        message = ''
+        message += '----------------- Options ---------------\n'
+        for k, v in sorted(vars(args).items()):
+            comment = ''
+            default = parser.get_default(k)
+            if v != default:
+                comment = '\t[default: %s]' % str(default)
+            message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
+        message += '----------------- End -------------------'
+        print(message)
+
+    print_options(parser, args)
+    return args
+
+
+def get_text(
+    image_features,
+    tokenizer,
+    model,
+    fc_path,
+    cal_detection_feat=True,
+    prefix_length=10,
+    device='cpu',
+):
+    mod = (
+        torch.hub.load_state_dict_from_url(fc_path, map_location='cpu', progress=True)
+        if fc_path.startswith('http')
+        else torch.load(fc_path, map_location='cpu')
+    )
+    weight, bias = mod['fc.weight'].to(device), mod['fc.bias'].to(device)
+    with torch.no_grad():
+        prob = nnf.linear(image_features, weight, bias).sigmoid().cpu().numpy()[0][0]
+        dict_prob = {False: 'Fake', True: 'Real'}
+        # print( f'\nPredicted prob: {prob}, {dict_prob[prob<0.5]}' )
+        # tmp=image_features;print(f'image_features: {tmp.shape}, max: {tmp.max()}, min: {tmp.min()}, mean: {tmp.mean()}')
+        if cal_detection_feat:
+            image_features = torch.mul(image_features, weight) + bias
+        image_features /= image_features.norm(2, dim=-1, keepdim=True)
+        prefix_embed = model.clip_project(image_features).reshape(1, prefix_length, -1)
+        generated_text_prefix = generate2(model, tokenizer, embed=prefix_embed)
+    return generated_text_prefix
+
+
+def get_clip_model(clip_name='openai/clip-vit-large-patch14', device='cpu'):
+    clipmodel = CLIPModel.from_pretrained(clip_name)
+    processor = CLIPProcessor.from_pretrained(clip_name)
+    del clipmodel.text_model
+    del clipmodel.text_projection
+    del clipmodel.logit_scale
+    clipmodel = clipmodel.to(device)
+    return clipmodel, processor
+
+
+def get_clipcap_model(model_path, prefix_length=10, device='cpu'):
+    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
+    model = ClipCaptionModel(prefix_length, prefix_size=768)
+    pretrained = (
+        torch.hub.load_state_dict_from_url(
+            model_path, map_location='cpu', progress=True
+        )
+        if model_path.startswith('http')
+        else torch.load(model_path, map_location='cpu')
+    )
+    model.load_state_dict(pretrained)
+    assert pretrained.keys() == model.state_dict().keys()
+    model = model.eval()
+    model = model.to(device)
+    return model, tokenizer
+
+
+def get_image_features(image_path, clipmodel, processor, device='cpu'):
+    with torch.no_grad():
+        image = PIL.Image.fromarray(io.imread(image_path))
+        inputs = processor(images=image, return_tensors='pt')
+        inputs['pixel_values'] = inputs['pixel_values'].to(device)
+        image_features = clipmodel.get_image_features(**inputs)
+    return image_features
+
+
+if __name__ == '__main__':
+    opt = parse_args()
+    device = torch.device(opt.device)
+
+    clipmodel, processor = get_clip_model(
+        clip_name='openai/clip-vit-large-patch14', device=device
+    )
+    model, tokenizer = get_clipcap_model(opt.model_path, device=device)
+
+    image_features = get_image_features(
+        opt.image_path, clipmodel, processor, device=device
+    )
+    text = get_text(
+        image_features,
+        tokenizer,
+        model,
+        opt.fc_path,
+        opt.cal_detection_feat,
+        device=device,
+    )
+    print(text)

detector_codes/C2P-CLIP-DeepfakeDetection-main/networks/trainer.py

@@ -0,0 +1,193 @@
+import torch
+import torch.nn as nn
+from peft import LoraConfig, get_peft_model
+from transformers import CLIPModel, CLIPProcessor
+
+from networks.base_model import BaseModel
+
+
+class CLIPModel_lora(nn.Module):
+    def __init__(
+        self,
+        name='openai/clip-vit-large-patch14-336',
+        num_classes=1,
+        lora_r=16,
+        lora_alpha=32,
+        lora_dropout=0.05,
+    ):
+        super(CLIPModel_lora, self).__init__()
+        self.model = CLIPModel.from_pretrained(name)
+        self.processor = CLIPProcessor.from_pretrained(name)
+        self.vision_tower = self.model.vision_model
+        self.vision_tower.requires_grad_(False)
+        self.model.text_model.requires_grad_(False)
+        self.model.visual_projection.requires_grad_(False)
+        self.model.text_projection.requires_grad_(False)
+        self.contrastive_loss = nn.CrossEntropyLoss()
+        self.model.logit_scale.requires_grad_(False)
+        lora_config = LoraConfig(
+            r=lora_r,
+            lora_alpha=lora_alpha,
+            target_modules=['q_proj', 'k_proj', 'v_proj'],
+            lora_dropout=lora_dropout,
+            bias='none',
+        )
+        self.vision_tower_lora = get_peft_model(self.vision_tower, lora_config)
+        self.model.fc = nn.Linear(768, num_classes)
+        torch.nn.init.normal_(self.model.fc.weight.data, 0.0, 0.02)
+
+    def encode_text(self, input_ids, attention_mask):
+        text_outputs = self.model.text_model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=None,
+            output_attentions=self.model.config.output_attentions,
+            output_hidden_states=self.model.config.output_hidden_states,
+            return_dict=self.model.config.return_dict,
+        )
+        text_embeds = text_outputs[1]
+        text_embeds = self.model.text_projection(text_embeds)
+        return text_embeds
+
+    def encode_image(self, img):
+        vision_outputs = self.vision_tower_lora(
+            pixel_values=img,
+            output_attentions=self.model.config.output_attentions,
+            output_hidden_states=self.model.config.output_hidden_states,
+            return_dict=self.model.config.return_dict,
+        )
+        pooled_output = vision_outputs[1]  # pooled_output
+        image_features = self.model.visual_projection(pooled_output)
+        return image_features
+
+    def forward(self, img, input_ids, attention_mask, cla=False):
+        # tmp = x; print(f'x: {tmp.shape}, max: {tmp.max()}, min: {tmp.min()}, mean: {tmp.mean()}')
+
+        image_embeds = self.encode_image(img)
+
+        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
+        classhead = self.model.fc(image_embeds)
+        if cla:
+            return classhead
+
+        text_embeds = self.encode_text(input_ids, attention_mask)
+        text_embeds = text_embeds / text_embeds.norm(p=2, dim=-1, keepdim=True)
+
+        logits_per_text = (
+            torch.matmul(text_embeds, image_embeds.t()) * self.model.logit_scale.exp()
+        )
+        logits_per_image = logits_per_text.t()
+
+        return logits_per_image, classhead.squeeze(1)
+
+    def forward_eval(self, img):
+        image_embeds = self.encode_image(img)
+        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
+        classhead = self.model.fc(image_embeds)
+        return classhead
+
+
+class Trainer(BaseModel):
+    def name(self):
+        return 'Trainer'
+
+    def __init__(self, opt):
+        super(Trainer, self).__init__(opt)
+
+        self.delr = opt.delr
+        self.claloss = opt.claloss
+
+        self.printOne = 1
+        self.model = CLIPModel_lora(
+            name=opt.clip,
+            lora_r=opt.lora_r,
+            lora_alpha=opt.lora_alpha,
+            lora_dropout=opt.lora_dropout,
+        )
+
+        net_params = sum(map(lambda x: x.numel(), self.model.model.parameters()))
+
+        print(f'Model parameters {net_params:,d}')
+
+        if self.isTrain:
+            self.loss_fn = nn.BCEWithLogitsLoss()
+            if opt.optim == 'adam':
+                self.optimizer = torch.optim.Adam(
+                    filter(lambda p: p.requires_grad, self.model.parameters()),
+                    lr=opt.lr,
+                    betas=(opt.beta1, 0.999),
+                )
+            elif opt.optim == 'sgd':
+                self.optimizer = torch.optim.SGD(
+                    self.model.parameters(), lr=opt.lr, momentum=0.0, weight_decay=0
+                )
+            elif opt.optim == 'adamw':
+                self.optimizer = torch.optim.AdamW(
+                    filter(lambda p: p.requires_grad, self.model.parameters()),
+                    lr=opt.lr,
+                    weight_decay=0.05,
+                    betas=(opt.beta1, 0.999),
+                    eps=1e-8,
+                )
+            else:
+                raise ValueError('optim should be [adam, sgd]')
+
+        if not self.isTrain or opt.continue_train:
+            self.load_networks(opt.epoch)
+
+        self.model = nn.DataParallel(self.model).cuda()
+
+    def adjust_learning_rate(self, min_lr=1e-6):
+        for param_group in self.optimizer.param_groups:
+            param_group['lr'] *= self.delr
+            if param_group['lr'] < min_lr:
+                return False
+        self.lr = param_group['lr']
+        print('*' * 25)
+        print(
+            f'Changing lr from {param_group["lr"] / self.delr} to {param_group["lr"]} with delr {self.delr}'
+        )
+        print('*' * 25)
+        return True
+
+    def set_input(self, input):
+        self.input = input[1].cuda()
+        self.text = input[2]
+        self.input_ids = input[3].cuda()
+        self.attention_mask = input[4].cuda()
+        self.label = input[5].cuda().float()
+
+    def forward(self):
+        self.output, self.classhead = self.model(
+            self.input, self.input_ids, self.attention_mask
+        )
+
+    def contrastive_loss(self, logits: torch.Tensor) -> torch.Tensor:
+
+        caption_loss = nn.functional.cross_entropy(
+            logits, torch.arange(len(logits), device=logits.device)
+        )
+        image_loss = nn.functional.cross_entropy(
+            logits.t(), torch.arange(len(logits), device=logits.device)
+        )
+        return (caption_loss + image_loss) / 2.0
+
+    def get_loss(self):
+        return self.model.clip_loss_input(self.input, self.text)
+
+    def optimize_parameters(self):
+        self.forward()
+
+        self.loss1 = sum(
+            [
+                self.contrastive_loss(output)
+                for output in torch.split(self.output, self.output.shape[1], dim=0)
+            ]
+        )
+        self.loss2 = self.claloss * self.loss_fn(self.classhead, self.label)
+        self.loss = self.loss1 + self.loss2
+        # self.loss1, self.loss2 = 0.0, 0.0
+        # self.loss  = self.loss_fn(self.classhead, self.label)
+        self.optimizer.zero_grad()
+        self.loss.backward()
+        self.optimizer.step()

detector_codes/C2P-CLIP-DeepfakeDetection-main/options/base_options.py

@@ -0,0 +1,131 @@
+import argparse
+import os
+import time
+import utils.util as util
+import torch
+#import models
+#import data
+ 
+
+class BaseOptions():
+    def __init__(self):
+        self.initialized = False
+
+    def initialize(self, parser):
+        parser.add_argument('--mode', default='binary')
+        parser.add_argument('--arch', type=str, default='res50', help='architecture for binary classification')
+
+        # data augmentation
+        parser.add_argument('--rz_interp',       default='bilinear')
+        parser.add_argument('--blur_prob',       type=float, default=0)
+        parser.add_argument('--blur_sig',        default='0.5')
+        parser.add_argument('--jpg_prob',        type=float, default=0)
+        parser.add_argument('--jpg_method',      default='cv2')
+        parser.add_argument('--jpg_qual',        default='75')
+
+        parser.add_argument('--dataroot',        default='./dataset/', help='path to images (should have subfolders trainA, trainB, valA, valB, etc)')
+        parser.add_argument('--textroot',        default='./Genimage_CNNDetection_CLIP_prefix_caption/', help='path to texts')
+
+        parser.add_argument('--classes',         default='', help='which classes to use, separated by comma. If empty, use all subfolders of dataroot')
+        parser.add_argument('--class_bal',       action='store_true')
+        parser.add_argument('--batch_size',      type=int, default=64, help='input batch size')
+        parser.add_argument('--loadSize',        type=int, default=256, help='scale images to this size')
+        parser.add_argument('--cropSize',        type=int, default=224, help='then crop to this size')
+        parser.add_argument('--gpu_ids',         type=str, default='0', help='gpu ids: e.g. 0  0,1,2, 0,2. use -1 for CPU')
+        parser.add_argument('--name',            type=str, default='experiment_name', help='name of the experiment. It decides where to store samples and models')
+        parser.add_argument('--epoch',           type=str, default='latest', help='which epoch to load? set to latest to use latest cached model')
+        parser.add_argument('--num_threads',     type=int, default=8, help='# threads for loading data')
+        parser.add_argument('--checkpoints_dir', type=str, default='./checkpoints', help='models are saved here')
+        parser.add_argument('--serial_batches',  action='store_true', help='if true, takes images in order to make batches, otherwise takes them randomly')
+        parser.add_argument('--resize_or_crop',  type=str, default='scale_and_crop', help='scaling and cropping of images at load time [resize_and_crop|crop|scale_width|scale_width_and_crop|none]')
+        parser.add_argument('--no_flip',         action='store_true', help='if specified, do not flip the images for data augmentation')
+        parser.add_argument('--init_type',       type=str, default='normal', help='network initialization [normal|xavier|kaiming|orthogonal]')
+        parser.add_argument('--init_gain',       type=float, default=0.02, help='scaling factor for normal, xavier and orthogonal.')
+        parser.add_argument('--suffix',          type=str,  default='', help='customized suffix: opt.name = opt.name + suffix: e.g., {model}_{netG}_size{loadSize}')
+        parser.add_argument('--delr_freq',       type=int, default=20, help='frequency of change lr')
+        parser.add_argument('--delr',            type=float, default=0.8, help='delr')
+        parser.add_argument('--seed',            type=int, default=123, help='seed')
+        parser.add_argument('--clip',            type=str, default='./clip-vit-large-patch14/', help='clip path')
+        parser.add_argument('--claloss',         type=float, default=0.5, help='fixed num layer')
+        parser.add_argument('--cates',           nargs='+', default=['Deepfake', 'Camera'])
+        parser.add_argument('--eval_freq',       type=int, default=200, help='eval frequency')
+        parser.add_argument('--lora_r',          type=int, default=16, help='eval frequency')
+        parser.add_argument('--lora_alpha',      type=int, default=32, help='eval frequency')
+        parser.add_argument('--lora_dropout',    type=float, default=0.1, help='eval frequency')
+        parser.add_argument('--lr', type=float, default=0.0001, help='initial learning rate for adam')
+
+        self.initialized = True
+        return parser
+
+    def gather_options(self):
+        # initialize parser with basic options
+        if not self.initialized:
+            parser = argparse.ArgumentParser(
+                formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+            parser = self.initialize(parser)
+
+        # get the basic options
+        opt, _ = parser.parse_known_args()
+        self.parser = parser
+
+        return opt
+        # return parser.parse_args()
+
+    def print_options(self, opt):
+        message = ''
+        message += '----------------- Options ---------------\n'
+        for k, v in sorted(vars(opt).items()):
+            comment = ''
+            default = self.parser.get_default(k)
+            if v != default:
+                comment = '\t[default: %s]' % str(default)
+            message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
+        message += '----------------- End -------------------'
+        print(message)
+
+        # save to the disk
+        
+        expr_dir = os.path.join(opt.checkpoints_dir, opt.name)
+        util.mkdirs(expr_dir)
+        file_name = os.path.join(expr_dir, 'opt.txt')
+        with open(file_name, 'wt') as opt_file:
+            opt_file.write(message)
+            opt_file.write('\n')
+
+    def parse(self, print_options=True):
+
+        opt = self.gather_options()
+        opt.isTrain = self.isTrain   # train or test
+        opt.imgroot = opt.dataroot
+        opt.name = '__'.join([opt.name, time.strftime("%Y_%m_%d_%H_%M_%S", time.localtime()), 'Seed_'+str(opt.seed), 'cates_'+'-'.join(opt.cates), 'claloss_'+str(opt.claloss), 'lora_r_'+str(opt.lora_r), 'lora_alpha_'+str(opt.lora_alpha), 'lora_dropout_'+str(opt.lora_dropout), 'lr_'+str(opt.lr)])
+
+        if opt.suffix:
+            suffix = ('_' + opt.suffix.format(**vars(opt))) if opt.suffix != '' else ''
+            opt.name = opt.name + suffix
+
+        if print_options:
+            self.print_options(opt)
+
+        # set gpu ids
+        str_ids = opt.gpu_ids.split(',')
+        opt.gpu_ids = []
+        for str_id in str_ids:
+            id = int(str_id)
+            if id >= 0:
+                opt.gpu_ids.append(id)
+        if len(opt.gpu_ids) > 0:
+            torch.cuda.set_device(opt.gpu_ids[0])
+
+        # additional
+        opt.classes = opt.classes.split(',')
+        opt.rz_interp = opt.rz_interp.split(',')
+        opt.blur_sig = [float(s) for s in opt.blur_sig.split(',')]
+        opt.jpg_method = opt.jpg_method.split(',')
+        opt.jpg_qual = [int(s) for s in opt.jpg_qual.split(',')]
+        if len(opt.jpg_qual) == 2:
+            opt.jpg_qual = list(range(opt.jpg_qual[0], opt.jpg_qual[1] + 1))
+        elif len(opt.jpg_qual) > 2:
+            raise ValueError("Shouldn't have more than 2 values for --jpg_qual.")
+
+        self.opt = opt
+        return self.opt

detector_codes/C2P-CLIP-DeepfakeDetection-main/options/test_options.py

@@ -0,0 +1,17 @@
+from .base_options import BaseOptions
+
+
+class TestOptions(BaseOptions):
+    def initialize(self, parser):
+        parser = BaseOptions.initialize(self, parser)
+        # parser.add_argument('--dataroot')
+        parser.add_argument('--model_path')
+        parser.add_argument('--no_resize', action='store_true')
+        parser.add_argument('--no_crop', action='store_true')
+        parser.add_argument('--eval', action='store_true', help='use eval mode during test time.')
+        parser.add_argument('--earlystop_epoch', type=int, default=15)
+        # parser.add_argument('--lr', type=float, default=0.00002, help='initial learning rate for adam')
+        parser.add_argument('--niter', type=int, default=0, help='# of iter at starting learning rate')
+
+        self.isTrain = False
+        return parser

detector_codes/C2P-CLIP-DeepfakeDetection-main/options/train_options.py

@@ -0,0 +1,26 @@
+from .base_options import BaseOptions
+
+ 
+class TrainOptions(BaseOptions):
+    def initialize(self, parser):
+        parser = BaseOptions.initialize(self, parser)
+        parser.add_argument('--earlystop_epoch', type=int, default=15)
+        parser.add_argument('--data_aug', action='store_true', help='if specified, perform additional data augmentation (photometric, blurring, jpegging)')
+        parser.add_argument('--optim', type=str, default='adam', help='optim to use [sgd, adam]')
+        parser.add_argument('--new_optim', action='store_true', help='new optimizer instead of loading the optim state')
+        parser.add_argument('--loss_freq', type=int, default=400, help='frequency of showing loss on tensorboard')
+        parser.add_argument('--save_latest_freq', type=int, default=2000, help='frequency of saving the latest results')
+        parser.add_argument('--save_epoch_freq', type=int, default=20, help='frequency of saving checkpoints at the end of epochs')
+        parser.add_argument('--continue_train', action='store_true', help='continue training: load the latest model')
+        parser.add_argument('--epoch_count', type=int, default=1, help='the starting epoch count, we save the model by <epoch_count>, <epoch_count>+<save_latest_freq>, ...')
+        parser.add_argument('--last_epoch', type=int, default=-1, help='starting epoch count for scheduler intialization')
+        parser.add_argument('--train_split', type=str, default='train', help='train, val, test, etc')
+        parser.add_argument('--val_split', type=str, default='val', help='train, val, test, etc')
+        parser.add_argument('--niter', type=int, default=1000, help='number of epochs')
+        parser.add_argument('--total_steps', type=int, default=1000, help='total_steps to train')
+        parser.add_argument('--beta1', type=float, default=0.9, help='momentum term of adam')
+        # parser.add_argument('--model_path')
+        # parser.add_argument('--no_resize', action='store_true')
+        # parser.add_argument('--no_crop', action='store_true')
+        self.isTrain = True
+        return parser

detector_codes/C2P-CLIP-DeepfakeDetection-main/requirements.txt

@@ -0,0 +1,93 @@
+--extra-index-url https://download.pytorch.org/whl/cu118
+accelerate==0.32.1
+aiosignal==1.3.1
+annotated-types==0.7.0
+anyio==4.8.0
+attrs==23.2.0
+certifi==2024.7.4
+charset-normalizer==3.3.2
+click==8.1.7
+clip @ git+https://github.com/openai/CLIP.git@dcba3cb2e2827b402d2701e7e1c7d9fed8a20ef1
+contourpy==1.2.1
+cycler==0.12.1
+DingtalkChatbot==1.5.7
+distro==1.9.0
+dlib==19.24.6
+einops==0.8.0
+exceptiongroup==1.2.2
+filelock==3.13.1
+fonttools==4.53.1
+frozenlist==1.4.1
+fsspec==2024.2.0
+ftfy==6.2.0
+h11==0.14.0
+httpcore==1.0.7
+httpx==0.28.1
+idna==3.7
+imageio==2.35.1
+Jinja2==3.1.3
+jiter==0.8.2
+joblib==1.4.2
+jsonschema==4.23.0
+jsonschema-specifications==2023.12.1
+kiwisolver==1.4.5
+lazy_loader==0.4
+lora-pytorch==0.2.0
+MarkupSafe==2.1.5
+matplotlib==3.9.1
+mpmath==1.3.0
+msgpack==1.0.8
+munch==4.0.0
+networkx==3.2.1
+numpy==1.26.3
+nvidia-cublas-cu11==11.11.3.6
+nvidia-cuda-cupti-cu11==11.8.87
+nvidia-cuda-nvrtc-cu11==11.8.89
+nvidia-cuda-runtime-cu11==11.8.89
+nvidia-cudnn-cu11==8.7.0.84
+nvidia-cufft-cu11==10.9.0.58
+nvidia-curand-cu11==10.3.0.86
+nvidia-cusolver-cu11==11.4.1.48
+nvidia-cusparse-cu11==11.7.5.86
+nvidia-nccl-cu11==2.20.5
+nvidia-nvtx-cu11==11.8.86
+openai==1.60.2
+opencv-contrib-python==4.10.0.84
+opencv-python==4.10.0.84
+packaging==24.1
+peft==0.11.1
+pillow==10.2.0
+protobuf==5.27.2
+psutil==6.0.0
+pydantic==2.10.6
+pydantic_core==2.27.2
+pyparsing==3.1.2
+python-dateutil==2.9.0.post0
+pytorch_wavelets @ file:///opt/data/private/tcc/GANS_BS1_project6_LLM/A_FatFormer/FatFormer/pytorch_wavelets
+PyYAML==6.0.1
+ray==2.32.0
+referencing==0.35.1
+regex==2024.5.15
+requests==2.32.3
+rpds-py==0.19.0
+safetensors==0.4.3
+scikit-image==0.24.0
+scikit-learn==1.5.1
+scipy==1.14.0
+setproctitle==1.3.3
+six==1.16.0
+sniffio==1.3.1
+sympy==1.12
+tensorboardX==2.6.2.2
+threadpoolctl==3.5.0
+tifffile==2024.8.28
+tokenizers==0.19.1
+torch==2.3.0
+torchaudio==2.3.0
+torchvision==0.18.0
+tqdm==4.66.4
+transformers==4.42.3
+triton==2.3.0
+typing_extensions==4.12.2
+urllib3==2.2.2
+wcwidth==0.2.13

detector_codes/C2P-CLIP-DeepfakeDetection-main/train.py

@@ -0,0 +1,150 @@
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import random
+import time
+
+import numpy as np
+import torch
+import torch.nn
+from data import create_dataloader
+from networks.trainer import Trainer
+from options.test_options import TestOptions
+from options.train_options import TrainOptions
+from utils.util import Logger
+from validate import validate
+
+
+def seed_torch(seed=1029):
+    random.seed(seed)
+    os.environ['PYTHONHASHSEED'] = str(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)  # if you are using multi-GPU.
+    torch.backends.cudnn.benchmark = False
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.enabled = False
+
+
+"""Currently assumes jpg_prob, blur_prob 0 or 1"""
+
+
+def get_val_opt():
+    val_opt = TrainOptions().parse(print_options=False)
+    val_opt.dataroot = '{}/{}/'.format(val_opt.dataroot, val_opt.val_split)
+    val_opt.isTrain = False
+    val_opt.no_resize = False
+    val_opt.no_crop = False
+    val_opt.serial_batches = True
+    val_opt.jpg_method = ['pil']
+    if len(val_opt.blur_sig) == 2:
+        b_sig = val_opt.blur_sig
+        val_opt.blur_sig = [(b_sig[0] + b_sig[1]) / 2]
+    if len(val_opt.jpg_qual) != 1:
+        j_qual = val_opt.jpg_qual
+        val_opt.jpg_qual = [int((j_qual[0] + j_qual[-1]) / 2)]
+
+    return val_opt
+
+
+if __name__ == '__main__':
+    opt = TrainOptions().parse()
+    seed_torch(opt.seed)
+
+    Test_dataroot = os.path.join(opt.dataroot, 'test')
+    opt.dataroot = '{}/{}/'.format(opt.dataroot, opt.train_split)
+    Logger(os.path.join(opt.checkpoints_dir, opt.name, 'log.log'))
+    Testopt = TestOptions().parse(print_options=False)
+    Test_vals = os.listdir(Test_dataroot)
+    data_loader = create_dataloader(opt)
+    model = Trainer(opt)
+
+    def testmodel(epoch=0):
+        print('*' * 25)
+        accs = []
+        aps = []
+        logs = [f'Testing end of {epoch}']
+        print(time.strftime('%Y_%m_%d_%H_%M_%S', time.localtime()))
+        for v_id, val in enumerate(Test_vals):
+            Testopt.dataroot = f'{Test_dataroot}/{val}'
+            # Testopt.classes = os.listdir(Testopt.dataroot) if multiclass[v_id] else ['']
+            Testopt.loadSize = opt.cropSize
+            Testopt.cropSize = opt.cropSize
+            Testopt.no_resize = False
+            Testopt.no_crop = False
+            Testopt.classes = ''
+            acc, ap, _, _, _, _ = validate(model.model, Testopt)
+            accs.append(acc)
+            aps.append(ap)
+            logs.append(
+                '({} {:10}) acc: {:.1f}; ap: {:.1f}'.format(
+                    v_id, val, acc * 100, ap * 100
+                )
+            )
+            print(logs[-1])
+        logs.append(
+            '({} {:10}) acc: {:.1f}; ap: {:.1f}'.format(
+                v_id + 1,
+                'Mean',
+                np.array(accs).mean() * 100,
+                np.array(aps).mean() * 100,
+            )
+        )
+        print(logs[-1])
+        print('*' * 25)
+        print(time.strftime('%Y_%m_%d_%H_%M_%S', time.localtime()))
+        return round(np.array(accs).mean() * 100, 4)
+
+    model.train()
+    for epoch in range(opt.niter):
+        epoch_start_time = time.time()
+        iter_data_time = time.time()
+        epoch_iter = 0
+
+        for i, data in enumerate(data_loader):
+            model.total_steps += 1
+            if model.total_steps > opt.total_steps:
+                break
+            epoch_iter += opt.batch_size
+            model.set_input(data)
+            model.optimize_parameters()
+            if model.total_steps % opt.loss_freq == 0:
+                print(
+                    time.strftime('%Y_%m_%d_%H_%M_%S', time.localtime()),
+                    'Train loss: {} loss1: {} loss2-cla: {} at step: {} lr {}'.format(
+                        model.loss,
+                        model.loss1,
+                        model.loss2,
+                        model.total_steps,
+                        model.lr,
+                    ),
+                )
+
+            # if model.total_steps % opt.eval_freq == 0:
+            #     print(os.getcwd())
+            #     print(f'==========total_steps {model.total_steps}=================')
+            #     model.eval()
+            #     testacc = testmodel(epoch)
+            #     model.save_networks( f'{str(epoch)}_total_steps_{str(model.total_steps)}_testacc_{str(testacc)}')
+            #     model.train()
+
+        if epoch % opt.delr_freq == 0 and epoch != 0:
+            print(
+                time.strftime('%Y_%m_%d_%H_%M_%S', time.localtime()),
+                'changing lr at the end of epoch %d, iters %d'
+                % (epoch, model.total_steps),
+            )
+            model.adjust_learning_rate()
+
+        model.eval()
+        testacc = testmodel(epoch)
+        model.save_networks(
+            f'{str(epoch)}_total_steps_{str(model.total_steps)}_testacc_{str(testacc)}'
+        )
+        print(
+            'saving the latest model %s (epoch %d, model.total_steps %d)'
+            % (opt.name, epoch, model.total_steps)
+        )
+        model.train()

detector_codes/C2P-CLIP-DeepfakeDetection-main/train_UniversalFakeDetect.sh

@@ -0,0 +1,21 @@
+CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/train.py \
+    --name ForenSynths_train_val_19test_dataset \
+    --dataroot ./ForenSynths_train_val_19test/ \
+    --classes car,cat,chair,horse \
+    --batch_size 128 \
+    --delr_freq 10 \
+    --lr 0.0004 \
+    --niter 1 \
+    --total_steps 800 \
+    --delr 0.9 \
+    --loadSize 256 \
+    --cropSize 224 \
+    --seed 123 \
+    --clip ./clip-vit-large-patch14/ \
+    --claloss 8.0 \
+    --cates Deepfake Camera \
+    --eval_freq 800 \
+    --loss_freq 50 \
+    --lora_r 6 \
+    --lora_alpha 6 \
+    --lora_dropout 0.8

detector_codes/C2P-CLIP-DeepfakeDetection-main/train_aigibench.py

@@ -0,0 +1,231 @@
+import os
+import random
+import sys
+import time
+
+import numpy as np
+import torch
+import torch.nn as nn
+from datasets import load_dataset
+from torch.utils.data import DataLoader, Dataset
+from torchvision import transforms
+from transformers import AutoTokenizer
+
+# Add the current directory to sys.path so it can find networks, options, etc.
+current_dir = os.path.dirname(os.path.abspath(__file__))
+if current_dir not in sys.path:
+    sys.path.insert(0, current_dir)
+
+from networks.trainer import Trainer
+from options.train_options import TrainOptions
+from utils.util import Logger
+
+
+def seed_torch(seed=1029):
+    random.seed(seed)
+    os.environ['PYTHONHASHSEED'] = str(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+    torch.backends.cudnn.benchmark = False
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.enabled = False
+
+
+class AIGIBenchDataset(Dataset):
+    def __init__(self, hf_data, opt, transform=None):
+        self.hf_data = hf_data
+        self.opt = opt
+        self.transform = transform
+
+        # Use CLIP model path or HF name for tokenizer
+        tokenizer_name = (
+            opt.clip if os.path.exists(opt.clip) else 'openai/clip-vit-large-patch14'
+        )
+        self.tokenizer = AutoTokenizer.from_pretrained(
+            tokenizer_name, model_max_length=77, padding_side='right', use_fast=False
+        )
+        self.tokenizer.pad_token_id = 0
+
+    def __len__(self):
+        return len(self.hf_data)
+
+    def __getitem__(self, idx):
+        item = self.hf_data[idx]
+        image = item['image'].convert('RGB')
+        label = item['label']
+        generator_id = item['generator']
+
+        # Generator mapping based on AIGIBench spec
+        gen_names = {0: 'Real', 1: 'ProGAN', 2: 'SD14'}
+        gen_name = gen_names.get(generator_id, 'Unknown')
+
+        cates = self.opt.cates
+        cates_len = len(cates) // 2
+
+        # Generic image description to guide the contrastive learning
+        text = f'An image produced by {gen_name}'
+
+        if label == 1:  # Fake (AI-generated)
+            text = (
+                f'{" ".join(cates[:cates_len])}. {text} {" ".join(cates[:cates_len])}.'
+            )
+        else:  # Real (Authentic)
+            text = (
+                f'{" ".join(cates[cates_len:])}. {text} {" ".join(cates[cates_len:])}.'
+            )
+
+        inputs = self.tokenizer(
+            [text],
+            padding='max_length',
+            max_length=self.tokenizer.model_max_length,
+            truncation=True,
+            return_tensors='pt',
+        )
+        input_ids = inputs['input_ids'][0]
+        attention_mask = inputs['attention_mask'][0]
+
+        if self.transform:
+            image = self.transform(image)
+
+        # Format: path, image, text, input_ids, attention_mask, label
+        return 'hf_dataset', image, text, input_ids, attention_mask, label
+
+
+def get_train_transforms(opt):
+    return transforms.Compose(
+        [
+            transforms.Resize(opt.loadSize),
+            transforms.RandomCrop(opt.cropSize),
+            transforms.RandomHorizontalFlip(),
+            transforms.ToTensor(),
+            transforms.Normalize(
+                mean=[0.48145466, 0.4578275, 0.40821073],
+                std=[0.26862954, 0.26130258, 0.27577711],
+            ),
+        ]
+    )
+
+
+def get_val_transforms(opt):
+    return transforms.Compose(
+        [
+            transforms.Resize(opt.loadSize),
+            transforms.CenterCrop(opt.cropSize),
+            transforms.ToTensor(),
+            transforms.Normalize(
+                mean=[0.48145466, 0.4578275, 0.40821073],
+                std=[0.26862954, 0.26130258, 0.27577711],
+            ),
+        ]
+    )
+
+
+if __name__ == '__main__':
+    # Parse options
+    opt = TrainOptions().parse()
+    seed_torch(opt.seed)
+
+    # Handle CLIP path if folder doesn't exist locally
+    if not os.path.exists(opt.clip):
+        print(
+            f"CLIP path {opt.clip} not found locally. Using 'openai/clip-vit-large-patch14' from HuggingFace."
+        )
+        opt.clip = 'openai/clip-vit-large-patch14'
+
+    # Initialize logger
+    log_path = os.path.join(opt.checkpoints_dir, opt.name)
+    if not os.path.exists(log_path):
+        os.makedirs(log_path)
+    Logger(os.path.join(log_path, 'log.log'))
+
+    print('Loading AIGIBench dataset from HuggingFace...')
+    ds = load_dataset('TheKernel01/AIGIBench')
+    train_data = ds['train']
+
+    val_data = ds['validation']
+
+    train_dataset = AIGIBenchDataset(
+        train_data, opt, transform=get_train_transforms(opt)
+    )
+    val_dataset = AIGIBenchDataset(val_data, opt, transform=get_val_transforms(opt))
+
+    train_loader = DataLoader(
+        train_dataset,
+        batch_size=opt.batch_size,
+        shuffle=True,
+        num_workers=opt.num_threads,
+    )
+    val_loader = DataLoader(
+        val_dataset,
+        batch_size=opt.batch_size,
+        shuffle=False,
+        num_workers=opt.num_threads,
+    )
+
+    # Initialize Trainer
+    model = Trainer(opt)
+
+    print(f'Starting training on {len(train_data)} samples...')
+    for epoch in range(opt.niter):
+        model.train()
+        epoch_start_time = time.time()
+
+        for i, data in enumerate(train_loader):
+            model.total_steps += 1
+
+            model.set_input(data)
+            model.optimize_parameters()
+
+            if model.total_steps % opt.loss_freq == 0:
+                print(
+                    f'{time.strftime("%Y_%m_%d_%H_%M_%S", time.localtime())} Train loss: {model.loss:.4f} loss1: {model.loss1:.4f} loss2-cla: {model.loss2:.4f} at step: {model.total_steps} lr {model.lr}'
+                )
+
+            if model.total_steps >= opt.total_steps:
+                break
+
+        # Validation at the end of epoch
+        print('Running validation...')
+        model.eval()
+        with torch.no_grad():
+            val_loss = 0
+            val_acc = 0
+            count = 0
+            # Limit validation samples for speed during training
+            max_val_samples = 2000
+
+            for i, data in enumerate(val_loader):
+                model.set_input(data)
+                # Trainer's model is DataParallel, access it directly for inference
+                _, classhead = model.model(
+                    model.input, model.input_ids, model.attention_mask
+                )
+
+                loss = nn.functional.binary_cross_entropy_with_logits(
+                    classhead, model.label
+                )
+                val_loss += loss.item() * len(model.label)
+
+                preds = (torch.sigmoid(classhead) > 0.5).float()
+                val_acc += (preds == model.label).sum().item()
+                count += len(model.label)
+
+                if count >= max_val_samples:
+                    break
+
+            avg_val_loss = val_loss / count
+            avg_val_acc = val_acc / count
+            print(
+                f'Epoch {epoch} | Val Loss: {avg_val_loss:.4f} | Val Acc: {avg_val_acc:.4f}'
+            )
+
+        # Save checkpoint
+        model.save_networks(f'epoch_{epoch}_acc_{avg_val_acc:.4f}')
+
+        if model.total_steps >= opt.total_steps:
+            print('Reached total_steps limit. Ending training.')
+            break
+
+    print('Training complete.')

detector_codes/C2P-CLIP-DeepfakeDetection-main/train_aigibench.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Training script for AIGIBench using C2P-CLIP
+# This command reproduces the configuration found in the latest checkpoint.
+
+python ./train_aigibench.py \
+    --name aigibench_full_train \
+    --arch res50 \
+    --batch_size 8 \
+    --lr 0.0001 \
+    --niter 1 \
+    --total_steps 5000 \
+    --loadSize 256 \
+    --cropSize 224 \
+    --seed 123 \
+    --clip openai/clip-vit-large-patch14 \
+    --claloss 0.5 \
+    --cates Deepfake Camera \
+    --eval_freq 200 \
+    --loss_freq 50 \
+    --lora_r 16 \
+    --lora_alpha 32 \
+    --lora_dropout 0.1 \
+    --num_threads 4

detector_codes/C2P-CLIP-DeepfakeDetection-main/train_genimage.sh

@@ -0,0 +1,19 @@
+CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/train.py \
+    --name genimage \
+    --dataroot ./GenImage_Dataset/ \
+    --classes sdv4 \
+    --batch_size 128 \
+    --lr 0.0002 \
+    --niter 1 \
+    --total_steps 100 \
+    --loadSize 256 \
+    --cropSize 224 \
+    --seed 123 \
+    --clip ./clip-vit-large-patch14/ \
+    --claloss 8 \
+    --cates Deepfake Camera \
+    --eval_freq 100 \
+    --loss_freq 10 \
+    --lora_r 6 \
+    --lora_alpha 6 \
+    --lora_dropout 0.5

detector_codes/C2P-CLIP-DeepfakeDetection-main/utils/logger.py

@@ -0,0 +1,214 @@
+import collections
+import logging
+import os
+import sys
+import time
+
+import numpy as np
+
+
+def create_logger(log_dir, phase='train'):
+    time_str = time.strftime('%Y-%m-%d-%H-%M')
+    log_file = '{}_{}.log'.format(time_str, phase)
+    final_log_file = os.path.join(log_dir, log_file)
+    logging.basicConfig(
+        filename=str(final_log_file),
+        format='%(asctime)s %(levelname)s: %(message)s',
+        level=logging.INFO,
+        datefmt='%Y-%m-%d %H:%M:%S',
+    )
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+    console = logging.StreamHandler()
+    logging.getLogger('').addHandler(console)
+    return logger
+
+
+class Progbar(object):
+    """Displays a progress bar.
+    # Arguments
+        target: Total number of steps expected, None if unknown.
+        width: Progress bar width on screen.
+        verbose: Verbosity mode, 0 (silent), 1 (verbose), 2 (semi-verbose)
+        stateful_metrics: Iterable of string names of metrics that
+            should *not* be averaged over time. Metrics in this list
+            will be displayed as-is. All others will be averaged
+            by the progbar before display.
+        interval: Minimum visual progress update interval (in seconds).
+    """
+
+    def __init__(
+        self, target, width=30, verbose=1, interval=0.05, stateful_metrics=None
+    ):
+        self.target = target
+        self.width = width
+        self.verbose = verbose
+        self.interval = interval
+        if stateful_metrics:
+            self.stateful_metrics = set(stateful_metrics)
+        else:
+            self.stateful_metrics = set()
+
+        self._dynamic_display = (
+            hasattr(sys.stdout, 'isatty') and sys.stdout.isatty()
+        ) or 'ipykernel' in sys.modules
+        self._total_width = 0
+        self._seen_so_far = 0
+        self._values = collections.OrderedDict()
+        self._start = time.time()
+        self._last_update = 0
+
+    def update(self, current, values=None):
+        """Updates the progress bar.
+        # Arguments
+            current: Index of current step.
+            values: List of tuples:
+                `(name, value_for_last_step)`.
+                If `name` is in `stateful_metrics`,
+                `value_for_last_step` will be displayed as-is.
+                Else, an average of the metric over time will be displayed.
+        """
+        values = values or []
+        for k, v in values:
+            if k not in self.stateful_metrics:
+                if k not in self._values:
+                    self._values[k] = [
+                        v * (current - self._seen_so_far),
+                        current - self._seen_so_far,
+                    ]
+                else:
+                    self._values[k][0] += v * (current - self._seen_so_far)
+                    self._values[k][1] += current - self._seen_so_far
+            else:
+                self._values[k] = v
+        self._seen_so_far = current
+
+        now = time.time()
+        info = ' - %.0fs' % (now - self._start)
+        if self.verbose == 1:
+            if (
+                now - self._last_update < self.interval
+                and self.target is not None
+                and current < self.target
+            ):
+                return
+
+            prev_total_width = self._total_width
+            if self._dynamic_display:
+                sys.stdout.write('\b' * prev_total_width)
+                sys.stdout.write('\r')
+            else:
+                sys.stdout.write('\n')
+
+            if self.target is not None:
+                numdigits = int(np.floor(np.log10(self.target))) + 1
+                barstr = '%%%dd/%d [' % (numdigits, self.target)
+                bar = barstr % current
+                prog = float(current) / self.target
+                prog_width = int(self.width * prog)
+                if prog_width > 0:
+                    bar += '=' * (prog_width - 1)
+                    if current < self.target:
+                        bar += '>'
+                    else:
+                        bar += '='
+                bar += '.' * (self.width - prog_width)
+                bar += ']'
+            else:
+                bar = '%7d/Unknown' % current
+
+            self._total_width = len(bar)
+            sys.stdout.write(bar)
+
+            if current:
+                time_per_unit = (now - self._start) / current
+            else:
+                time_per_unit = 0
+            if self.target is not None and current < self.target:
+                eta = time_per_unit * (self.target - current)
+                if eta > 3600:
+                    eta_format = '%d:%02d:%02d' % (
+                        eta // 3600,
+                        (eta % 3600) // 60,
+                        eta % 60,
+                    )
+                elif eta > 60:
+                    eta_format = '%d:%02d' % (eta // 60, eta % 60)
+                else:
+                    eta_format = '%ds' % eta
+
+                info = ' - ETA: %s' % eta_format
+            else:
+                if time_per_unit >= 1:
+                    info += ' %.0fs/step' % time_per_unit
+                elif time_per_unit >= 1e-3:
+                    info += ' %.0fms/step' % (time_per_unit * 1e3)
+                else:
+                    info += ' %.0fus/step' % (time_per_unit * 1e6)
+
+            for k in self._values:
+                info += ' - %s:' % k
+                if isinstance(self._values[k], list):
+                    avg = np.mean(self._values[k][0] / max(1, self._values[k][1]))
+                    if abs(avg) > 1e-3:
+                        info += ' %.4f' % avg
+                    else:
+                        info += ' %.4e' % avg
+                else:
+                    info += ' %s' % self._values[k]
+
+            self._total_width += len(info)
+            if prev_total_width > self._total_width:
+                info += ' ' * (prev_total_width - self._total_width)
+
+            if self.target is not None and current >= self.target:
+                info += '\n'
+
+            sys.stdout.write(info)
+            sys.stdout.flush()
+
+        elif self.verbose == 2:
+            if self.target is None or current >= self.target:
+                for k in self._values:
+                    info += ' - %s:' % k
+                    avg = np.mean(self._values[k][0] / max(1, self._values[k][1]))
+                    if avg > 1e-3:
+                        info += ' %.4f' % avg
+                    else:
+                        info += ' %.4e' % avg
+                info += '\n'
+
+                sys.stdout.write(info)
+                sys.stdout.flush()
+
+        self._last_update = now
+
+    def add(self, n, values=None):
+        self.update(self._seen_so_far + n, values)
+
+
+class AverageMeter(object):
+    """Computes and stores the average and current value"""
+
+    def __init__(self):
+        self.reset()
+
+    def reset(self):
+        self.val = 0
+        self.avg = 0
+        self.sum = 0
+        self.count = 0
+
+    def update(self, val, n=1):
+        self.val = val
+        self.sum += val * n
+        self.count += n
+        self.avg = self.sum / (0.0001 + self.count)
+
+    def __str__(self):
+        """String representation for logging"""
+        # for values that should be recorded exactly e.g. iteration number
+        if self.count == 0:
+            return str(self.val)
+        # for stats
+        return '%.4f (%.4f)' % (self.val, self.avg)

detector_codes/C2P-CLIP-DeepfakeDetection-main/utils/util.py

@@ -0,0 +1,52 @@
+import os
+import sys
+
+import torch
+
+
+def mkdirs(paths):
+    if isinstance(paths, list) and not isinstance(paths, str):
+        for path in paths:
+            mkdir(path)
+    else:
+        mkdir(paths)
+
+
+def mkdir(path):
+    if not os.path.exists(path):
+        os.makedirs(path)
+
+
+def unnormalize(tens, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]):
+    # assume tensor of shape NxCxHxW
+    return (
+        tens * torch.Tensor(std)[None, :, None, None]
+        + torch.Tensor(mean)[None, :, None, None]
+    )
+
+
+class Logger(object):
+    """Log stdout messages."""
+
+    def __init__(self, outfile):
+        self.terminal = sys.stdout
+        self.log = open(outfile, 'a')
+        sys.stdout = self
+
+    def write(self, message):
+        self.terminal.write(message)
+        self.log.write(message)
+
+    def flush(self):
+        self.terminal.flush()
+
+    def isatty(self):
+        return self.terminal.isatty()
+
+
+def printSet(set_str):
+    set_str = str(set_str)
+    num = len(set_str)
+    print('=' * num * 3)
+    print(' ' * num + set_str)
+    print('=' * num * 3)

detector_codes/C2P-CLIP-DeepfakeDetection-main/validate.py

@@ -0,0 +1,23 @@
+import numpy as np
+import torch
+from data import create_dataloader
+from sklearn.metrics import accuracy_score, average_precision_score
+
+
+def validate(model, opt):
+    data_loader = create_dataloader(opt)
+
+    with torch.no_grad():
+        y_true, y_pred = [], []
+        for path, img, text, input_ids, attention_mask, label in data_loader:
+            y_pred.extend(
+                model(img.cuda(), None, None, cla=True).sigmoid().flatten().tolist()
+            )
+            y_true.extend(label.flatten().tolist())
+
+    y_true, y_pred = np.array(y_true), np.array(y_pred)
+    r_acc = accuracy_score(y_true[y_true == 0], y_pred[y_true == 0] > 0.5)
+    f_acc = accuracy_score(y_true[y_true == 1], y_pred[y_true == 1] > 0.5)
+    acc = accuracy_score(y_true, y_pred > 0.5)
+    ap = average_precision_score(y_true, y_pred)
+    return acc, ap, r_acc, f_acc, y_true, y_pred

detector_codes/C2P-DINOv2-main/dataset.py

@@ -0,0 +1,45 @@
+import torch
+from torch.utils.data import Dataset
+from torchvision import transforms
+
+
+class AIGIBenchDataset(Dataset):
+    def __init__(self, hf_data, transform=None):
+        self.hf_data = hf_data
+        self.transform = transform
+
+    def __len__(self):
+        return len(self.hf_data)
+
+    def __getitem__(self, idx):
+        item = self.hf_data[idx]
+        image = item['image'].convert('RGB')
+        label = item['label']  # 0 for Real, 1 for Fake
+
+        if self.transform:
+            image = self.transform(image)
+
+        return image, torch.tensor(label, dtype=torch.float32)
+
+
+def get_train_transforms(size=224):
+    return transforms.Compose(
+        [
+            transforms.Resize(size + 32),
+            transforms.RandomCrop(size),
+            transforms.RandomHorizontalFlip(),
+            transforms.ToTensor(),
+            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+        ]
+    )
+
+
+def get_val_transforms(size=224):
+    return transforms.Compose(
+        [
+            transforms.Resize(size + 32),
+            transforms.CenterCrop(size),
+            transforms.ToTensor(),
+            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+        ]
+    )