readme

README.md

@@ -1,77 +1,106 @@
-# DeepFakeDefenders
+<h2 align="center"> <a href="">DeepFake Defenders</a></h2>
+<h5 align="center"> If you like our project, please give us a star ⭐ on GitHub for latest update.  </h5>
+
+<h5 align="center">
+    
+<!-- PROJECT SHIELDS -->
+[![License](https://img.shields.io/badge/License-Apache%202.0-yellow)](https://github.com/VisionRush/DeepFakeDefenders/blob/main/LICENSE) 
+![GitHub contributors](https://img.shields.io/github/contributors/VisionRush/DeepFakeDefenders)
+[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FVisionRush%2FDeepFakeDefenders&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=Visitors&edge_flat=false)](https://hits.seeyoufarm.com)
+![GitHub Repo stars](https://img.shields.io/github/stars/VisionRush/DeepFakeDefenders)
+[![GitHub issues](https://img.shields.io/github/issues/VisionRush/DeepFakeDefenders?color=critical&label=Issues)](https://github.com/PKU-YuanGroup/MoE-LLaVA/issues?q=is%3Aopen+is%3Aissue)
+[![GitHub closed issues](https://img.shields.io/github/issues-closed/VisionRush/DeepFakeDefenders?color=success&label=Issues)](https://github.com/PKU-YuanGroup/MoE-LLaVA/issues?q=is%3Aissue+is%3Aclosed)  <br>
+
+</h5>
 
 <p align='center'>  
-  <img src='https://github.com/HighwayWu/DeepFakeDefenders/blob/main/imgs/competition.png' width='850'/>
+  <img src='./images/competition_title.png' width='850'/>
 </p>
 
-1st place solution for [The Global Multimedia Deepfake Detection (Image Track)](https://www.kaggle.com/competitions/multi-ffdi/overview) by "JTGroup" team.
+💡 We also provide [[中文文档 / CHINESE DOC](README_zh.md)]. We very welcome and appreciate your contributions to this project.
 
-## Table of Contents
+## 📣 News
 
-- [Motivation](#motivation)
-- [Framework](#framework)
-- [Dependency](#dependency)
-- [Inference](#inference)
-- [Training](#training)
-- [Acknowledgement](#acknowledgement)
+* **[2024.09.05]**  🔥 We officially released the initial version of Deepfake defenders, and we won the third prize in the deepfake challenge at [[the conference on the bund](https://www.atecup.cn/deepfake)].
 
+## 🚀 Quickly Start
 
-## Motivation
-The visualization of features in the official split of training and validation sets reveals two key insights: 1) the similarity in distribution between the training and validation sets makes it challenging to evaluate the model's performance on unseen test sets; 2) the feature types of fake data are richer compared to real data. This insight inspired us to initially use unsupervised clustering to partition the challenging validation set, followed by augmenting the types of real and fake data based on adversarial learning principles. These insights led to the development of our proposed generalizable deepfake detection method.
+### 1. Pretrained Models Preparation 
 
-<p align='center'>  
-  <img src='https://github.com/HighwayWu/DeepFakeDefenders/blob/main/imgs/motivation.jpg' width='850'/>
-</p>
+Before getting started, please place the ImageNet-1K pretrained weight files in the `./pre_model` directory. The download links for the weights are provided below:
+```
+RepLKNet: https://drive.google.com/file/d/1vo-P3XB6mRLUeDzmgv90dOu73uCeLfZN/view?usp=sharing
+ConvNeXt: https://dl.fbaipublicfiles.com/convnext/convnext_base_1k_384.pth
+```
 
+### 2. Training from Scratch
 
-## Framework
-The framework of our proposed method consists of two primary stages: Data Preparing and Training. In the Data Preparing stage, we focus on augmenting the existing dataset by generating new data through image editing and Stable Diffusion (SD) techniques. We then perform clustering to reassemble the dataset, aiming to enhance both the detection performance and the robustness of our method. In the Training stage, we introduce a series of expert models and optimize them using three types of losses: $L_{\mathsf{KL}}$, $L_{\mathsf{NCE}}$, and $L_{\mathsf{CE}}$. This multi-loss approach ensures that the model can effectively differentiate between authentic and manipulated images. Please refer to the tech report (will be on arXiv later) for more details.
+#### 2.1 Modifying the dataset path
 
-<p align='center'>  
-  <img src='https://github.com/HighwayWu/DeepFakeDefenders/blob/main/imgs/framework.jpg' width='850'/>
-</p>
+Place the training-set **(\*.txt)** file, validation-set **(\*.txt)** file, and label **(\*.txt)** file required for training in the dataset folder and name them with the same file name (there are various txt examples under dataset)
+
+#### 2.2 Modifying the Hyper-parameters
 
-## Dependency
-- torch 1.9.0
-- timm 1.0.8
-- scikit-learn 1.2.1
-- torch-ema 0.3
-- torch_kmeans 0.2.0
-- albumentations 1.3.1
+For the two models (RepLKNet and ConvNeXt) used, the following parameters need to be changed in `main_train.py`:
 
-## Inference
+```python
+# For RepLKNet
+cfg.network.name = 'replknet'; cfg.train.batch_size = 16
+# For ConvNeXt
+cfg.network.name = 'convnext'; cfg.train.batch_size = 24
+```
 
-```bash
->> python infer.py
+#### 2.3 Using the training script
+##### Multi-GPUs:（8 GPUs were used）
+```shell
+bash main.sh
 ```
-Enter the image path and expect the following result:
-```bash
->> demo/img_1_should_be_0.0016829653177410.jpg
->> Prediction of [demo/img_1_should_be_0.0016829653177410.jpg] being Deepfake: 0.001683078
+##### Single-GPU：
+```shell
+CUDA_VISIBLE_DEVICES=0 python main_train_single_gpu.py
 ```
 
-**Note: The pretrained weights can be downloaded from [Baidu Pan](https://pan.baidu.com/s/1hh6Rub60T7UXok5rqACffQ?pwd=gxu5).**
+#### 2.4 Model Assembling
 
-## Training
-The training code (e.g., unsupervised clustering and joint optimization loss) and detailed technical report will be available shortly.
+Replace the ConvNeXt trained model path and the RepLKNet trained model path in `merge.py`, and execute `python merge.py` to obtain the final inference test model.
 
-## Acknowledgement
-- THIS WORK WAS PERFORMED IN PART AT SICC WHICH IS SUPPORTED BY SKL-IOTSC, UNIVERSITY OF MACAU.
+### 2.5 Inference
 
-<p align='left'>  
-  <img src='https://github.com/HighwayWu/DeepFakeDefenders/blob/main/imgs/organization.png' width='450'/>
-</p>
+The following example uses the **POST** request interface to request the image path as the request parameter, and the response output is the deepfake score predicted by the model.
+
+```python
+#!/usr/bin/env python
+# -*- coding:utf-8 -*-
+import requests
+import json
+import requests
+import json
 
-- INCLUSION·CONFERENCE ON THE BUND, THE ORGANIZER OF THE COMPETITION.
+header = {
+    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'
+}
+
+url = 'http://ip:10005/inter_api'
+image_path = './dataset/val_dataset/51aa9b8d0da890cd1d0c5029e3d89e3c.jpg'
+data_map = {'img_path':image_path}
+response = requests.post(url, data=json.dumps(data_map), headers=header)
+content = response.content
+print(json.loads(content))
+```
 
-## License
-This work is licensed under a
-[Creative Commons Attribution-NonCommercial 4.0 International License][cc-by-nc].
+### 3. Deploy in Docker
+#### Building
 
-[![CC BY-NC 4.0][cc-by-nc-image]][cc-by-nc]
+```shell
+sudo docker build  -t vision-rush-image:1.0.1 --network host .
+```
+
+#### Running
+
+```shell
+sudo docker run -d --name  vision_rush_image  --gpus=all  --net host  vision-rush-image:1.0.1
+```
 
-[cc-by-nc]: https://creativecommons.org/licenses/by-nc/4.0/
-[cc-by-nc-image]: https://licensebuttons.net/l/by-nc/4.0/88x31.png
-[cc-by-nc-shield]: https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg
+## Star History
 
-For inquiries or to obtain permission for commercial use, please contact  Yiming Chen (yc17486@umac.mo).
+[![Star History Chart](https://api.star-history.com/svg?repos=VisionRush/DeepFakeDefenders&type=Date)](https://star-history.com/#DeepFakeDefenders/DeepFakeDefenders&Date)

README_zh.md

@@ -0,0 +1,91 @@
+<h2 align="center"> <a href="">DeepFake Defenders</a></h2>
+<h5 align="center"> 如果您喜欢我们的项目，请在 GitHub 上给我们一个Star ⭐ 以获取最新更新。  </h5>
+
+<h5 align="center">
+    
+<!-- PROJECT SHIELDS -->
+[![License](https://img.shields.io/badge/License-Apache%202.0-yellow)](https://github.com/VisionRush/DeepFakeDefenders/blob/main/LICENSE) 
+![GitHub contributors](https://img.shields.io/github/contributors/VisionRush/DeepFakeDefenders)
+[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FVisionRush%2FDeepFakeDefenders&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=Visitors&edge_flat=false)](https://hits.seeyoufarm.com)
+![GitHub Repo stars](https://img.shields.io/github/stars/VisionRush/DeepFakeDefenders)
+[![GitHub issues](https://img.shields.io/github/issues/VisionRush/DeepFakeDefenders?color=critical&label=Issues)](https://github.com/PKU-YuanGroup/MoE-LLaVA/issues?q=is%3Aopen+is%3Aissue)
+[![GitHub closed issues](https://img.shields.io/github/issues-closed/VisionRush/DeepFakeDefenders?color=success&label=Issues)](https://github.com/PKU-YuanGroup/MoE-LLaVA/issues?q=is%3Aissue+is%3Aclosed)  <br>
+
+</h5>
+
+<p align='center'>  
+  <img src='./images/competition_title.png' width='850'/>
+</p>
+
+💡 我们在这里提供了[[英文文档 / ENGLISH DOC](README.md)]，我们十分欢迎和感谢您能够对我们的项目提出建议和贡献。
+
+## 📣 News
+
+* **[2024.09.05]**  🔥 我们正式发布了Deepfake Defenders的初始版本，并在Deepfake挑战赛中获得了三等奖 
+[[外滩大会](https://www.atecup.cn/deepfake)].
+
+## 🚀 快速开始
+### 一、预训练模型准备
+在开始使用之前，请将模型的ImageNet-1K预训练权重文件放置在`./pre_model`目录下，权重下载链接如下:
+```
+RepLKNet: https://drive.google.com/file/d/1vo-P3XB6mRLUeDzmgv90dOu73uCeLfZN/view?usp=sharing
+ConvNeXt: https://dl.fbaipublicfiles.com/convnext/convnext_base_1k_384.pth
+```
+
+### 二、训练
+
+#### 1. 更改数据集路径
+将训练所需的训练集txt文件、验证集txt文件以及标签txt文件分别放置在dataset文件夹下，并命名为相同的文件名（dataset下有各个txt示例）
+#### 2. 更改超参数
+针对所采用的两个模型，在main_train.py中分别需要更改如下参数：
+```python
+RepLKNet---cfg.network.name = 'replknet'; cfg.train.batch_size = 16
+ConvNeXt---cfg.network.name = 'convnext'; cfg.train.batch_size = 24
+```
+
+#### 3. 启动训练
+##### 单机多卡训练：（8卡）
+```shell
+bash main.sh
+```
+##### 单机单卡训练：
+```shell
+CUDA_VISIBLE_DEVICES=0 python main_train_single_gpu.py
+```
+
+#### 4. 模型融合
+在merge.py中更改ConvNeXt模型路径以及RepLKNet模型路径，执行python merge.py后获取最终推理测试模型。
+
+#### 5. 推理
+
+示例如下，通过post请求接口请求，请求参数为图像路径，响应输出为模型预测的deepfake分数
+
+```python
+#!/usr/bin/env python
+# -*- coding:utf-8 -*-
+import requests
+import json
+import requests
+import json
+
+header = {
+    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'
+}
+
+url = 'http://ip:10005/inter_api'
+image_path = './dataset/val_dataset/51aa9b8d0da890cd1d0c5029e3d89e3c.jpg'
+data_map = {'img_path':image_path}
+response = requests.post(url, data=json.dumps(data_map), headers=header)
+content = response.content
+print(json.loads(content))
+```
+
+### 三、docker
+#### 1. docker构建
+    sudo docker build  -t vision-rush-image:1.0.1 --network host .
+#### 2. 容器启动
+    sudo docker run -d --name  vision_rush_image  --gpus=all  --net host  vision-rush-image:1.0.1
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=VisionRush/DeepFakeDefenders&type=Date)](https://star-history.com/#DeepFakeDefenders/DeepFakeDefenders&Date)