---
license: mit
tags:
- face-generation
- computer-vision
- vision-transformer
- deepfake
- image-generation
- pytorch
- research-only
- vit
- cross-attention
language:
- en
library_name: pytorch
pipeline_tag: image-to-image
---

# FaceForge Generator: Vision Transformer-based Face Manipulation

[![Paper](https://img.shields.io/badge/Paper-Zenodo-blue)](https://doi.org/10.5281/zenodo.18530439)
[![GitHub](https://img.shields.io/badge/GitHub-Repository-black)](https://github.com/Huzaifanasir95/FaceForge)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)

🎨 **252M Parameters | ViT-Based | Baseline Training Complete**

⚠️ **RESEARCH USE ONLY** - This model is for academic research and developing detection systems.

## Model Description

FaceForge Generator is a sophisticated Vision Transformer-based facial manipulation system that learns to synthesize realistic face swaps. The model combines dual ViT encoders, cross-attention mechanisms, transformer decoders, and CNN upsamplers to generate high-quality facial manipulations.

**Key Features:**
- 🏗️ 252 million trainable parameters
- 🔄 Dual encoder architecture for source and target faces
- 🎯 Cross-attention fusion mechanism
- 🖼️ Generates 224×224 RGB face images
- ⚡ ~300ms inference time per image
- 📉 Achieved 0.204 validation loss after 3 epochs

## Model Architecture

```
FaceForge Generator (252.5M parameters)
│
├── ViT Encoders (172M params)
│   ├── Source Encoder: ViT-B/16 (86M)
│   │   └── 12 layers, 768-dim, 12 heads
│   └── Target Encoder: ViT-B/16 (86M)
│       └── 12 layers, 768-dim, 12 heads
│
├── Cross-Attention Module (14M params)
│   ├── 2 layers, 8 heads
│   ├── FFN: 768 → 3072 → 768
│   └── Dropout: 0.1
│
├── Transformer Decoder (58M params)
│   ├── 256 learnable queries (16×16)
│   ├── 6 decoder layers, 8 heads
│   └── 2D positional embeddings
│
└── CNN Upsampler (9M params)
    ├── TransposeConv: 768→512→256→128→64
    ├── 4 upsampling stages (16×16 → 224×224)
    └── Conv: 64→32→3 + Tanh
```

## Training Progress

### Baseline Training (3 Epochs)

| Epoch | Train Loss | Val Loss | Time (min) |
|-------|-----------|----------|------------|
| 1 | 0.2873 | 0.2804 | 227.5 |
| 2 | 0.2432 | 0.2304 | 231.2 |
| 3 | 0.2143 | 0.2043 | 228.8 |

**Total Training Time:** 11.5 hours (687.5 minutes)

### Loss Reduction
- Training loss: 0.287 → 0.214 (25.3% reduction)
- Validation loss: 0.280 → 0.204 (27.1% reduction)
- Minimal overfitting (train-val gap: 0.010)

## Usage

### Installation

```bash
pip install torch torchvision timm pillow numpy
```

### Loading the Model

```python
import torch
import torch.nn as nn
import timm
from torchvision import transforms

class FaceForgeGenerator(nn.Module):
    def __init__(self):
        super().__init__()
        # Source and Target ViT Encoders
        self.source_encoder = timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=0)
        self.target_encoder = timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=0)
        
        # Cross-attention (implement your architecture)
        # Transformer decoder
        # CNN upsampler
        # ... (see full architecture in paper)
    
    def forward(self, source_face, target_face):
        # Encode both faces
        source_features = self.source_encoder.forward_features(source_face)
        target_features = self.target_encoder.forward_features(target_face)
        
        # Cross-attention fusion
        fused_features = self.cross_attention(source_features, target_features)
        
        # Decode to spatial map
        spatial_features = self.transformer_decoder(fused_features)
        
        # Upsample to 224×224
        generated_face = self.cnn_upsampler(spatial_features)
        
        return generated_face

# Load checkpoint
model = FaceForgeGenerator()
checkpoint = torch.load('generator_best.pth', map_location='cpu')
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Preprocessing
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
])

# Generate face swap
def generate_face_swap(source_path, target_path):
    source = transform(Image.open(source_path).convert('RGB')).unsqueeze(0)
    target = transform(Image.open(target_path).convert('RGB')).unsqueeze(0)
    
    with torch.no_grad():
        generated = model(source, target)
    
    # Denormalize and convert to PIL
    generated = (generated[0] * 0.5 + 0.5).clamp(0, 1)
    generated = transforms.ToPILImage()(generated)
    
    return generated

# Example
result = generate_face_swap("source.jpg", "target.jpg")
result.save("generated.jpg")
```

## Training Details

### Dataset
- **Source:** FaceForensics++ (c40 compression)
- **Training:** 7,000 face images (triplets: source, target, ground truth)
- **Validation:** 1,500 face images
- **Resolution:** 224×224 RGB

### Hyperparameters
```yaml
optimizer: AdamW
learning_rate: 1e-4
betas: [0.9, 0.999]
weight_decay: 1e-4
batch_size: 16
epochs: 3 (baseline)
loss_function: L1 (Mean Absolute Error)
lr_schedule: Cosine Annealing (1e-4 → 1e-6)
```

### Training Configuration
- **Hardware:** CPU
- **Throughput:** ~32 samples/minute
- **Batch Processing:** 219 train batches, 47 val batches per epoch
- **Best Model:** Saved at epoch 3

## Current Status

⚠️ **Baseline Training:** This model has completed 3 epochs of baseline training. For production-quality face generation, extended training (15-20 epochs) is recommended.

**Current Capabilities:**
- ✅ Learns pose transfer
- ✅ Captures facial structures
- ✅ Shows convergence trend
- ⏳ Some blur in generated images (expected at baseline)
- ⏳ Benefits from extended training

## Use Cases

### Research Applications
1. **Detector Training:** Generate challenging samples for deepfake detection
2. **Adversarial Training:** Min-max game with detector
3. **Understanding Manipulation:** Study how synthetic faces are created
4. **Benchmark Creation:** Generate test sets for evaluation

### Educational Uses
- Demonstrate face generation techniques
- Teach computer vision concepts
- Illustrate transformer architectures
- Show attention mechanism visualization

## Limitations

1. **Training Duration:** Only 3 epochs completed; extended training needed for photo-realism
2. **Blur:** Generated faces show some blur at baseline stage
3. **Dataset Scale:** Trained on 10K images; larger datasets would improve quality
4. **Single Frame:** Doesn't consider temporal consistency for video
5. **Compute:** Large model (252M params) requires significant memory

## Ethical Guidelines

⚠️ **Responsible Use Required**

This model is intended for:
✅ Academic research
✅ Deepfake detection development
✅ Educational demonstrations
✅ Ethical AI studies

**Prohibited uses:**
❌ Creating misinformation
❌ Identity theft or impersonation
❌ Non-consensual face manipulation
❌ Malicious content creation

**Recommendations:**
- Watermark generated content
- Maintain audit logs
- Require user consent
- Implement content filters

## Future Improvements

Planned enhancements:
- [ ] Extended training (15-20 epochs)
- [ ] Perceptual loss functions (VGG, LPIPS)
- [ ] GAN-based adversarial training
- [ ] Multi-scale architecture
- [ ] Attention visualization
- [ ] Video temporal consistency

## Citation

```bibtex
@techreport{nasir2026faceforge,
  title={FaceForge: A Deep Learning Framework for Facial Manipulation Generation and Detection},
  author={Nasir, Huzaifa},
  institution={National University of Computer and Emerging Sciences},
  year={2026},
  doi={10.5281/zenodo.18530439}
}
```

## Links

- 📄 **Paper:** https://doi.org/10.5281/zenodo.18530439
- 💻 **Code:** https://github.com/Huzaifanasir95/FaceForge
- 🔍 **Detector Model:** https://huggingface.co/Huzaifanasir95/faceforge-detector
- 📓 **Notebooks:** See repository for training/inference notebooks

## Architecture Details

### Vision Transformer Encoder
- **Patch Size:** 16×16
- **Patches:** 196 + 1 CLS token
- **Embedding Dim:** 768
- **Layers:** 12
- **Attention Heads:** 12
- **MLP Ratio:** 4.0

### Cross-Attention Mechanism
- **Query:** Source features
- **Key/Value:** Target features
- **Attention:** Multi-head (8 heads)
- **FFN Expansion:** 4× (768 → 3072 → 768)

### CNN Upsampler
- **Input:** 768×16×16
- **Output:** 3×224×224
- **Stages:** 4 transpose convolutions
- **Kernel:** 4×4, Stride: 2, Padding: 1
- **Activation:** ReLU → Tanh (output)

## License

This model is released under CC BY 4.0 license. Use responsibly and ethically.

## Author

**Huzaifa Nasir**  
National University of Computer and Emerging Sciences (NUCES)  
Islamabad, Pakistan  
📧 nasirhuzaifa95@gmail.com

## Acknowledgments

- Vision Transformer (Dosovitskiy et al.)
- FaceForensics++ dataset
- PyTorch and timm libraries
- Open-source AI community