README.md

---
title: Content Moderation Demo
emoji: 👁
colorFrom: green
colorTo: indigo
sdk: gradio
sdk_version: 6.6.0
app_file: app.py
pinned: false
---

# Content Moderation Demo (Image & Video)

A lightweight, end-to-end **content moderation prototype** for **images and short videos**, built as part of my **B.Sc. Computer Science thesis** at **Flensburg University of Applied Sciences**, showcasing models trained and evaluated for content moderation.
It lets you pick a model, select categories, tune a threshold, and get a clear **SAFE / RISKY** verdict with per-category scores.
For videos, it can detect risky segments via frame sampling and optionally **remove flagged intervals** (FFmpeg required).


**Live demo & resources**
- 🤗 Space: https://huggingface.co/spaces/onullusoy/content_moderation_demo  
- 🤗 Dataset: https://huggingface.co/datasets/onullusoy/harmful-contents  
- GitHub mirror (same codebase): https://github.com/onullusoy/content_moderation_demo
- Thesis repo (train/eval/results): `content_moderation_thesis` *(coming soon)*

---

## What you can do

### Image moderation
- Upload an image, choose a model, pick categories, set a threshold
- Get:
  - **Verdict:** SAFE / RISKY
  - **Scores table:** probability per category
  - Optional: **Top “selected tags”** (for EVA-based models)

### Video moderation (≤ 60s)
- Upload a short video and set a **sampling FPS** (e.g., 1 FPS)
- Get:
  - Detected risky **segments** with start/end timestamps and peak category probability
  - Optional: **Redacted output** (flagged intervals removed) if FFmpeg is available

---

## Categories

Default multi-label categories:
- `alcohol`, `drugs`, `weapons`, `gambling`, `nudity`, `sexy`, `smoking`, `violence`

Additionally, NudeNet linear-probe models expose a single category:
- `sexual`

> Note: Categories & thresholds are for experimentation and evaluation. They are not a safety guarantee.

---

## Included models (pluggable via `model_registry.py`)

Model dropdown options:
- `clip-multilabel`  
  OpenCLIP (ViT-L/14) image encoder + custom multi-label linear head.
- `wdeva02-multilabel`  
  WD-EVA02 tagger backbone + custom head (also exposes top “selected tags”).
- `animetimm-multilabel`  
  EVA-02 (animetimm) backbone + custom head (also exposes top “selected tags”).
- `clip-nudenet-lp`  
  CLIP + linear probe (sexual-only).
- `siglip-nudenet-lp`  
  SigLIP + linear probe (sexual-only).

---

## Run locally

### 0) Clone with Git LFS (required)
This repository stores **weights and example files via Git LFS**.

```bash
git lfs install
git clone https://huggingface.co/spaces/onullusoy/content_moderation_demo
cd content_moderation_demo
git lfs pull
```

### 1) Install dependencies
```bash
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt
```

### 2) (Optional) Enable video redaction
Redaction requires `ffmpeg` + `ffprobe` available on PATH.

### 3) Start the app
```bash
python app.py
```

---

## How it works (high-level)

- **Image**: preprocess → model inference → per-category probabilities → thresholded verdict  
- **Video**:
  - Extract frames at sampling FPS (FFmpeg)
  - Run image model per-frame
  - Merge hit frames into contiguous time segments
  - Optional: output a “redacted” video by removing flagged intervals

---

## Repository structure

- `app.py` — Gradio UI (Image + Video tabs)
- `model_registry.py` — Model registry & category definitions
- `src/models/` — Minimal model wrappers (CLIP multi-label head, EVA head-preserving adapter, linear probes)
- `weights/` — Trained heads / probes
- `examples/` — Small sample inputs
- `video_utils.py` — FFmpeg helpers + segment merging

---


## Disclaimer

This is a **research/demo prototype** intended for experimentation and evaluation.  
False positives/negatives are expected depending on model choice, threshold, and content domain.


### License / Usage
The provided **weights** are for **research/non-commercial use** only.  
Commercial use requires independent rights checks (models, data, third parties).