Improve model card: add library, paper link, and correct pipeline tag
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,26 +1,28 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
language:
|
| 4 |
-
|
|
|
|
|
|
|
|
|
|
| 5 |
tags:
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
base_model: saberzl/SIDA-7B
|
| 13 |
---
|
| 14 |
|
| 15 |
# PIXAR-7B
|
| 16 |
|
| 17 |
-
**PIXAR-7B** is a Vision-Language Model (VLM) for image tampering analysis, introduced in the paper
|
| 18 |
|
| 19 |
Given a query image, PIXAR-7B jointly performs:
|
| 20 |
-
- **Binary** — real or tampered
|
| 21 |
-
- **Object
|
| 22 |
-
- **Pixel-level
|
| 23 |
-
- **Natural
|
| 24 |
|
| 25 |
## Model Description
|
| 26 |
|
|
@@ -28,7 +30,8 @@ Given a query image, PIXAR-7B jointly performs:
|
|
| 28 |
- **Model type:** Multimodal Vision-Language Model for Image Tampering Detection
|
| 29 |
- **License:** MIT
|
| 30 |
- **Base model:** [SIDA-7B](https://huggingface.co/saberzl/SIDA-7B) (LLaVA + LLaMA-2)
|
| 31 |
-
- **Paper:** From Masks to Pixels and Meaning:
|
|
|
|
| 32 |
|
| 33 |
### Architecture
|
| 34 |
|
|
@@ -42,17 +45,17 @@ PIXAR-7B is built on a LLaVA + LLaMA-2 backbone with LoRA fine-tuning (rank 8),
|
|
| 42 |
|
| 43 |
## Key Contributions
|
| 44 |
|
| 45 |
-
Existing tampering benchmarks rely on coarse object masks as ground truth, which conflates unedited pixels inside the mask with actual tamper evidence and misses subtle edits outside the mask. PIXAR replaces binary masks with **per-pixel difference maps** $D = |I_\text{orig} - I_\text{gen}|$, thresholded at a tunable $\tau$ to produce dynamic ground truth $M_\tau$ that captures edits at multiple scales.
|
| 46 |
|
| 47 |
The **PIXAR benchmark** provides:
|
| 48 |
-
- **420K+ training pairs** with pixel-level $M_\tau$ maps, semantic class labels, and natural language descriptions
|
| 49 |
-
- **40K balanced test pairs** spanning 8 manipulation types (replace, remove, splice, inpaint, attribute change, colorization, etc.)
|
| 50 |
|
| 51 |
PIXAR-7B achieves **2.6× IoU improvement** over prior state of the art on the PIXAR benchmark.
|
| 52 |
|
| 53 |
## How to Get Started
|
| 54 |
|
| 55 |
-
For interactive inference, see the
|
| 56 |
|
| 57 |
```bash
|
| 58 |
python chat.py --version jiachengcui888/PIXAR-7B --precision bf16 --seg_prompt_mode seg_only
|
|
@@ -75,11 +78,11 @@ Fine-tuned with DeepSpeed on a LLaVA + LLaMA-2 backbone using LoRA (rank 8). Key
|
|
| 75 |
| Batch size | 2 |
|
| 76 |
| Precision | bf16 |
|
| 77 |
| Threshold τ | 0.05 |
|
| 78 |
-
| text | 3.0 |
|
| 79 |
-
| cls | 1.0 |
|
| 80 |
-
| bce | 1.0 |
|
| 81 |
-
| dice | 1.0 |
|
| 82 |
-
| sem | 0.5 |
|
| 83 |
|
| 84 |
|
| 85 |
## Evaluation
|
|
@@ -89,5 +92,10 @@ PIXAR-7B achieves **2.6× IoU improvement** over prior SOTA on the PIXAR test be
|
|
| 89 |
## Citation
|
| 90 |
|
| 91 |
```bibtex
|
| 92 |
-
|
| 93 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model: saberzl/SIDA-7B
|
| 3 |
language:
|
| 4 |
+
- en
|
| 5 |
+
license: mit
|
| 6 |
+
library_name: transformers
|
| 7 |
+
pipeline_tag: image-text-to-text
|
| 8 |
tags:
|
| 9 |
+
- image-tampering-detection
|
| 10 |
+
- fake-image-detection
|
| 11 |
+
- pixel-level-localization
|
| 12 |
+
- multimodal
|
| 13 |
+
- vision-language-model
|
| 14 |
+
- segmentation
|
|
|
|
| 15 |
---
|
| 16 |
|
| 17 |
# PIXAR-7B
|
| 18 |
|
| 19 |
+
**PIXAR-7B** is a Vision-Language Model (VLM) for image tampering analysis, introduced in the paper [From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering](https://huggingface.co/papers/2603.20193).
|
| 20 |
|
| 21 |
Given a query image, PIXAR-7B jointly performs:
|
| 22 |
+
- **Binary Classification** — real or tampered.
|
| 23 |
+
- **Object Classification** — identifies which of 81 COCO categories was modified.
|
| 24 |
+
- **Pixel-level Localization** — generates a segmentation mask over the tampered region.
|
| 25 |
+
- **Natural Language Description** — describes what was changed and how.
|
| 26 |
|
| 27 |
## Model Description
|
| 28 |
|
|
|
|
| 30 |
- **Model type:** Multimodal Vision-Language Model for Image Tampering Detection
|
| 31 |
- **License:** MIT
|
| 32 |
- **Base model:** [SIDA-7B](https://huggingface.co/saberzl/SIDA-7B) (LLaVA + LLaMA-2)
|
| 33 |
+
- **Paper:** [From Masks to Pixels and Meaning](https://huggingface.co/papers/2603.20193)
|
| 34 |
+
- **Repository:** [GitHub](https://github.com/VILA-Lab/PIXAR)
|
| 35 |
|
| 36 |
### Architecture
|
| 37 |
|
|
|
|
| 45 |
|
| 46 |
## Key Contributions
|
| 47 |
|
| 48 |
+
Existing tampering benchmarks rely on coarse object masks as ground truth, which often conflates unedited pixels inside the mask with actual tamper evidence and misses subtle edits outside the mask. PIXAR replaces binary masks with **per-pixel difference maps** $D = |I_\text{orig} - I_\text{gen}|$, thresholded at a tunable $\tau$ to produce dynamic ground truth $M_\tau$ that captures edits at multiple scales.
|
| 49 |
|
| 50 |
The **PIXAR benchmark** provides:
|
| 51 |
+
- **420K+ training pairs** with pixel-level $M_\tau$ maps, semantic class labels, and natural language descriptions.
|
| 52 |
+
- **40K balanced test pairs** spanning 8 manipulation types (replace, remove, splice, inpaint, attribute change, colorization, etc.).
|
| 53 |
|
| 54 |
PIXAR-7B achieves **2.6× IoU improvement** over prior state of the art on the PIXAR benchmark.
|
| 55 |
|
| 56 |
## How to Get Started
|
| 57 |
|
| 58 |
+
For interactive inference, see the official implementation on GitHub:
|
| 59 |
|
| 60 |
```bash
|
| 61 |
python chat.py --version jiachengcui888/PIXAR-7B --precision bf16 --seg_prompt_mode seg_only
|
|
|
|
| 78 |
| Batch size | 2 |
|
| 79 |
| Precision | bf16 |
|
| 80 |
| Threshold τ | 0.05 |
|
| 81 |
+
| text loss weight | 3.0 |
|
| 82 |
+
| cls loss weight | 1.0 |
|
| 83 |
+
| bce loss weight | 1.0 |
|
| 84 |
+
| dice loss weight | 1.0 |
|
| 85 |
+
| sem loss weight | 0.5 |
|
| 86 |
|
| 87 |
|
| 88 |
## Evaluation
|
|
|
|
| 92 |
## Citation
|
| 93 |
|
| 94 |
```bibtex
|
| 95 |
+
@article{shang2026masks,
|
| 96 |
+
title={From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering},
|
| 97 |
+
author={Shang, Xinyi and Tang, Yi and Cui, Jiacheng and Elhagry, Ahmed and Al Khatib, Salwa K. and Bsharat, Sondos Mahmoud and Liu, Jiacheng and Zhao, Xiaohan and Xue, Jing-Hao and Li, Hao and Khan, Salman and Shen, Zhiqiang},
|
| 98 |
+
journal={arXiv preprint arXiv:2603.20193},
|
| 99 |
+
year={2026}
|
| 100 |
+
}
|
| 101 |
+
```
|