Improve model card: add library, paper link, and correct pipeline tag

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +34 -26
README.md CHANGED
@@ -1,26 +1,28 @@
1
  ---
2
- license: mit
3
  language:
4
- - en
 
 
 
5
  tags:
6
- - image-tampering-detection
7
- - fake-image-detection
8
- - pixel-level-localization
9
- - multimodal
10
- - vision-language-model
11
- - segmentation
12
- base_model: saberzl/SIDA-7B
13
  ---
14
 
15
  # PIXAR-7B
16
 
17
- **PIXAR-7B** is a Vision-Language Model (VLM) for image tampering analysis, introduced in the paper *"From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering"*.
18
 
19
  Given a query image, PIXAR-7B jointly performs:
20
- - **Binary** — real or tampered
21
- - **Object classification** — identifies which of 81 COCO categories was modified
22
- - **Pixel-level localization** — generates a segmentation mask over the tampered region
23
- - **Natural language description** — describes what was changed and how
24
 
25
  ## Model Description
26
 
@@ -28,7 +30,8 @@ Given a query image, PIXAR-7B jointly performs:
28
  - **Model type:** Multimodal Vision-Language Model for Image Tampering Detection
29
  - **License:** MIT
30
  - **Base model:** [SIDA-7B](https://huggingface.co/saberzl/SIDA-7B) (LLaVA + LLaMA-2)
31
- - **Paper:** From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering
 
32
 
33
  ### Architecture
34
 
@@ -42,17 +45,17 @@ PIXAR-7B is built on a LLaVA + LLaMA-2 backbone with LoRA fine-tuning (rank 8),
42
 
43
  ## Key Contributions
44
 
45
- Existing tampering benchmarks rely on coarse object masks as ground truth, which conflates unedited pixels inside the mask with actual tamper evidence and misses subtle edits outside the mask. PIXAR replaces binary masks with **per-pixel difference maps** $D = |I_\text{orig} - I_\text{gen}|$, thresholded at a tunable $\tau$ to produce dynamic ground truth $M_\tau$ that captures edits at multiple scales.
46
 
47
  The **PIXAR benchmark** provides:
48
- - **420K+ training pairs** with pixel-level $M_\tau$ maps, semantic class labels, and natural language descriptions
49
- - **40K balanced test pairs** spanning 8 manipulation types (replace, remove, splice, inpaint, attribute change, colorization, etc.)
50
 
51
  PIXAR-7B achieves **2.6× IoU improvement** over prior state of the art on the PIXAR benchmark.
52
 
53
  ## How to Get Started
54
 
55
- For interactive inference, see the [project repository](https://github.com/VILA-Lab/PIXAR):
56
 
57
  ```bash
58
  python chat.py --version jiachengcui888/PIXAR-7B --precision bf16 --seg_prompt_mode seg_only
@@ -75,11 +78,11 @@ Fine-tuned with DeepSpeed on a LLaVA + LLaMA-2 backbone using LoRA (rank 8). Key
75
  | Batch size | 2 |
76
  | Precision | bf16 |
77
  | Threshold τ | 0.05 |
78
- | text | 3.0 |
79
- | cls | 1.0 |
80
- | bce | 1.0 |
81
- | dice | 1.0 |
82
- | sem | 0.5 |
83
 
84
 
85
  ## Evaluation
@@ -89,5 +92,10 @@ PIXAR-7B achieves **2.6× IoU improvement** over prior SOTA on the PIXAR test be
89
  ## Citation
90
 
91
  ```bibtex
92
-
93
- ```
 
 
 
 
 
 
1
  ---
2
+ base_model: saberzl/SIDA-7B
3
  language:
4
+ - en
5
+ license: mit
6
+ library_name: transformers
7
+ pipeline_tag: image-text-to-text
8
  tags:
9
+ - image-tampering-detection
10
+ - fake-image-detection
11
+ - pixel-level-localization
12
+ - multimodal
13
+ - vision-language-model
14
+ - segmentation
 
15
  ---
16
 
17
  # PIXAR-7B
18
 
19
+ **PIXAR-7B** is a Vision-Language Model (VLM) for image tampering analysis, introduced in the paper [From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering](https://huggingface.co/papers/2603.20193).
20
 
21
  Given a query image, PIXAR-7B jointly performs:
22
+ - **Binary Classification** — real or tampered.
23
+ - **Object Classification** — identifies which of 81 COCO categories was modified.
24
+ - **Pixel-level Localization** — generates a segmentation mask over the tampered region.
25
+ - **Natural Language Description** — describes what was changed and how.
26
 
27
  ## Model Description
28
 
 
30
  - **Model type:** Multimodal Vision-Language Model for Image Tampering Detection
31
  - **License:** MIT
32
  - **Base model:** [SIDA-7B](https://huggingface.co/saberzl/SIDA-7B) (LLaVA + LLaMA-2)
33
+ - **Paper:** [From Masks to Pixels and Meaning](https://huggingface.co/papers/2603.20193)
34
+ - **Repository:** [GitHub](https://github.com/VILA-Lab/PIXAR)
35
 
36
  ### Architecture
37
 
 
45
 
46
  ## Key Contributions
47
 
48
+ Existing tampering benchmarks rely on coarse object masks as ground truth, which often conflates unedited pixels inside the mask with actual tamper evidence and misses subtle edits outside the mask. PIXAR replaces binary masks with **per-pixel difference maps** $D = |I_\text{orig} - I_\text{gen}|$, thresholded at a tunable $\tau$ to produce dynamic ground truth $M_\tau$ that captures edits at multiple scales.
49
 
50
  The **PIXAR benchmark** provides:
51
+ - **420K+ training pairs** with pixel-level $M_\tau$ maps, semantic class labels, and natural language descriptions.
52
+ - **40K balanced test pairs** spanning 8 manipulation types (replace, remove, splice, inpaint, attribute change, colorization, etc.).
53
 
54
  PIXAR-7B achieves **2.6× IoU improvement** over prior state of the art on the PIXAR benchmark.
55
 
56
  ## How to Get Started
57
 
58
+ For interactive inference, see the official implementation on GitHub:
59
 
60
  ```bash
61
  python chat.py --version jiachengcui888/PIXAR-7B --precision bf16 --seg_prompt_mode seg_only
 
78
  | Batch size | 2 |
79
  | Precision | bf16 |
80
  | Threshold τ | 0.05 |
81
+ | text loss weight | 3.0 |
82
+ | cls loss weight | 1.0 |
83
+ | bce loss weight | 1.0 |
84
+ | dice loss weight | 1.0 |
85
+ | sem loss weight | 0.5 |
86
 
87
 
88
  ## Evaluation
 
92
  ## Citation
93
 
94
  ```bibtex
95
+ @article{shang2026masks,
96
+ title={From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering},
97
+ author={Shang, Xinyi and Tang, Yi and Cui, Jiacheng and Elhagry, Ahmed and Al Khatib, Salwa K. and Bsharat, Sondos Mahmoud and Liu, Jiacheng and Zhao, Xiaohan and Xue, Jing-Hao and Li, Hao and Khan, Salman and Shen, Zhiqiang},
98
+ journal={arXiv preprint arXiv:2603.20193},
99
+ year={2026}
100
+ }
101
+ ```