README.md

---
license: mit
tags:
  - computer-vision
  - face-recognition
  - face-verification
  - similarity-metric
  - metric-learning
  - pytorch
datasets:
  - casia-webface
metrics:
  - accuracy
library_name: pytorch
pipeline_tag: image-feature-extraction
---

# IMG — Relational Pattern-Based Similarity Metric

**A Universal Similarity Metric for Computer Vision**

[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.21232756-blue)](https://doi.org/10.5281/zenodo.21232756)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Author:** Imam Ghozali — Independent Researcher
📧 imam.gh98@gmail.com

---

## Model Description

Traditional similarity metrics such as cosine similarity compare embedding vectors through **global angular relationships**.

**IMG** introduces a different paradigm: instead of comparing absolute vector values, IMG compares **local relational patterns** inside the embedding.

The proposed framework consists of three complementary metrics:

1. **IMG Sign Score**
2. **AMP IMG Score**
3. **Chain Score**

> **Note:** This work does **not** propose replacing cosine similarity. Instead, IMG is proposed as an *alternative* similarity metric. Experimental results suggest that the optimal similarity metric depends on how the embedding itself is learned.

---

## Relational Learning Hypothesis

In Javanese, one expresses gratitude as **"matur suwun"**; in Sundanese, the same sentiment is conveyed as **"hatur nuhun"**. Despite different surface structures, both phrases encode identical meaning through internally consistent relational patterns.

This linguistic observation inspired the central hypothesis of this work:

> **Identity can be encoded through consistent relational patterns rather than absolute values.**

Instead of forcing embeddings to occupy a specific angular position, the proposed method trains the network to preserve **local relational consistency**. Similarity is then evaluated by comparing relational patterns rather than absolute vector orientation.


![image](https://cdn-uploads.huggingface.co/production/uploads/6a4c9e235ff747e2704b812e/NnyxNIh8pSXs1t7JNDJmb.png)


---

## Relational Training Objective

Unlike ArcFace, which explicitly optimizes cosine similarity using Angular Margin Loss:

$$
L_{ArcFace} = -\log \frac{e^{s\cos(\theta_y+m)}}{e^{s\cos(\theta_y+m)}+\sum_j e^{s\cos\theta_j}}
$$

the proposed method directly optimizes the desired similarity metric itself.

For two embeddings $E_1, E_2 \in \mathbb{R}^{1024}$, the objective is to maximize their **local sign agreement**.

### Soft Sign Agreement

For each embedding dimension:

$$
a_i = \frac{\tanh(\beta E_{1,i} E_{2,i}) + 1}{2}
$$

where a positive product indicates agreement and a negative product indicates disagreement. Unlike a hard sign comparison, the hyperbolic tangent provides a smooth, differentiable approximation.

### Sliding Window Aggregation

For each sliding window:

$$
S_k = \sum_{i=k}^{k+W-1} a_i
$$

with window size $W = 11$ and threshold $T = 8$.

### Differentiable Matching Gate

$$
M_k = \sigma\big(50(S_k - T + 0.5)\big)
$$

which approximates:

$$
M_k \approx
\begin{cases}
1 & \text{if } S_k \ge T \\
0 & \text{if } S_k < T
\end{cases}
$$

while remaining differentiable.

### IMG Sign Score

$$
IMG(E_1, E_2) = \frac{1}{N}\sum_{k=1}^{N} M_k, \qquad N = d - W + 1
$$

### Relational Loss

For positive pairs:

$$
L_{same} = (1 - IMG)^2
$$

For negative pairs:

$$
L_{diff} = IMG^2
$$

Final objective:

$$
L = L_{same} + L_{diff}
$$

This is exactly the objective used during training — no angular-margin loss, cosine loss, or triplet loss is involved.

---

## Training Data & Hyperparameters

| Hyperparameter | Value |
|---|---:|
| Dataset | CASIA-WebFace |
| Identities | 10,572 |
| Images | ~490k aligned faces |
| Embedding Dimension | 1024 |
| Batch Size | 16 |
| Optimizer | Adam |
| Learning Rate | 1×10⁻⁴ |
| Epochs | 50 |
| Warm-up | 5 |
| Scheduler | Cosine Annealing |
| Weight Decay | 1×10⁻⁵ |

Positive pairs consist of two images belonging to the same identity, while negative pairs are randomly sampled from different identities.

---

## Why Does This Matter?

Traditional face-recognition losses optimize embeddings for cosine similarity. The proposed approach instead optimizes embeddings directly for the intended inference metric. Consequently:

- Embeddings trained with **Angular Margin Loss** naturally favor **cosine similarity**.
- Embeddings trained with the proposed **relational loss** naturally favor **IMG Sign**.

This suggests that **the similarity metric and the embedding loss should be designed together rather than independently.**

---

## Key Idea

| Metric | What it measures |
|---|---|
| **Cosine Similarity** | Global vector direction |
| **IMG Sign** | Local relational sign patterns |
| **AMP IMG** | Relational patterns + local amplitude consistency |
| **Chain Score** | Continuity of matching relational patterns |

---

## Model Architecture

**SW357 Block**

```
Conv2 → Conv3 → Conv4 → Conv5 → Conv6 → Conv7 → Conv8 → Conv9 → Conv10
     → Global Average Pooling → FC → BatchNorm
```

| Property | Value |
|---|---|
| Parameters | 2,774,176 |
| Model Size (FP32) | 10.58 MB |
| Training Dataset | CASIA-WebFace (490k aligned images, 10,572 identities) |

---

## Evaluation Results

### SW357 Embedding (native)

| Dataset | IMG Sign | AMP | Chain | Cosine |
|----------|---------:|-------:|-------:|-------:|
| LFW | 96.27% | 90.45% | 95.12% | 95.53% |
| AgeDB-30 | 78.80% | 74.22% | 72.87% | 77.22% |
| CALFW | 78.73% | 74.92% | 76.87% | 78.32% |
| CPLFW | 76.85% | 68.88% | 75.23% | 74.62% |
| **Combined** | **81.02%** | **77.41%** | **79.30%** | **79.49%** |

### ArcFace Evaluation (relational metric tested on external embedding)

| Dataset | IMG Sign | AMP | Chain | Cosine |
|----------|---------:|-------:|-------:|-------:|
| LFW | 99.58% | 99.48% | 97.02% | 99.82% |
| AgeDB-30 | 96.85% | 93.92% | 73.62% | 98.07% |
| CALFW | 95.62% | 94.52% | 84.18% | 96.10% |
| CPLFW | 93.22% | 91.33% | 77.13% | 94.45% |

**Observation:** Cosine remains the best metric for ArcFace because ArcFace is explicitly optimized using Angular Margin Loss. However, IMG Sign remains highly competitive despite never being used during ArcFace training.

---

## Main Finding

Results suggest that **Similarity Metric** and **Embedding Loss Function** should be considered together:

- Embeddings trained with **Angular Margin Loss** naturally favor **cosine similarity**.
- Embeddings trained with the proposed **relational loss** naturally favor **IMG Sign**.

**Therefore, there is no universally best similarity metric.** The optimal metric depends on how the embedding space is learned.

---
## Domain-Agnostic Potential (Beyond Computer Vision)

While evaluated on face verification, the core mathematics of the **IMG Framework** are inherently domain-agnostic. Because it discards absolute magnitude dependency and focuses entirely on local sign-pattern agreements, this framework can be generalized to non-visual embeddings:

*   **Audio & Speech Processing:** By applying IMG to audio spectrogram embeddings, the metric can eliminate amplitude/volume variations (gain changes), establishing a noise-robust framework for voice biometrics.
*   **Structural Bioinformatics:** In protein structural analysis, exact physical distances fluctuate due to environment/simulations. IMG can be applied to capture invariant relational topology patterns between amino acids rather than relying on strict absolute spatial coordinates.
---
## Metric Reference Implementations

### IMG Sign Score

```python
def img_sign_score_np(e1, e2):
    n = len(e1) - WINDOW_SIZE + 1
    mc = 0
    for i in range(n):
        s1 = np.where(e1[i:i+WINDOW_SIZE] >= 0, 1, -1)
        s2 = np.where(e2[i:i+WINDOW_SIZE] >= 0, 1, -1)
        if np.sum(s1 == s2) >= THRESHOLD:
            mc += 1
    return mc / n
```

### AMP IMG Score

```python
def amp_img_score_np(e1, e2):
    n = len(e1) - WINDOW_SIZE + 1
    total = 0
    for i in range(n):
        w1 = e1[i:i+WINDOW_SIZE]
        w2 = e2[i:i+WINDOW_SIZE]
        s1 = np.where(w1 >= 0, 1, -1)
        s2 = np.where(w2 >= 0, 1, -1)
        if np.sum(s1 == s2) >= THRESHOLD:
            a1 = np.mean(np.abs(w1))
            a2 = np.mean(np.abs(w2))
            total += max(0, 1 - abs(a1 - a2) / max(a1, a2, 1e-6))
    return total / n
```

### Chain Score

```python
def chain_score_np(e1, e2):
    n = len(e1) - WINDOW_SIZE + 1
    flags = []
    for i in range(n):
        ...
    total = sum(flags)
    img_sign = total / n
    ...
    avg_chain = total / n_chains
    diff = avg_chain - NEUTRAL_LEN
    score = img_sign + (
        REWARD_RATE * diff
        if diff >= 0
        else PUNISH_RATE * diff
    ) / 100
    return np.clip(score, 0, 1)
```

---

## Datasets

| Dataset | Link | Description |
|---------|------|-------------|
| CASIA-WebFace aligned | [Kaggle](https://www.kaggle.com/datasets/luongkhang04/aligned-casia) | Training dataset, aligned & cropped, 490k images, 10,572 identities |
| Benchmark (LFW/AgeDB/CALFW/CPLFW) | [Kaggle](https://www.kaggle.com/datasets/yakhyokhuja/agedb-30-calfw-cplfw-lfw-aligned-112x112) | Validation datasets, pre-aligned 112×112 |

```
train/
  train_sw357_conv10_imgsign_a100.py  — Training on A100/Colab
  train_eval_sw357_conv10_gtx.py      — 1-epoch test on GTX
  train_eval_sw357_conv13_gtx.py      — Conv13 variant test
  precrop_casia.py                    — Pre-crop CASIA with MTCNN

eval/
  eval_lfw_gtx_chain_conv10.py        — Eval Conv10 + Chain Score (GTX)
  eval_lfw_gtx_imgsign_conv10.py      — Eval Conv10 IMG Sign (GTX)
  eval_benchmarks_a100.py             — Multi-dataset benchmark (A100)
  eval_metric_comparison_a100.py      — FaceNet/ArcFace metric test

app/
  face_compare_conv10.py              — Desktop UI comparison app (tkinter)
```

---

## How to Use

### 1. Install dependencies

```bash
pip install torch torchvision facenet-pytorch insightface Pillow numpy scikit-learn
```

### 2. Download checkpoint

Place `best_model_epoch39_plateau.pth` in your working directory.

Mirror download link: https://zenodo.org/records/21232756

### 3. Eval on LFW

```bash
# Edit CKPT_PATH and LFW_DIR in the script first
python eval_lfw_gtx_imgsign_conv10.py
```

### 4. Run comparison app

```bash
python face_compare_conv10.py
```

---

## Voting System

Three metrics, one threshold (from IMG Sign sweep):

```
2/3 or 3/3 pass → ✅ MATCH
1/3 pass        → ⚠️  UNCERTAIN
0/3 pass        → ❌ DIFFERENT
```

---



![diff](https://cdn-uploads.huggingface.co/production/uploads/6a4c9e235ff747e2704b812e/IFKe-tLJ7KHKCOeHsAtkR.png)
![same](https://cdn-uploads.huggingface.co/production/uploads/6a4c9e235ff747e2704b812e/BOt0LBdZuUJMlLRdd4i2R.png)

During development of the interactive ablation visualizer, a preliminary observation was made using the custom polygon occlusion tool:

Observation: Region-specific embedding sensitivity

When occluding specific facial regions (e.g., right eye) using a custom polygon mask and comparing the resulting embedding changes across two different individuals:


Same person, different photos: occluding the same region produces delta spikes at similar embedding dimensions across both photos
Different people: occluding the same region produces delta spikes at different embedding dimensions, or in some cases near-zero delta for one person (e.g., when glasses obscure the region)


Example (custom polygon, right eye region):

Same person:
  Photo 1: changed 4/1014 windows = 0.4%  spike_delta: 110
  Photo 2: changed 1/1014 windows = 0.1%  spike_delta: 100
  → spike locations visually correlated

Different people:
  Photo 1: changed 4/1014 windows = 0.4%  spike_delta: 110
  Photo 2: changed 0/1014 windows = 0.0%  spike_delta: 97
  → spike locations differ significantly

This observation suggests that the IMG Sign MSE loss function, through its overlapping sliding window structure, may induce implicit spatial organization in the embedding space — where different facial regions influence different embedding dimensions. However, this has not yet been formally tested and should be treated as a preliminary observation pending rigorous evaluation.


⚠️ This is an informal observation from the visualization tool, not a validated experimental result. Formal ablation study with multiple identities and statistical analysis is planned as future work.
---

---

## Conclusion

IMG is proposed as an alternative similarity metric rather than a replacement for cosine similarity. Experiments indicate that cosine similarity performs best for embeddings trained with angular-margin objectives, while IMG Sign performs best for embeddings trained with the proposed relational objective. The framework is model-agnostic and can be applied to embeddings generated by different architectures.

---

## Citation

If you use this work, please cite via:

- **Zenodo (DOI):** https://doi.org/10.5281/zenodo.21232755
- **GitHub:** https://github.com/imamgh11/imgnet
- **Hugging Face:** https://huggingface.co/imghost11/imgnetV1

---

## License

This project is licensed under the **MIT License**.