--- license: mit tags: - computer-vision - face-recognition - face-verification - similarity-metric - metric-learning - pytorch datasets: - casia-webface metrics: - accuracy library_name: pytorch pipeline_tag: image-feature-extraction --- # IMG — Relational Pattern-Based Similarity Metric **A Universal Similarity Metric for Computer Vision** [![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.21232756-blue)](https://doi.org/10.5281/zenodo.21232756) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) **Author:** Imam Ghozali — Independent Researcher 📧 imam.gh98@gmail.com --- ## Model Description Traditional similarity metrics such as cosine similarity compare embedding vectors through **global angular relationships**. **IMG** introduces a different paradigm: instead of comparing absolute vector values, IMG compares **local relational patterns** inside the embedding. The proposed framework consists of three complementary metrics: 1. **IMG Sign Score** 2. **AMP IMG Score** 3. **Chain Score** > **Note:** This work does **not** propose replacing cosine similarity. Instead, IMG is proposed as an *alternative* similarity metric. Experimental results suggest that the optimal similarity metric depends on how the embedding itself is learned. --- ## Relational Learning Hypothesis In Javanese, one expresses gratitude as **"matur suwun"**; in Sundanese, the same sentiment is conveyed as **"hatur nuhun"**. Despite different surface structures, both phrases encode identical meaning through internally consistent relational patterns. This linguistic observation inspired the central hypothesis of this work: > **Identity can be encoded through consistent relational patterns rather than absolute values.** Instead of forcing embeddings to occupy a specific angular position, the proposed method trains the network to preserve **local relational consistency**. Similarity is then evaluated by comparing relational patterns rather than absolute vector orientation. ![image](https://cdn-uploads.huggingface.co/production/uploads/6a4c9e235ff747e2704b812e/NnyxNIh8pSXs1t7JNDJmb.png) --- ## Relational Training Objective Unlike ArcFace, which explicitly optimizes cosine similarity using Angular Margin Loss: $$ L_{ArcFace} = -\log \frac{e^{s\cos(\theta_y+m)}}{e^{s\cos(\theta_y+m)}+\sum_j e^{s\cos\theta_j}} $$ the proposed method directly optimizes the desired similarity metric itself. For two embeddings $E_1, E_2 \in \mathbb{R}^{1024}$, the objective is to maximize their **local sign agreement**. ### Soft Sign Agreement For each embedding dimension: $$ a_i = \frac{\tanh(\beta E_{1,i} E_{2,i}) + 1}{2} $$ where a positive product indicates agreement and a negative product indicates disagreement. Unlike a hard sign comparison, the hyperbolic tangent provides a smooth, differentiable approximation. ### Sliding Window Aggregation For each sliding window: $$ S_k = \sum_{i=k}^{k+W-1} a_i $$ with window size $W = 11$ and threshold $T = 8$. ### Differentiable Matching Gate $$ M_k = \sigma\big(50(S_k - T + 0.5)\big) $$ which approximates: $$ M_k \approx \begin{cases} 1 & \text{if } S_k \ge T \\ 0 & \text{if } S_k < T \end{cases} $$ while remaining differentiable. ### IMG Sign Score $$ IMG(E_1, E_2) = \frac{1}{N}\sum_{k=1}^{N} M_k, \qquad N = d - W + 1 $$ ### Relational Loss For positive pairs: $$ L_{same} = (1 - IMG)^2 $$ For negative pairs: $$ L_{diff} = IMG^2 $$ Final objective: $$ L = L_{same} + L_{diff} $$ This is exactly the objective used during training — no angular-margin loss, cosine loss, or triplet loss is involved. --- ## Training Data & Hyperparameters | Hyperparameter | Value | |---|---:| | Dataset | CASIA-WebFace | | Identities | 10,572 | | Images | ~490k aligned faces | | Embedding Dimension | 1024 | | Batch Size | 16 | | Optimizer | Adam | | Learning Rate | 1×10⁻⁴ | | Epochs | 50 | | Warm-up | 5 | | Scheduler | Cosine Annealing | | Weight Decay | 1×10⁻⁵ | Positive pairs consist of two images belonging to the same identity, while negative pairs are randomly sampled from different identities. --- ## Why Does This Matter? Traditional face-recognition losses optimize embeddings for cosine similarity. The proposed approach instead optimizes embeddings directly for the intended inference metric. Consequently: - Embeddings trained with **Angular Margin Loss** naturally favor **cosine similarity**. - Embeddings trained with the proposed **relational loss** naturally favor **IMG Sign**. This suggests that **the similarity metric and the embedding loss should be designed together rather than independently.** --- ## Key Idea | Metric | What it measures | |---|---| | **Cosine Similarity** | Global vector direction | | **IMG Sign** | Local relational sign patterns | | **AMP IMG** | Relational patterns + local amplitude consistency | | **Chain Score** | Continuity of matching relational patterns | --- ## Model Architecture **SW357 Block** ``` Conv2 → Conv3 → Conv4 → Conv5 → Conv6 → Conv7 → Conv8 → Conv9 → Conv10 → Global Average Pooling → FC → BatchNorm ``` | Property | Value | |---|---| | Parameters | 2,774,176 | | Model Size (FP32) | 10.58 MB | | Training Dataset | CASIA-WebFace (490k aligned images, 10,572 identities) | --- ## Evaluation Results ### SW357 Embedding (native) | Dataset | IMG Sign | AMP | Chain | Cosine | |----------|---------:|-------:|-------:|-------:| | LFW | 96.27% | 90.45% | 95.12% | 95.53% | | AgeDB-30 | 78.80% | 74.22% | 72.87% | 77.22% | | CALFW | 78.73% | 74.92% | 76.87% | 78.32% | | CPLFW | 76.85% | 68.88% | 75.23% | 74.62% | | **Combined** | **81.02%** | **77.41%** | **79.30%** | **79.49%** | ### ArcFace Evaluation (relational metric tested on external embedding) | Dataset | IMG Sign | AMP | Chain | Cosine | |----------|---------:|-------:|-------:|-------:| | LFW | 99.58% | 99.48% | 97.02% | 99.82% | | AgeDB-30 | 96.85% | 93.92% | 73.62% | 98.07% | | CALFW | 95.62% | 94.52% | 84.18% | 96.10% | | CPLFW | 93.22% | 91.33% | 77.13% | 94.45% | **Observation:** Cosine remains the best metric for ArcFace because ArcFace is explicitly optimized using Angular Margin Loss. However, IMG Sign remains highly competitive despite never being used during ArcFace training. --- ## Main Finding Results suggest that **Similarity Metric** and **Embedding Loss Function** should be considered together: - Embeddings trained with **Angular Margin Loss** naturally favor **cosine similarity**. - Embeddings trained with the proposed **relational loss** naturally favor **IMG Sign**. **Therefore, there is no universally best similarity metric.** The optimal metric depends on how the embedding space is learned. --- ## Domain-Agnostic Potential (Beyond Computer Vision) While evaluated on face verification, the core mathematics of the **IMG Framework** are inherently domain-agnostic. Because it discards absolute magnitude dependency and focuses entirely on local sign-pattern agreements, this framework can be generalized to non-visual embeddings: * **Audio & Speech Processing:** By applying IMG to audio spectrogram embeddings, the metric can eliminate amplitude/volume variations (gain changes), establishing a noise-robust framework for voice biometrics. * **Structural Bioinformatics:** In protein structural analysis, exact physical distances fluctuate due to environment/simulations. IMG can be applied to capture invariant relational topology patterns between amino acids rather than relying on strict absolute spatial coordinates. --- ## Metric Reference Implementations ### IMG Sign Score ```python def img_sign_score_np(e1, e2): n = len(e1) - WINDOW_SIZE + 1 mc = 0 for i in range(n): s1 = np.where(e1[i:i+WINDOW_SIZE] >= 0, 1, -1) s2 = np.where(e2[i:i+WINDOW_SIZE] >= 0, 1, -1) if np.sum(s1 == s2) >= THRESHOLD: mc += 1 return mc / n ``` ### AMP IMG Score ```python def amp_img_score_np(e1, e2): n = len(e1) - WINDOW_SIZE + 1 total = 0 for i in range(n): w1 = e1[i:i+WINDOW_SIZE] w2 = e2[i:i+WINDOW_SIZE] s1 = np.where(w1 >= 0, 1, -1) s2 = np.where(w2 >= 0, 1, -1) if np.sum(s1 == s2) >= THRESHOLD: a1 = np.mean(np.abs(w1)) a2 = np.mean(np.abs(w2)) total += max(0, 1 - abs(a1 - a2) / max(a1, a2, 1e-6)) return total / n ``` ### Chain Score ```python def chain_score_np(e1, e2): n = len(e1) - WINDOW_SIZE + 1 flags = [] for i in range(n): ... total = sum(flags) img_sign = total / n ... avg_chain = total / n_chains diff = avg_chain - NEUTRAL_LEN score = img_sign + ( REWARD_RATE * diff if diff >= 0 else PUNISH_RATE * diff ) / 100 return np.clip(score, 0, 1) ``` --- ## Datasets | Dataset | Link | Description | |---------|------|-------------| | CASIA-WebFace aligned | [Kaggle](https://www.kaggle.com/datasets/luongkhang04/aligned-casia) | Training dataset, aligned & cropped, 490k images, 10,572 identities | | Benchmark (LFW/AgeDB/CALFW/CPLFW) | [Kaggle](https://www.kaggle.com/datasets/yakhyokhuja/agedb-30-calfw-cplfw-lfw-aligned-112x112) | Validation datasets, pre-aligned 112×112 | ``` train/ train_sw357_conv10_imgsign_a100.py — Training on A100/Colab train_eval_sw357_conv10_gtx.py — 1-epoch test on GTX train_eval_sw357_conv13_gtx.py — Conv13 variant test precrop_casia.py — Pre-crop CASIA with MTCNN eval/ eval_lfw_gtx_chain_conv10.py — Eval Conv10 + Chain Score (GTX) eval_lfw_gtx_imgsign_conv10.py — Eval Conv10 IMG Sign (GTX) eval_benchmarks_a100.py — Multi-dataset benchmark (A100) eval_metric_comparison_a100.py — FaceNet/ArcFace metric test app/ face_compare_conv10.py — Desktop UI comparison app (tkinter) ``` --- ## How to Use ### 1. Install dependencies ```bash pip install torch torchvision facenet-pytorch insightface Pillow numpy scikit-learn ``` ### 2. Download checkpoint Place `best_model_epoch39_plateau.pth` in your working directory. Mirror download link: https://zenodo.org/records/21232756 ### 3. Eval on LFW ```bash # Edit CKPT_PATH and LFW_DIR in the script first python eval_lfw_gtx_imgsign_conv10.py ``` ### 4. Run comparison app ```bash python face_compare_conv10.py ``` --- ## Voting System Three metrics, one threshold (from IMG Sign sweep): ``` 2/3 or 3/3 pass → ✅ MATCH 1/3 pass → ⚠️ UNCERTAIN 0/3 pass → ❌ DIFFERENT ``` --- ![diff](https://cdn-uploads.huggingface.co/production/uploads/6a4c9e235ff747e2704b812e/IFKe-tLJ7KHKCOeHsAtkR.png) ![same](https://cdn-uploads.huggingface.co/production/uploads/6a4c9e235ff747e2704b812e/BOt0LBdZuUJMlLRdd4i2R.png) During development of the interactive ablation visualizer, a preliminary observation was made using the custom polygon occlusion tool: Observation: Region-specific embedding sensitivity When occluding specific facial regions (e.g., right eye) using a custom polygon mask and comparing the resulting embedding changes across two different individuals: Same person, different photos: occluding the same region produces delta spikes at similar embedding dimensions across both photos Different people: occluding the same region produces delta spikes at different embedding dimensions, or in some cases near-zero delta for one person (e.g., when glasses obscure the region) Example (custom polygon, right eye region): Same person: Photo 1: changed 4/1014 windows = 0.4% spike_delta: 110 Photo 2: changed 1/1014 windows = 0.1% spike_delta: 100 → spike locations visually correlated Different people: Photo 1: changed 4/1014 windows = 0.4% spike_delta: 110 Photo 2: changed 0/1014 windows = 0.0% spike_delta: 97 → spike locations differ significantly This observation suggests that the IMG Sign MSE loss function, through its overlapping sliding window structure, may induce implicit spatial organization in the embedding space — where different facial regions influence different embedding dimensions. However, this has not yet been formally tested and should be treated as a preliminary observation pending rigorous evaluation. ⚠️ This is an informal observation from the visualization tool, not a validated experimental result. Formal ablation study with multiple identities and statistical analysis is planned as future work. --- --- ## Conclusion IMG is proposed as an alternative similarity metric rather than a replacement for cosine similarity. Experiments indicate that cosine similarity performs best for embeddings trained with angular-margin objectives, while IMG Sign performs best for embeddings trained with the proposed relational objective. The framework is model-agnostic and can be applied to embeddings generated by different architectures. --- ## Citation If you use this work, please cite via: - **Zenodo (DOI):** https://doi.org/10.5281/zenodo.21232755 - **GitHub:** https://github.com/imamgh11/imgnet - **Hugging Face:** https://huggingface.co/imghost11/imgnetV1 --- ## License This project is licensed under the **MIT License**.