README.md

---
license: mit
tags:
- text
- mujltimodal
- comics
- panel-embeddings
- contrastive-learning
- feature-extraction
---

# Comic Panel VLM Embedder v1

Comic Panel VLM Embedder v1 is a multimodal panel feature extraction model designed to produce rich 512-dimensional embeddings for individual comic book panels.

It represents **Stage 3** of the [Comic Analysis Framework v2.0](https://github.com/RichardScottOZ/Comic-Analysis), and is trained downstream of [CoSMo v4](https://huggingface.co/RichardScottOZ/cosmo-v4) (the page classifier that filters raw archives to narrative content). Its embeddings are intended as input to Stage 4 sequence modeling and similarity search over comic page collections.

This v1 iteration uses **VLM-enriched panel descriptions** generated by Gemini 2.5 Flash Lite — replacing the sparse OCR text used in prior versions — enabling the model to ground panel embeddings in narrative content (character descriptions, dialogue, mood, scene context) rather than raw detected text alone.

---

## Model Architecture

The model is based on the `PanelFeatureExtractor` class. It fuses three independent modalities per panel into a single 512-dim embedding using an **adaptive gated fusion** mechanism.

### 1. Visual Encoder — Dual Backbone (`~111M params, frozen`)

| Component | Model | Output Dim |
|---|---|---|
| SigLIP | `google/siglip-base-patch16-224` | 768 → 512 |
| ResNet50 | `timm/resnet50` (pretrained) | 2048 → 512 |

Both visual backbones are frozen during training. Their 512-dim features are combined using a learned **attention fusion**: a 2-layer MLP computes a softmax weight over the two streams, producing a single 512-dim visual feature.

### 2. Text Encoder (`~22M params, frozen`)

| Component | Model | Output Dim |
|---|---|---|
| Sentence Transformer | `sentence-transformers/all-MiniLM-L6-v2` | 384 → 512 |

Panel text is constructed from the VLM analysis JSON: `description` + joined `text_content[].text` dialogue. The backbone is frozen; a single linear projection maps 384 → 512.

### 3. Compositional Encoder (`~0.2M params, trainable`)

A 3-layer MLP encodes 7 spatial/layout features per panel:

| Index | Feature |
|---|---|
| 0 | Aspect ratio (w/h) |
| 1 | Relative area (panel / page) |
| 2–4 | Reserved (zeros, future expansion) |
| 5 | Normalized centre X |
| 6 | Normalized centre Y |

Panel bounding boxes are sourced from the VLM JSON `box_2d` field (`[y1, x1, y2, x2]` in 0–1000 normalised coordinates, converted from Gemini 2.5 Flash Lite output).

### 4. Adaptive Fusion (`~0.8M params, trainable`)

An `AdaptiveFusion` module independently normalises each modality with `LayerNorm(512)`, then computes a 3-way softmax gate over the concatenation of all three features plus optional modality presence indicators. The final embedding is a weighted sum of the three normalised modalities plus a small learned residual.

**Total: ~115M params | ~2.5M trainable (frozen backbones)**

---

## Training

| Setting | Value |
|---|---|
| Training pages | 923,860 narrative comic pages |
| Val pages | 48,624 |
| Page source | 1.2M page archive, filtered by CoSMo v4 PSS labels |
| VLM annotation | Gemini 2.5 Flash Lite (panel description, dialogue, characters, mood) |
| Epochs | 9 (best checkpoint at epoch 9) |
| Batch size | 8 pages (up to 16 panels each) |
| Image size | 224 × 224 |
| Optimiser | AdamW (lr=1e-4, weight_decay=0.01) |
| Scheduler | CosineAnnealingWarmRestarts |
| Backbones | Frozen |

### Training Objectives

```
Loss = 1.0 × L_contrastive + 0.5 × L_reconstruction + 0.3 × L_modality_alignment
```

- **Contrastive**: Panels from the same page should be mutually similar (temperature=0.07)
- **Reconstruction**: Predict one masked panel embedding from the remaining context
- **Modality alignment**: Cross-entropy alignment between vision and text embeddings for the same panel

### Loss Curve

| Epoch | Train Loss | Val Loss |
|---|---|---|
| 1 | 2.639 | 3.156 |
| 2 | 2.509 | 2.848 |
| 3 | 2.477 | 2.762 |
| 4 | 2.459 | 2.687 |
| 5 | 2.448 | 2.631 |
| 6 | 2.438 | 2.603 |
| 7 | 2.431 | 2.594 |
| 8 | 2.423 | 2.562 |
| **9** | **2.431** | **2.561** ✅ |

---

## Input Format

The model operates per-panel. For a given page it expects:

- **Panel image crop**: `(3, 224, 224)` float32 tensor, normalised with ImageNet mean/std
- **Panel text tokens**: `input_ids` + `attention_mask` from `all-MiniLM-L6-v2` tokenizer (max 128 tokens), constructed from `description + dialogue`
- **Compositional features**: `(7,)` float32 tensor of spatial/layout values
- **Modality mask**: `(3,)` binary indicator — `[has_vision, has_text, has_comp]`

At the page level, up to 16 panels are batched together with zero-padding. A boolean `panel_mask` `(N,)` indicates valid vs padded slots.

---

## Output Format

```python
panel_embeddings: (N_panels, 512)  # float32, one embedding per panel
```

When run over a full dataset via `generate_stage3_embeddings_vlm.py`, output is stored in a Zarr store:

```
stage3_embeddings_vlm.zarr/
├── panel_embeddings   shape: (N_pages, 16, 512)  float32
└── panel_masks        shape: (N_pages, 16)        bool
```

---

## Usage

Because the model requires VLM-annotated panel JSONs, inference uses the pipeline scripts from the [Comic Analysis Repository](https://github.com/RichardScottOZ/Comic-Analysis).

### Quick Start — Load Model

```python
import torch
from stage3_panel_features_framework import PanelFeatureExtractor

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = PanelFeatureExtractor(
    visual_backbone='both',
    visual_fusion='attention',
    feature_dim=512,
    freeze_backbones=True
).to(device)

checkpoint = torch.load('best_model_vlm.pt', map_location=device)
state_dict = checkpoint.get('model_state_dict', checkpoint)
model.load_state_dict(state_dict)
model.eval()
```

### Generate Embeddings for a Dataset

```bash
python src/version2/generate_stage3_embeddings_vlm.py \
  --manifest manifests/master_manifest_20251229.csv \
  --vlm_cache_dir /data/vlm_cache \
  --pss_labels pss_labels_v1.json \
  --checkpoint checkpoints/stage3_vlm/best_model_vlm.pt \
  --output_zarr stage3_embeddings_vlm.zarr \
  --output_metadata stage3_metadata_vlm.json \
  --batch_size 64 \
  --num_workers 8
```

---

## Intended Use

This model is designed as an intermediate representation layer in a comic analysis pipeline:

1. **CoSMo v4** classifies pages → filters to narrative pages
2. **This model** embeds panels → 512-dim per-panel features
3. **Stage 4 (PanelSequenceTransformer)** contextualises panel sequences → strip embeddings
4. **Stage 5 search** performs similarity search over the final embeddings

Panel embeddings from this model are suitable for:
- Similarity search over individual panels (find visually/narratively similar panels)
- Input to sequence models that require panel-level features
- Downstream clustering or classification of panels

They are **not** recommended for cover/advertisement pages — the model was trained exclusively on narrative story pages and its embedding space reflects that distribution.

---

## Citation

If you use this model or the Comic Analysis Framework, please reference the repository:

```
@misc{comic-analysis-framework,
  author = {RichardScottOZ},
  title  = {Comic Analysis Framework v2.0},
  year   = {2026},
  url    = {https://github.com/RichardScottOZ/Comic-Analysis}
}
```