README.md

---
language: en
tags:
  - vision
  - comics
  - sequence-modeling
  - transformer
  - feature-extraction
  - narrative-understanding
license: mit
---

# Comic Strip Encoder v1 (Stage 4)

This model is a **Transformer sequence encoder** designed to generate narrative-aware, contextualized embeddings of comic book page strips. It serves as "Stage 4" of the [Comic Analysis Framework v2.0](https://github.com/RichardScottOZ/Comic-Analysis).

Where [comic-panel-vlm-v1 (Stage 3)](https://huggingface.co/RichardScottOZ/comic-panel-vlm-v1) generates a **512-dimensional embedding per panel in isolation**, this model takes a full page's worth of panel embeddings (a strip) and runs them through a Transformer encoder. Every output embedding is then **conditioned on the panels surrounding it** — the model has learned what a panel means *in the context of the story around it*. The outputs are:

- **`contextualized_panels`** `(N, 512)` — per-panel embeddings enriched with sequential narrative context
- **`strip_embeddings`** `(512,)` — a single vector summarising an entire page/strip

These are intended as the primary inputs for downstream retrieval, reranking, and narrative analysis tasks (Stage 5).

## Model Architecture

The `comic-strip-encoder-v1` is a **BERT-style Transformer Encoder** (`Stage4SequenceModel`):

1. **Input Projection**: Linear layer mapping 512-d panel embeddings into the model's d_model space.
2. **Positional Encoding**: Learned positional encodings for panel sequence order.
3. **Panel Sequence Transformer**:
   - 6 Transformer encoder layers
   - 8 attention heads
   - Pre-norm (LayerNorm before attention) for training stability
   - Attention masking for variable-length strips (max 16 panels)
4. **Strip Aggregation**: A learned `[CLS]`-style query attends over all panel outputs to produce a single strip-level vector.
5. **Task-Specific Heads** (7 total, used during training):

| Head | Task | Paper |
| :--- | :--- | :--- |
| `ReadingOrderHead` | Pairwise panel ordering (adjacency matrix) | ComicsPAP |
| `PanelPickingHead` | Select missing panel from candidates | ComicsPAP |
| `CharacterCoherenceHead` | Visual identity consistency across panels | ComicsPAP |
| `VisualClosureHead` | Action continuation plausibility | ComicsPAP |
| `TextClosureHead` | Dialogue continuation plausibility | ComicsPAP |
| `CaptionRelevanceHead` | Text-visual alignment scoring | ComicsPAP |
| `TextClozeHead` | Select correct dialogue given visual context | Text-Cloze |

At inference time only the Transformer backbone + strip aggregator are required for embedding generation. The task heads can be used directly for scoring tasks.

## Training Data & Methodology

The model was trained on sequences of panel embeddings generated by [comic-panel-vlm-v1](https://huggingface.co/RichardScottOZ/comic-panel-vlm-v1) across approximately **1 million comic pages**, filtered for narrative/story content by Stage 2 (CoSMo PSS).

### Research Foundation

- **ComicsPAP** ([arXiv:2503.08561](https://arxiv.org/abs/2503.08561)): Five discriminative tasks for sequential comic understanding. State-of-the-art LMMs perform near chance on these tasks; domain-trained sequence models are necessary.
- **Text-Cloze** ([arXiv:2403.03719](https://arxiv.org/abs/2403.03719)): Multimodal transformers outperform RNNs by ~10% on dialogue cloze tasks; domain-adapted encoders are critical.

### Training Objectives

```
L_total = Σ(weighted task losses) + 0.5 * L_contrastive + 0.3 * L_reading_order
```

Task weights during multi-task training:

```python
task_weights = {
    'panel_picking':        1.0,   # Primary ComicsPAP task
    'text_cloze':           1.0,   # Primary Text-Cloze task
    'reading_order':        0.7,
    'visual_closure':       0.8,
    'text_closure':         0.8,
    'character_coherence':  0.5,
    'caption_relevance':    0.5,
}
```

**Key design choice — discriminative not generative**: candidates are selected from a pool rather than generated, following the ComicsPAP framework. This makes training tractable and evaluation unambiguous.

## Usage

The codebase is available at the [Comic Analysis GitHub Repository](https://github.com/RichardScottOZ/Comic-Analysis) under `src/version2/stage4_sequence_modeling_framework.py`.

### Example: Generating Strip & Panel Embeddings

```python
import torch
from stage4_sequence_modeling_framework import Stage4SequenceModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 1. Initialize model
model = Stage4SequenceModel(d_model=512, num_layers=6, nhead=8).to(device)

# Load weights from Hugging Face
state_dict = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/RichardScottOZ/comic-strip-encoder-v1/resolve/main/best_model.pt",
    map_location=device
)
model.load_state_dict(state_dict['model_state_dict'])
model.eval()

# 2. Inputs: panel embeddings from comic-panel-vlm-v1
# panel_embeddings: (B, N, 512)  — N panels on one page, up to 16
# panel_mask:       (B, N)       — True where panel exists

panel_embeddings = torch.randn(1, 6, 512).to(device)  # 1 page, 6 panels
panel_mask = torch.ones(1, 6, dtype=torch.bool).to(device)

# 3. Generate embeddings
with torch.no_grad():
    outputs = model(panel_embeddings, panel_mask)

contextualized_panels = outputs['contextualized_panels']  # (1, 6, 512)
strip_embedding       = outputs['strip_embedding']        # (1, 512)

print(f"Contextualized panels: {contextualized_panels.shape}")
print(f"Strip embedding:       {strip_embedding.shape}")
```

### Example: Reading Order Scoring

```python
with torch.no_grad():
    # order_matrix[0, i, j] = score indicating if panel i comes before panel j
    order_matrix = model.reading_order_head(panel_embeddings)  # (1, N, N)

# Compute sorting order based on average row scores
predicted_order = order_matrix[0].sum(dim=1).argsort(descending=True)
print(f"Predicted reading order: {predicted_order.tolist()}")
```

### Example: Panel Picking (ComicsPAP-style)

```python
# context: panels from the strip with one masked out
# candidates: 5 panel embeddings (1 correct, 4 distractors)
context_emb    = contextualized_panels[:, :5, :]   # (1, 5, 512)
candidate_embs = torch.randn(1, 5, 512).to(device) # (1, 5 candidates, 512)

with torch.no_grad():
    scores = model.panel_picking_head(context_emb.mean(dim=1), candidate_embs)
    predicted_idx = scores.argmax(dim=-1)
print(f"Predicted panel index: {predicted_idx.item()}")
```

## Pipeline Position

```
Stage 1: Raw Comics → Panel crops + OCR text
Stage 2: CoSMo (PSS) → Narrative page classification
Stage 3: comic-panel-vlm-v1 → Multimodal panel embeddings (V + T + Composition) → (N, 512)
Stage 4: comic-strip-encoder-v1 → Contextualized panel + strip embeddings ← THIS MODEL
Stage 5: Storage & Query → Zarr store + semantic search
```

## Intended Use

- **Narrative reranking**: Stage 3 retrieves top-N candidates; Stage 4 strip embeddings rerank by sequence coherence.
- **Story-level similarity**: Encode a query as a single panel → Stage 4 → compare strip embeddings across a corpus (story-level search, not panel-level).
- **Reading order auditing**: Use the `ReadingOrderHead` pairwise matrix to verify or correct panel sequencing in digitised comics.
- **Narrative flow verification**: Score a proposed page sequence for coherence using the closure heads.
- **Localisation/dialogue drift auditing**: Use the `TextClozeHead` to flag pages where dialogue is likely misattributed or out of order.

## Limitations

- **Fixed max sequence length**: 16 panels per page (memory constraint at training time).
- **Discriminative only**: Task heads require candidate sets; not a generative model.
- **Page-level only**: Does not model multi-page narrative arcs.
- **Upstream dependency**: Requires Stage 3 (`comic-panel-vlm-v1`) embeddings as input; raw images are not accepted directly.
- **No explicit character re-identification**: The `CharacterCoherenceHead` scores visual consistency but does not track named characters across pages.

## Performance Expectations

| Task | Expected Accuracy | Random Baseline |
| :--- | :--- | :--- |
| Panel Picking | 60–70% | 20% |
| Visual Closure | 55–65% | 20% |
| Text Closure | 50–60% | 20% |
| Reading Order | 75–85% | 50% |
| Text-Cloze | 50–60% | 25% |

## Citation

If you use this model or the associated framework, please link back to the [Comic Analysis GitHub Repository](https://github.com/RichardScottOZ/Comic-Analysis).

Related work this model is based on:

```bibtex
@article{comicspap2025,
  title={ComicsPAP: A Panel-Aware Pipeline for Comic Understanding},
  year={2025},
  url={https://arxiv.org/abs/2503.08561}
}

@article{textcloze2024,
  title={Text-Cloze: Multimodal Dialogue Prediction in Comics},
  year={2024},
  url={https://arxiv.org/abs/2403.03719}
}
```