Update README.md

add readme basics

README.md

@@ -1,3 +1,202 @@
 ---
+language: en
+tags:
+  - vision
+  - comics
+  - sequence-modeling
+  - transformer
+  - feature-extraction
+  - narrative-understanding
 license: mit
 ---
+
+# Comic Strip Encoder v1 (Stage 4)
+
+This model is a **Transformer sequence encoder** designed to generate narrative-aware, contextualized embeddings of comic book page strips. It serves as "Stage 4" of the [Comic Analysis Framework v2.0](https://github.com/RichardScottOZ/Comic-Analysis).
+
+Where [comic-panel-vlm-v1 (Stage 3)](https://huggingface.co/RichardScottOZ/comic-panel-vlm-v1) generates a **512-dimensional embedding per panel in isolation**, this model takes a full page's worth of panel embeddings (a strip) and runs them through a Transformer encoder. Every output embedding is then **conditioned on the panels surrounding it** — the model has learned what a panel means *in the context of the story around it*. The outputs are:
+
+- **`contextualized_panels`** `(N, 512)` — per-panel embeddings enriched with sequential narrative context
+- **`strip_embeddings`** `(512,)` — a single vector summarising an entire page/strip
+
+These are intended as the primary inputs for downstream retrieval, reranking, and narrative analysis tasks (Stage 5).
+
+## Model Architecture
+
+The `comic-strip-encoder-v1` is a **BERT-style Transformer Encoder** (`Stage4SequenceModel`):
+
+1. **Input Projection**: Linear layer mapping 512-d panel embeddings into the model's d_model space.
+2. **Positional Encoding**: Learned positional encodings for panel sequence order.
+3. **Panel Sequence Transformer**:
+   - 6 Transformer encoder layers
+   - 8 attention heads
+   - Pre-norm (LayerNorm before attention) for training stability
+   - Attention masking for variable-length strips (max 16 panels)
+4. **Strip Aggregation**: A learned `[CLS]`-style query attends over all panel outputs to produce a single strip-level vector.
+5. **Task-Specific Heads** (7 total, used during training):
+
+| Head | Task | Paper |
+| :--- | :--- | :--- |
+| `ReadingOrderHead` | Pairwise panel ordering (adjacency matrix) | ComicsPAP |
+| `PanelPickingHead` | Select missing panel from candidates | ComicsPAP |
+| `CharacterCoherenceHead` | Visual identity consistency across panels | ComicsPAP |
+| `VisualClosureHead` | Action continuation plausibility | ComicsPAP |
+| `TextClosureHead` | Dialogue continuation plausibility | ComicsPAP |
+| `CaptionRelevanceHead` | Text-visual alignment scoring | ComicsPAP |
+| `TextClozeHead` | Select correct dialogue given visual context | Text-Cloze |
+
+At inference time only the Transformer backbone + strip aggregator are required for embedding generation. The task heads can be used directly for scoring tasks.
+
+## Training Data & Methodology
+
+The model was trained on sequences of panel embeddings generated by [comic-panel-vlm-v1](https://huggingface.co/RichardScottOZ/comic-panel-vlm-v1) across approximately **1 million comic pages**, filtered for narrative/story content by Stage 2 (CoSMo PSS).
+
+### Research Foundation
+
+- **ComicsPAP** ([arXiv:2503.08561](https://arxiv.org/abs/2503.08561)): Five discriminative tasks for sequential comic understanding. State-of-the-art LMMs perform near chance on these tasks; domain-trained sequence models are necessary.
+- **Text-Cloze** ([arXiv:2403.03719](https://arxiv.org/abs/2403.03719)): Multimodal transformers outperform RNNs by ~10% on dialogue cloze tasks; domain-adapted encoders are critical.
+
+### Training Objectives
+
+```
+L_total = Σ(weighted task losses) + 0.5 * L_contrastive + 0.3 * L_reading_order
+```
+
+Task weights during multi-task training:
+
+```python
+task_weights = {
+    'panel_picking':        1.0,   # Primary ComicsPAP task
+    'text_cloze':           1.0,   # Primary Text-Cloze task
+    'reading_order':        0.7,
+    'visual_closure':       0.8,
+    'text_closure':         0.8,
+    'character_coherence':  0.5,
+    'caption_relevance':    0.5,
+}
+```
+
+**Key design choice — discriminative not generative**: candidates are selected from a pool rather than generated, following the ComicsPAP framework. This makes training tractable and evaluation unambiguous.
+
+## Usage
+
+The codebase is available at the [Comic Analysis GitHub Repository](https://github.com/RichardScottOZ/Comic-Analysis) under `src/version2/stage4_sequence_modeling_framework.py`.
+
+### Example: Generating Strip & Panel Embeddings
+
+```python
+import torch
+from stage4_sequence_modeling_framework import Stage4SequenceModel
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# 1. Initialize model
+model = Stage4SequenceModel(d_model=512, num_layers=6, nhead=8).to(device)
+
+# Load weights from Hugging Face
+state_dict = torch.hub.load_state_dict_from_url(
+    "https://huggingface.co/RichardScottOZ/comic-strip-encoder-v1/resolve/main/best_model.pt",
+    map_location=device
+)
+model.load_state_dict(state_dict['model_state_dict'])
+model.eval()
+
+# 2. Inputs: panel embeddings from comic-panel-vlm-v1
+# panel_embeddings: (B, N, 512)  — N panels on one page, up to 16
+# panel_mask:       (B, N)       — True where panel exists
+
+panel_embeddings = torch.randn(1, 6, 512).to(device)  # 1 page, 6 panels
+panel_mask = torch.ones(1, 6, dtype=torch.bool).to(device)
+
+# 3. Generate embeddings
+with torch.no_grad():
+    outputs = model(panel_embeddings, panel_mask)
+
+contextualized_panels = outputs['contextualized_panels']  # (1, 6, 512)
+strip_embedding       = outputs['strip_embedding']        # (1, 512)
+
+print(f"Contextualized panels: {contextualized_panels.shape}")
+print(f"Strip embedding:       {strip_embedding.shape}")
+```
+
+### Example: Reading Order Scoring
+
+```python
+with torch.no_grad():
+    # order_matrix[0, i, j] = score indicating if panel i comes before panel j
+    order_matrix = model.reading_order_head(panel_embeddings)  # (1, N, N)
+
+# Compute sorting order based on average row scores
+predicted_order = order_matrix[0].sum(dim=1).argsort(descending=True)
+print(f"Predicted reading order: {predicted_order.tolist()}")
+```
+
+### Example: Panel Picking (ComicsPAP-style)
+
+```python
+# context: panels from the strip with one masked out
+# candidates: 5 panel embeddings (1 correct, 4 distractors)
+context_emb    = contextualized_panels[:, :5, :]   # (1, 5, 512)
+candidate_embs = torch.randn(1, 5, 512).to(device) # (1, 5 candidates, 512)
+
+with torch.no_grad():
+    scores = model.panel_picking_head(context_emb.mean(dim=1), candidate_embs)
+    predicted_idx = scores.argmax(dim=-1)
+print(f"Predicted panel index: {predicted_idx.item()}")
+```
+
+## Pipeline Position
+
+```
+Stage 1: Raw Comics → Panel crops + OCR text
+Stage 2: CoSMo (PSS) → Narrative page classification
+Stage 3: comic-panel-vlm-v1 → Multimodal panel embeddings (V + T + Composition) → (N, 512)
+Stage 4: comic-strip-encoder-v1 → Contextualized panel + strip embeddings ← THIS MODEL
+Stage 5: Storage & Query → Zarr store + semantic search
+```
+
+## Intended Use
+
+- **Narrative reranking**: Stage 3 retrieves top-N candidates; Stage 4 strip embeddings rerank by sequence coherence.
+- **Story-level similarity**: Encode a query as a single panel → Stage 4 → compare strip embeddings across a corpus (story-level search, not panel-level).
+- **Reading order auditing**: Use the `ReadingOrderHead` pairwise matrix to verify or correct panel sequencing in digitised comics.
+- **Narrative flow verification**: Score a proposed page sequence for coherence using the closure heads.
+- **Localisation/dialogue drift auditing**: Use the `TextClozeHead` to flag pages where dialogue is likely misattributed or out of order.
+
+## Limitations
+
+- **Fixed max sequence length**: 16 panels per page (memory constraint at training time).
+- **Discriminative only**: Task heads require candidate sets; not a generative model.
+- **Page-level only**: Does not model multi-page narrative arcs.
+- **Upstream dependency**: Requires Stage 3 (`comic-panel-vlm-v1`) embeddings as input; raw images are not accepted directly.
+- **No explicit character re-identification**: The `CharacterCoherenceHead` scores visual consistency but does not track named characters across pages.
+
+## Performance Expectations
+
+| Task | Expected Accuracy | Random Baseline |
+| :--- | :--- | :--- |
+| Panel Picking | 60–70% | 20% |
+| Visual Closure | 55–65% | 20% |
+| Text Closure | 50–60% | 20% |
+| Reading Order | 75–85% | 50% |
+| Text-Cloze | 50–60% | 25% |
+
+## Citation
+
+If you use this model or the associated framework, please link back to the [Comic Analysis GitHub Repository](https://github.com/RichardScottOZ/Comic-Analysis).
+
+Related work this model is based on:
+
+```bibtex
+@article{comicspap2025,
+  title={ComicsPAP: A Panel-Aware Pipeline for Comic Understanding},
+  year={2025},
+  url={https://arxiv.org/abs/2503.08561}
+}
+
+@article{textcloze2024,
+  title={Text-Cloze: Multimodal Dialogue Prediction in Comics},
+  year={2024},
+  url={https://arxiv.org/abs/2403.03719}
+}
+```