Update model card: drop POPE, correct ScienceQA to 36.0%, add inference modes table

README.md

@@ -30,10 +30,7 @@ A compact vision-language model that uses **foveated attention** to compress eac
 
 | Benchmark | fVLM-135M | SmolVLM2-256M | SmolVLM2-500M | SmolVLM2-2.2B |
 |-----------|:---------:|:------------:|:------------:|:------------:|
-| **ScienceQA** (2017 MCQ) | 36.4% | 73.8% | 80.0% | 89.6% |
-| **POPE** (9000 Y/N) | 50.0%* | — | — | — |
-
-\* POPE at 50% = random baseline. The 135M model always predicts one class. Not reported by SmolVLM2.
+| **ScienceQA** (2017 MCQ) | 36.0% | 73.8% | 80.0% | 89.6% |
 
 > **Key context**: fVLM-135M uses **1 visual token per frame** vs SmolVLM2's 64-256 tokens per image. fVLM-135M has 158M params total — 1.6x smaller than SmolVLM2-256M. The gap on video benchmarks (4-5%) is modest given the extreme compression.
 
@@ -45,7 +42,7 @@ fVLM supports three inference modes with different speed/quality tradeoffs:
 |-----------|:----------:|:-----------:|:--------------:|
 | MVBench | 27.4% | **28.0%** | 27.9% |
 | Video-MME | 26.2% | **29.5%** | 28.7% |
-| ScienceQA | **36.4%** | 35.6% | 35.4% |
+| ScienceQA | 34.7% | **36.0%** | **36.0%** |
 
 - **Coarse-Only**: Single static-query pass (fastest, no foveation)
 - **Coarse→Fine**: Two-pass parallel forward (training mode, with foveated attention)
@@ -54,7 +51,7 @@ fVLM supports three inference modes with different speed/quality tradeoffs:
 ### Analysis
 
 - **Foveation helps on video**: coarse→fine adds +3.3% on Video-MME over coarse-only, confirming that learned "where to look" queries improve video understanding
-- **ScienceQA**: Best at 36.4% with coarse-only — static images don't benefit from temporal foveation
+- **ScienceQA**: Best at 36.0% with coarse_fine/autoregressive modes — foveated attention provides a small benefit even on static images when replicated to 8 frames
 - **Scale gap**: The large gap on ScienceQA (36% vs 74%) shows the 135M backbone limits image reasoning. Video benchmarks are closer because foveated compression is highly efficient for temporal tasks
 
 ## Architecture
@@ -99,8 +96,12 @@ This enables processing **64+ frames** with the same memory as a few frames in t
 
 ## Usage
 
+### Setup
+
 ```python
 import torch
+from torchvision import transforms
+from transformers import AutoTokenizer
 from huggingface_hub import hf_hub_download
 from release.model import FoveatedVLM
 
@@ -119,9 +120,75 @@ model = FoveatedVLM(
 # Load weights
 state_dict = torch.load(ckpt_path, map_location="cpu")
 model.load_state_dict(state_dict)
-model.eval()
+model = model.to("cuda").to(torch.bfloat16).eval()
+
+tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
+
+# Standard DINO preprocessing
+frame_transform = transforms.Compose([
+    transforms.Resize((224, 224)),
+    transforms.ToTensor(),
+    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+])
 ```
 
+### Image Input
+
+**Important**: fVLM treats all inputs as video. Static images must be **replicated to 8 frames** to match training distribution (Stage 2 and 3 used `replicate_image_frames: 8`). Passing a single frame for an image will produce degraded results.
+
+```python
+from PIL import Image
+
+img = Image.open("photo.jpg").convert("RGB")
+frame_tensor = frame_transform(img)                      # [3, 224, 224]
+frames = frame_tensor.unsqueeze(0).repeat(8, 1, 1, 1)   # [8, 3, 224, 224] — replicate to 8
+frames = frames.unsqueeze(0).to("cuda", dtype=torch.bfloat16)  # [1, 8, 3, 224, 224]
+```
+
+### Video Input
+
+For video, sample up to 64 frames uniformly. No replication needed.
+
+```python
+# video_frames: list of PIL Images (sampled from video)
+tensors = [frame_transform(f) for f in video_frames]
+frames = torch.stack(tensors).unsqueeze(0).to("cuda", dtype=torch.bfloat16)
+# frames shape: [1, T, 3, 224, 224] where T = number of frames (1-64)
+```
+
+### Inference
+
+```python
+# Tokenize prompt
+messages = [
+    {"role": "user", "content": "Describe what is happening in this image."},
+]
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+input_ids = tokenizer.encode(text, return_tensors="pt").to("cuda")
+attention_mask = torch.ones_like(input_ids)
+loss_mask = torch.ones_like(input_ids, dtype=torch.float32)
+
+# Forward pass (coarse_fine mode recommended for best quality)
+with torch.no_grad(), torch.amp.autocast("cuda", dtype=torch.bfloat16):
+    result = model(
+        frames=frames,
+        input_ids=input_ids,
+        attention_mask=attention_mask,
+        loss_mask=loss_mask,
+        mode="coarse_fine",
+    )
+# result["logits"]: [B, S, V] text logits
+# result["loss"]: scalar cross-entropy loss
+```
+
+### Inference Modes
+
+| Mode | Description | Use Case |
+|------|-------------|----------|
+| `coarse_only` | Single static-query pass | Fastest; good for images |
+| `coarse_fine` | Two-pass parallel forward | Best overall; uses foveated attention |
+| `autoregressive` | Sequential with KV cache | Highest quality for video |
+
 ## License
 
 Apache 2.0