Update README.md

README.md

@@ -16,190 +16,118 @@ tags:
 
 # ZwZ2-4B
 
-### From Seeing Details to Connecting Visual Evidence
+<!-- Replace PAPER_URL, PROJECT_URL, and COLLECTION_URL with the final v2 URLs. -->
+<p align="center">
+  📃 <a href="PAPER_URL">Paper</a> | 🏠 <a href="PROJECT_URL">Project</a> | 🤗 <a href="COLLECTION_URL">Collection</a>
+</p>
 
-**ZwZ2 releases:** [4B](https://huggingface.co/inclusionAI/ZwZ2-4B) · [9B](https://huggingface.co/inclusionAI/ZwZ2-9B) · [27B](https://huggingface.co/inclusionAI/ZwZ2-27B) · [Dataset](https://huggingface.co/datasets/inclusionAI/ZwZ2-RL-VQA)
+## ✨ Model Summary
 
-**First-generation ZwZ:** [Paper](https://huggingface.co/papers/2602.11858) · [Code](https://github.com/inclusionAI/Zooming-without-Zooming) · [Collection](https://huggingface.co/collections/inclusionAI/zooming-without-zooming)
+**ZwZ2-4B** is the **Qwen3.5-4B**-based model in **Zooming-without-Zooming-v2 (ZwZ2)**. It learns to connect visual evidence across regions through comparison, aggregation, matching, and multi-step reasoning. Training uses full images; inference requires no zooming tools.
 
-## Model Summary
+On **25 benchmarks across six domains**, it improves over its base model on every benchmark, increasing overall accuracy from **49.41 to 54.79 (+5.39 points)**. Gains extend from fine-grained perception to spatial reasoning, long-context understanding, multimodal agents, and general visual reasoning.
 
-**ZwZ2-4B** is a vision-language model built on **Qwen3.5-4B**, released as part of the 4B/9B/27B Zooming-without-Zooming-v2 family. As the second generation of **Zooming without Zooming**, it extends fine-grained perception from recognizing local details to **connecting evidence**: locating relevant visual facts, binding them to the right entities, recovering their relations, and composing them into an answer.
+## ⚙️ Method and Data
 
-We introduce **EvidenceConnect**, a data synthesis framework that uses privileged, zoomed views of related regions within a single image to construct questions requiring comparison, aggregation, matching, or multi-step composition. Independent VLM answer agreement and drop-one region tests assess answer reliability and evidence dependence. Verified questions are converted into multiple-choice form and paired with full-image policy views for reinforcement learning with verifiable rewards (**RLVR**), using **GRPO** without an additional supervised fine-tuning stage.
+**EvidenceConnect** builds verified multi-region questions from SA-1B images:
 
-Although trained on single images, **ZwZ2-4B** improves over **Qwen3.5-4B** on all **25 evaluated benchmarks across six domains**. Its overall average increases from **49.41 to 54.79 (+5.39 percentage points)**. Improvements extend from fine-grained and spatial perception to multimodal long-context understanding, embodied spatial reasoning, multimodal agents, and general visual reasoning.
+1. Group 3–7 related regions and generate questions using privileged zoomed views.
+2. Check answers through cross-model agreement and drop-one region tests.
+3. Convert verified questions to four-option tasks and train with **GRPO**, without an additional supervised fine-tuning stage.
 
-Individual benchmark scores are accuracies (%) from the second-generation manuscript; domain means and overall scores follow the latest 25-benchmark results table; Δ denotes absolute percentage-point improvement over the corresponding backbone. Domain averages are unweighted within each domain. Overall follows the latest 25-benchmark results table; it is not a mean of the six domain scores.
+All three model sizes use approximately **20K training examples**. The policy sees full images with colored region outlines; privileged crop layouts are used only for data construction. See [ZwZ2-RL-VQA](https://huggingface.co/datasets/inclusionAI/ZwZ2-RL-VQA) for the data schema and synthesis process.
 
-### Performance Highlights
+## 📊 Performance
 
-![ZwZ2-4B, ZwZ2-9B and ZwZ2-27B on eight representative benchmarks](assets/zwz2_performance_highlights.png)
+![ZwZ2 performance on eight representative benchmarks](assets/zwz2_performance_highlights.png)
 
-Accuracy (%) on eight representative benchmarks: ZoomBench, Perception-Bench, MMSI-Bench, SPAR-Bench, MileBench, ERQA, ViewSpatial-Bench, and MMStar. Full 25-benchmark results are below; overall scores for ZwZ2-4B / 9B / 27B are 54.79 / 56.59 / 60.51.
-
-[View vector PDF](assets/zwz2_performance_highlights_editable.pdf)
-
-<!-- RESULTS_IMAGE_START -->
-### Benchmark Results
+[Editable vector PDF](assets/zwz2_performance_highlights_editable.pdf). The figure shows eight representative benchmarks; the complete results follow.
 
 ![All 25 benchmark results](assets/zwz2_4b_results.png)
 
-[View full-resolution table](assets/zwz2_4b_results.png)
-
-<!-- RESULTS_IMAGE_END -->
-
-**Evaluation scope.** **MMLongBench\*** denotes the visual-core aggregate of **25 non-Counting tasks from V-RAG, Image MM-NIAH, and VH-Multi**, not the full benchmark. Starred entries use the subset or aggregate specified in the manuscript.
-
-## Key Features
-
-- 🔍 **From Details to Evidence Connections:** Learn to compare, aggregate, and compose facts across related regions, beyond recognizing individual visual details.
-- 🎯 **Verified Visual Supervision:** Cross-model answer checks and region-ablation tests filter unreliable labels and questions that do not depend sufficiently on their selected evidence.
-- 📈 **Broad Transfer from Single Images:** Improvements extend to long-context multimodal understanding, embodied spatial reasoning, and multimodal agents.
-- ⚡ **Full-Image Inference:** The perception models operate on full-image inputs without requiring the privileged zoomed crop layouts used during data synthesis.
-
-## How It Works
-
-Zooming-without-Zooming-v2 develops the training-time zooming principle of the first-generation ZwZ into **multi-region evidence connection**:
+[Full-resolution results](assets/zwz2_4b_results.png). Scores are accuracies (%); domain and overall averages weight the corresponding benchmarks equally. **MMLongBench\*** uses the visual core: V-RAG, Image MM-NIAH, and VH-Multi, comprising **25 non-Counting tasks**. Advanced-model image resolution is constrained by their serving platforms.
 
-1. **Discover related evidence.** Screen visually rich images, propose localized regions, and group **3–7 regions** that jointly support a compositional question.
-2. **Synthesize from privileged views.** Use zoomed region layouts to construct questions involving aggregation, comparison, matching, or dependency chains.
-3. **Verify answers and evidence.** Retain reference answers supported by independent VLM families, and use drop-one tests to assess whether the selected regions are needed.
-4. **Build the policy-view task.** Convert each verified question into four-option MCQ form. The training model receives the full image with colored region outlines identifying the referenced regions, rather than the privileged zoomed layouts.
-5. **Train with verifiable rewards.** Apply GRPO directly to the base VLM, rewarding the correct final option letter. Privileged crops are used in data construction, not supplied as additional training-time policy observations.
+## 🚀 Getting Started
 
-## Quickstart
-
-### Installation
-
-Install a hardware-compatible PyTorch build first, then the inference dependencies:
+Install a hardware-compatible PyTorch build, then:
 
 ```bash
 pip install "transformers==5.13.1" accelerate pillow
+export MODEL_PATH="/path/to/ZwZ2-4B"
+export IMAGE_PATH="/path/to/image.jpg"
 ```
 
-The example targets Transformers 5.13.1. All three sizes use `Qwen3_5ForConditionalGeneration`, including the Qwen3.6-27B backbone. Use sufficient accelerator memory for the selected model; `device_map="auto"` distributes weights across the available devices. See the [Transformers Qwen3.5 documentation](https://huggingface.co/docs/transformers/model_doc/qwen3_5).
-
-Keep the weights, processor, tokenizer, and `chat_template.jinja` in the same downloaded model directory.
-
-### Inference
-
-Set `MODEL_PATH` to the downloaded model directory and `IMAGE_PATH` to an existing image file:
-
-```bash
-export MODEL_PATH="/absolute/path/to/ZwZ2-4B"
-export IMAGE_PATH="/absolute/path/to/image.jpg"
-```
+Keep the weights, processor, tokenizer, and `chat_template.jinja` together. All three sizes use `Qwen3_5ForConditionalGeneration`, including the Qwen3.6-27B-based model.
 
 ```python
 import os
-import re
 from pathlib import Path
 import torch
-from transformers import Qwen3_5ForConditionalGeneration, AutoProcessor
+from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
 
 model_path = os.environ["MODEL_PATH"]
-assert Path(model_path, "config.json").is_file(), "MODEL_PATH must be a model directory"
-assert Path(os.environ["IMAGE_PATH"]).is_file(), "IMAGE_PATH must be an existing image"
 processor = AutoProcessor.from_pretrained(model_path)
-# Preserve the training template, including its default answer-tag instruction.
-processor.chat_template = Path(model_path, "chat_template.jinja").read_text(encoding="utf-8")
+processor.chat_template = Path(model_path, "chat_template.jinja").read_text()
 model = Qwen3_5ForConditionalGeneration.from_pretrained(
-    model_path,
-    dtype=torch.bfloat16,
-    device_map="auto",
+    model_path, dtype=torch.bfloat16, device_map="auto"
 ).eval()
 
-messages = [{
-    "role": "user",
-    "content": [
-        {"type": "image", "url": os.environ["IMAGE_PATH"]},
-        {"type": "text", "text": "Describe the relevant objects and their spatial relationships."},
-    ],
-}]
+messages = [{"role": "user", "content": [
+    {"type": "image", "url": os.environ["IMAGE_PATH"]},
+    {"type": "text", "text": "Describe the objects and their spatial relationships."},
+]}]
 inputs = processor.apply_chat_template(
-    messages,
-    tokenize=True,
-    add_generation_prompt=True,
-    return_dict=True,
-    return_tensors="pt",
-    enable_thinking=False,
+    messages, tokenize=True, add_generation_prompt=True,
+    return_dict=True, return_tensors="pt", enable_thinking=False,
 ).to(model.device)
 
 with torch.inference_mode():
-    generated = model.generate(
-        **inputs,
-        max_new_tokens=2048,
-        do_sample=False,
-    )
-
-completion = generated[:, inputs["input_ids"].shape[1]:]
+    output = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
 response = processor.batch_decode(
-    completion, skip_special_tokens=True
+    output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
 )[0]
 print(response)
-
-# Final-answer extraction; this also supports non-MCQ answers.
-matches = re.findall(r"<answer>(.*?)</answer>", response, flags=re.S)
-if matches:
-    print("Final answer:", matches[-1].strip())
-else:
-    print("No <answer> tag found; inspect the raw response above.")
 ```
 
-Use a plain local image path, not a `file://` URI. The example uses **non-thinking mode** explicitly. The packaged template supplies the default training instruction, including the `<answer>...</answer>` final-answer convention. Supplying a custom system message replaces that default instruction. The example's output budget is for demonstration; benchmark reproduction uses benchmark-specific generation settings.
+Use the packaged template with **non-thinking mode**. Its default system instruction requests `<answer>...</answer>`; for MCQs, compare the extracted letter with the bare reference (e.g. `A`). A custom system message replaces that default instruction. For multiple images, add image entries in order and let the processor insert visual tokens. The 2,048-token output budget above is a demonstration setting; benchmark generation settings vary by task.
 
-<details>
-<summary><strong>Multiple-image input and answer extraction</strong></summary>
+## 🧪 Evaluation
 
-Multiple-image questions use the same message format: place the image items in their intended order, followed by the text question. Let the processor insert visual tokens; do not manually add another set of `<image>` tokens to structured messages.
+From the released code repository, use the same **configure → prepare → evaluate** workflow:
 
-For multiple-choice tasks, the training label is a bare letter, e.g. `A`, while the response convention is `<answer>A</answer>`. Extract the tagged answer before comparing it with the label. Record malformed, conflicting, or truncated responses explicitly. Benchmark adapters may have their own answer-extraction policies, which should be preserved when reproducing reported results.
+```bash
+cd eval
+python -m pip install -e '.[api,data]'
+python -m zwz_eval init --directory local/my-eval --bench hr4k,hr8k
+```
 
-Non-thinking mode disables the dedicated thinking channel, but does not guarantee that the model never generates explanatory text. Multi-image capacity and usable context depend on model configuration, image tokenization, runtime limits, and available memory.
+Edit `local/my-eval/model.json` with your model/judge paths, GPU settings, and output directory, and `datasets.json` with the downloaded benchmark paths. Then run:
 
-</details>
+```bash
+bash scripts/prepare_benchmarks.sh local/my-eval
+bash scripts/evaluate.sh local/my-eval
+```
 
-## Training Data
+See the code repository's `eval/README.md` for all 25 benchmark IDs and data requirements. Match its prompts, task selections, preprocessing, generation limits, and judging settings to reproduce the results.
 
-Zooming-without-Zooming-v2 is trained on approximately **20K EvidenceConnect examples** synthesized from **SA-1B** images. The questions connect evidence across regions within a single image and end in four-option answers suitable for rule-based rewards.
+## 📝 Usage Notes
 
-The policy receives full-image views; zoomed crops provide privileged information during data construction. The data are designed around transferable evidence-connection skills rather than downstream benchmark-specific formats. The models are post-trained with **GRPO**, without an additional supervised fine-tuning stage.
+The Transformers example loads a downloaded model directory. Choose accelerator memory appropriate for the checkpoint; automatic device mapping distributes the weights but does not remove their memory requirements. Use an existing local image path in the example.
 
-This second-generation dataset is distinct from the first-generation **ZwZ-RL-VQA** dataset.
+For multiple-choice evaluation, preserve the original question, option order, image order, and answer parser. Keep the complete raw response alongside the extracted answer so malformed or truncated responses remain inspectable. Benchmark scores are taken from the latest results table; the quickstart demonstrates inference rather than reproducing every benchmark configuration.
 
-## Model Family
+## 🤗 Models and Data
 
-| Release | Backbone |
+| Model | Base model |
 |---|---|
-| ZwZ2-4B | Qwen3.5-4B |
-| ZwZ2-9B | Qwen3.5-9B |
-| ZwZ2-27B | Qwen3.6-27B |
-
-This repository contains **ZwZ2-4B**. The three sizes share the EvidenceConnect research framework; results and runtime requirements should be interpreted for the specific checkpoint.
-
-## Related Work
-
-The following citation refers to the **first generation**, not the second-generation manuscript:
-
-```bibtex
-@article{wei2026zooming,
-  title={Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception},
-  author={Wei, Lai and He, Liangbo and Lan, Jun and Dong, Lingzhong and Cai, Yutong and Li, Siyuan and Zhu, Huijia and Wang, Weiqiang and Kong, Linghe and Wang, Yue and Zhang, Zhuosheng and Huang, Weiran},
-  journal={arXiv preprint arXiv:2602.11858},
-  year={2026}
-}
-```
-
-## License
-
-Please refer to the license accompanying the released weights and the applicable backbone license for usage terms.
+| [ZwZ2-4B](https://huggingface.co/inclusionAI/ZwZ2-4B) | Qwen3.5-4B |
+| [ZwZ2-9B](https://huggingface.co/inclusionAI/ZwZ2-9B) | Qwen3.5-9B |
+| [ZwZ2-27B](https://huggingface.co/inclusionAI/ZwZ2-27B) | Qwen3.6-27B |
 
-<details>
-<summary><strong>Limitations and evaluation notes</strong></summary>
+[🤗 ZwZ2-RL-VQA training dataset](https://huggingface.co/datasets/inclusionAI/ZwZ2-RL-VQA)
 
-This model can hallucinate details, misread text, and make counting or spatial-relation errors. Automated synthesis checks reduce label noise but do not remove it completely. Performance varies with image resolution, visual complexity, context length, and prompt format.
+## 📄 Limitations and License
 
-Individual benchmark results are from Table 1 of the second-generation manuscript; domain means and overall scores follow the latest 25-benchmark results table. Overall is reported as provided in the latest 25-benchmark results table. The listed backbones, task subsets, reasoning mode, image processing, generation limits, and scoring rules must be matched for reproducible comparisons. The quickstart is an inference example, not a reproduction script.
+The model can hallucinate details, misread text, and make counting or spatial errors. Performance depends on image resolution, context length, and prompt format. Refer to the license accompanying the released weights and the applicable backbone license for usage terms.
 
-</details>
+**First-generation ZwZ:** [Paper](https://huggingface.co/papers/2602.11858) · [Code](https://github.com/inclusionAI/Zooming-without-Zooming).