Add files using upload-large-folder tool

README.md

@@ -1,17 +1,213 @@
 ---
 license: mit
+library_name: transformers
+pipeline_tag: image-text-to-text
+language:
+- en
+tags:
+- multimodal
+- vision
+- video
+- long-video
+- token-selection
+- compression
+- qwen2.5-vl
+- qtsplus
 ---
-## ✨ Cite our work
-If you find this repo useful, please consider citing: 
+
+<p>
+  <h1>
+  <img src="./assets/logo_with_glasses.svg" height=150px align="right"/>
+  Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
+  </h1>
+</p>
+
+[![arXiv](https://img.shields.io/badge/arXiv-2511.11910-b31b1b.svg)](https://arxiv.org/abs/2511.11910)
+[![Website](https://img.shields.io/badge/%F0%9F%8C%8E%20Website-Official%20Page-blue)](https://qtsplus.github.io/)
+[![Github](https://img.shields.io/badge/github-repo-green?logo=github)](https://github.com/Siyou-Li/QTSplus)
+
+## Model Description
+![](./assets/qtsplus.svg)
+
+QTSplus-3B-FT is a Qwen2.5-VL–based multimodal LLM finetuned with Query‑Aware Token Selector (QTS+), a lightweight visual token selection module that acts as an information gate between the vision encoder and the LLM.
+
+- Query‑aware selection: scores vision tokens via cross‑attention against the input text query.
+- Adaptive retention: predicts an instance‑specific budget and keeps only the most relevant tokens.
+- Temporal reasoning: a small re‑encoder preserves temporal order with absolute time cues.
+- Efficient long‑video understanding: up to 89% vision token compression and 28% end‑to‑end latency reduction on long videos (see paper for details).
+
+Model sources
+- Paper: Seeing the Forest and the Trees (document.pdf; arXiv:2511.11910)
+- Website: https://qtsplus.github.io/
+- Code: https://github.com/Siyou-Li/QTSplus
+
+## Intended Uses & Limitations
+
+Intended uses
+- Long‑video question answering and captioning
+- Multi‑image reasoning and story understanding
+- Efficient multimodal chat with reduced latency on long inputs
+
+Limitations
+- May miss fine details if the predicted retention budget is too small.
+- Inherits biases and failure modes from the base Qwen2.5‑VL model and training data.
+- Not a safety‑aligned system; outputs may be inaccurate or unsafe without human oversight.
+
+## Quick Start
+
+The repository is designed around a conda‑based Python 3.11 environment with a CUDA‑enabled GPU. The commands below are taken directly from `environment.sh` and provide a reproducible setup on recent Linux distributions.
+
+1. **Create and activate the conda environment**
+
+```bash
+conda create -n qtsplus python=3.11 -y
+conda activate qtsplus
+```
+
+2. **Install toolchain and CUDA toolkit**
+
+```bash
+conda install conda-forge::gcc=11 conda-forge::gxx=11 -y
+conda install nvidia/label/cuda-12.8.1::cuda-toolkit -y
+conda install av -c conda-forge -y
+```
+
+3. **Install PyTorch with CUDA 12.8 support**
+
+```bash
+pip3 install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu128
+```
+
+4. **Install core Python libraries**
+
+```bash
+pip install transformers==4.57.1
+DS_BUILD_CUTLASS_OPS=0 DS_BUILD_RAGGED_DEVICE_OPS=0 DS_BUILD_EVOFORMER_ATTN=0 pip install deepspeed
+pip install accelerate pandas wandb matplotlib scikit-learn datasets evaluate ftfy sentencepiece bitsandbytes
+```
+
+5. **Install FlashAttention (prebuilt wheel)**
+
+```bash
+pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.4.22/flash_attn-2.8.1+cu128torch2.9-cp311-cp311-linux_x86_64.whl
+```
+
+This wheel is specific to Linux x86_64, CUDA 12.8, PyTorch 2.9.0 and Python 3.11; if you deviate from this configuration, you will need to install a compatible FlashAttention build instead.
+
+6. **Verify installation**
+
+After installation, you should be able to run:
+
+```bash
+python -c "import torch, transformers, deepspeed, accelerate; print(torch.cuda.is_available())"
+```
+
+which should print `True` on a correctly configured GPU machine.
+
+Video example
+```python
+import torch, glob, os
+from transformers import AutoModelForCausalLM, AutoProcessor
+from qwen_vl_utils import process_vision_info
+
+model_id = "AlpachinoNLP/QTSplus-3B-FT"
+device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float16
+
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to(dtype=dtype, device=device).eval()
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+
+question = "Summarize the key events in this video."
+video_path = "/path/to/video.mp4"
+
+messages = [{
+    "role": "user",
+    "content": [
+        {"type": "video", "video": video_path, "max_pixels": 360*420, "fps": 1.0},
+        {"type": "text", "text": question},
+    ],
+}]
+
+chat = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+_, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
+
+inputs = processor(text=[chat], images=None, videos=video_inputs, padding=True, return_tensors="pt", **video_kwargs)
+inputs = inputs.to(dtype=torch.float16, device=device)
+
+# Pack vision inputs for QTS+
+pixel_values_videos = inputs.pop("pixel_values_videos", None)
+video_grid_thw = inputs.pop("video_grid_thw", None)
+inputs.pop("second_per_grid_ts", None)
+vision_input = None
+if pixel_values_videos is not None and video_grid_thw is not None:
+    vision_input = {"pixel_values_videos": pixel_values_videos, "video_grid_thw": video_grid_thw}
+
+# Text ids from the question only (exclude special/system/vision tokens)
+question_ids = processor.tokenizer(question, return_tensors="pt", add_special_tokens=False).input_ids.to(dtype=torch.long, device=device)
+
+out_ids = model.generate(vision_input=vision_input, input_ids=inputs.input_ids, question_input_ids=question_ids, max_new_tokens=256)
+trimmed = [o[len(i):] for i, o in zip(inputs.input_ids, out_ids)]
+text = processor.batch_decode(trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=True)
+print(text[0])
+```
+
+Multiple images (treated as a video sequence)
+```python
+images_dir = "/path/to/images"
+image_list = sorted(glob.glob(os.path.join(images_dir, "*.jpg"))) or sorted(glob.glob(os.path.join(images_dir, "*.jpeg")))
+messages = [{
+    "role": "user",
+    "content": [
+        {"type": "video", "video": image_list},
+        {"type": "text", "text": "What story do these images tell?"},
+    ],
+}]
+
+chat = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+_, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
+inputs = processor(text=[chat], images=None, videos=video_inputs, padding=True, return_tensors="pt", **video_kwargs).to(dtype=torch.float16, device=device)
+
+pixel_values_videos = inputs.pop("pixel_values_videos", None)
+video_grid_thw = inputs.pop("video_grid_thw", None)
+inputs.pop("second_per_grid_ts", None)
+vision_input = {"pixel_values_videos": pixel_values_videos, "video_grid_thw": video_grid_thw}
+
+out = model.generate(vision_input=vision_input, input_ids=inputs.input_ids, max_new_tokens=256)
+print(processor.decode(out[0], skip_special_tokens=True))
+```
+
+Notes
+- The chat template is applied via `processor.apply_chat_template` and expects the messages schema shown above.
+- QTS+ expects the vision payload under the `vision_input` keyword argument during generation.
+- For fully offline use, pass `local_files_only=True` to `from_pretrained` calls once the files are cached locally.
+
+## Efficiency & Controls
+
+The following QTS+ hyperparameters in `config.json` control compression and selection behavior:
+- `qts_plus_rho_min` / `qts_plus_rho_max`: min/max retention ratio bounds.(default: 0.05 / 0.5)
+- `qts_plus_tau_s`: scoring temperature for cross‑attention.(default: 0.5)
+- `qts_plus_nmax`: hard cap on selected tokens per sample. （default: 25600)
+These trade off detail vs. speed/memory. See the paper for guidance, ablations, and latency/throughput measurements.
+
+
+## Safety, Bias, and Limitations
+
+- Outputs may be factually incorrect, biased, or unsafe. Do not use without human oversight.
+- QTS+ compresses the vision stream; extremely small budgets may drop rare but important details.
+- Inherits safety/bias characteristics from the underlying Qwen2.5‑VL model and training data.
+
+## Citation
+
+If you find this work helpful, please cite:
 
 ```bibtex
 @misc{li2025seeingforesttreesqueryaware,
-      title={Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models}, 
-      author={Siyou Li and Huanan Wu and Juexi Shao and Yinghao Ma and Yujian Gan and Yihao Luo and Yuwei Wang and Dong Nie and Lu Wang and Wengqing Wu and Le Zhang and Massimo Poesio and Juntao Yu},
-      year={2025},
-      eprint={2511.11910},
-      archivePrefix={arXiv},
-      primaryClass={cs.CV},
-      url={https://arxiv.org/abs/2511.11910}, 
+  title        = {Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models},
+  author       = {Siyou Li and Huanan Wu and Juexi Shao and Yinghao Ma and Yujian Gan and Yihao Luo and Yuwei Wang and Dong Nie and Lu Wang and Wengqing Wu and Le Zhang and Massimo Poesio and Juntao Yu},
+  year         = {2025},
+  eprint       = {2511.11910},
+  archivePrefix= {arXiv},
+  primaryClass = {cs.CV},
+  url          = {https://arxiv.org/abs/2511.11910}
 }
 ```