Update README.md

README.md

@@ -1,61 +1,266 @@
 ---
-library_name: transformers
-license: other
-base_model: Qwen/Qwen3-VL-8B-Instruct
-tags:
-- llama-factory
-- full
-- generated_from_trainer
-model-index:
-- name: Qwen3-VL-8B-Instruct
-  results: []
+license: apache-2.0
+language:
+- en
+base_model:
+- Qwen/Qwen3-VL-8B-Instruct
+pipeline_tag: image-text-to-text
 ---
+# OctoMed-8B Model Card
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+## OctoMed-8B
 
-# Qwen3-VL-8B-Instruct
+OctoMed-8B is a high-performance multimodal medical reasoning model created through large-scale data curation and supervised fine-tuning (SFT). To support reliable clinical reasoning, we developed a scalable data pipeline that distills structured reasoning traces from DeepSeek-R1 and GPT-4o and produced the largest multimodal medical reasoning dataset to date with more than 8 million traces and 6.8 billion response tokens.
 
-This model is a fine-tuned version of [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) on the OctoMed_HeadQA, the OctoMed_MedQA-5options, the OctoMed_MedMCQA, the OctoMed_II-Medical-SFT, the OctoMed_PMC-Instruct, the OctoMed_RexVQA, the OctoMed_GLM-5.1-Reasoning, the OctoMed_KIMI-K2.5-Reasoning, the OctoMed_GLM-Multiturn-CoT, the OctoMed_GPT-OSS-120B-Reasoning, the OctoMed_Gemini-Hard-Reasoning, the OctoMed_Opus-Reasoning-Filtered, the OctoMed_OrzMath, the OctoMed_Medical-Reasoning-SFT-Mega, the OctoMed_BloodMNIST, the OctoMed_DermaMNIST, the OctoMed_BreastMNIST, the OctoMed_PAD_UFES_20, the OctoMed_RetinaMNIST, the OctoMed_PneumoniaMNIST, the OctoMed_TissueMNIST, the OctoMed_OCTMNIST, the OctoMed_BrainTumor, the OctoMed_CoronaHack, the OctoMed_BCSS, the OctoMed_Aptos, the OctoMed_Messidor2, the OctoMed_SLAKE, the OctoMed_VQA-RAD and the OctoMed_Identity datasets.
+Using Qwen3-VL-8B-Instruct as the base model, OctoMed-8B is trained on this curated corpus and achieves strong, robust performance on a wide range of out-of-distribution medical benchmarks.
 
-## Model description
+OctoMed-8B produces internal reasoning traces in `<think>...</think>` tokens before writing out its final answer. In general, the model has a tendency to think longer for harder or ill-defined questions, while sticking to shorter reasoning traces for easier queries.
 
-More information needed
+## Requirements
 
-## Intended uses & limitations
+We recommend installing the transformers version used in our experiments and other dependencies with this command:
 
-More information needed
+```bash
+pip install transformers==4.57.1 accelerate==1.12.0 torchvision==0.24.1 qwen-vl-utils==0.0.14
+```
 
-## Training and evaluation data
+## Quickstart
 
-More information needed
+### Inference with HF Transformers 🤗
 
-## Training procedure
+Here we show a code snippet to show you how chat with OctoMed-8B using `transformers` and `qwen_vl_utils`:
 
-### Training hyperparameters
+```python
+import torch
+from transformers import Qwen3VLForConditionalGeneration, AutoTokenizer, AutoProcessor
+from qwen_vl_utils import process_vision_info
 
-The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 1
-- eval_batch_size: 8
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 8
-- gradient_accumulation_steps: 64
-- total_train_batch_size: 512
-- total_eval_batch_size: 64
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 0.1
-- num_epochs: 1.0
+# default: Load the model on the available device(s)
+model = Qwen3VLForConditionalGeneration.from_pretrained(
+    "OctoMed/OctoMed-8B", dtype=torch.bfloat16, device_map="auto"
+)
 
-### Training results
+# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
+# model = Qwen3VLForConditionalGeneration.from_pretrained(
+#     "OctoMed/OctoMed-8B",
+#     dtype=torch.bfloat16,
+#     attn_implementation="flash_attention_2",
+#     device_map="auto",
+# )
 
+# The default range for the number of visual tokens per image in the model is 4-16384.
+# You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
+min_pixels = 262144
+max_pixels = 262144
+processor = AutoProcessor.from_pretrained("OctoMed/OctoMed-8B", min_pixels=min_pixels, max_pixels=max_pixels)
 
+# Text-Only Query
+# messages = [
+#     {
+#         "role": "user",
+#         "content": [
+#             {"type": "text", "text": "I've had a persistent dry cough for two weeks but no fever. Could this be allergies, and when should I see a doctor?"},
+#         ],
+#     }
+# ]
 
-### Framework versions
+# General Query
+# messages = [
+#     {
+#         "role": "user",
+#         "content": [
+#             {
+#                 "type": "image",
+#                 "image": "https://cdn.ncbi.nlm.nih.gov/pmc/blobs/51b2/10835941/13323b55fbb5/13256_2024_4349_Fig1_HTML.jpg",
+#             },
+#             {"type": "text", "text": "Describe this image."},
+#         ],
+#     }
+# ]
 
-- Transformers 5.2.0
-- Pytorch 2.9.0a0+145a3a7bda.nv25.10
-- Datasets 4.0.0
-- Tokenizers 0.22.2
+# Multiple Choice Query
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": "https://cdn.ncbi.nlm.nih.gov/pmc/blobs/51b2/10835941/13323b55fbb5/13256_2024_4349_Fig1_HTML.jpg",
+            },
+            {"type": "text", "text": "What orientation was the MRI in image B taken in?\nA. Axial\nB. Coronal\nC. Sagittal\nD. Oblique\n\nPlease reason step-by-step, and put your final answer within \\boxed{}."},
+        ],
+    }
+]
+
+# Preparation for inference
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+
+inputs = inputs.to(device="cuda")
+
+# Inference: Generation of the output
+generated_ids = model.generate(**inputs, max_new_tokens=8192)
+generated_ids_trimmed = [
+    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)
+print(output_text)
+```
+
+### Inference with vLLM
+
+Here we show an example of how to use OctoMed with vLLM (tested with vLLM==0.11.2 and transformers==4.57.1):
+
+```python
+from vllm import LLM, SamplingParams
+from transformers import AutoProcessor
+
+min_pixels = 262144
+max_pixels = 262144
+processor = AutoProcessor.from_pretrained("OctoMed/OctoMed-8B", min_pixels=min_pixels, max_pixels=max_pixels)
+
+llm = LLM(
+    model="OctoMed/OctoMed-8B",
+    trust_remote_code=True,
+    dtype="bfloat16",
+    max_model_len=8192,
+    tensor_parallel_size=4,
+    gpu_memory_utilization=0.8,
+    limit_mm_per_prompt={"image": 1}
+)
+
+# Set up sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=8192,
+)
+
+image_data = []
+
+# Text-Only Query
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "Explain the difference between type 1 and type 2 diabetes."},
+        ],
+    }
+]
+
+# General Query
+# image_data = ['https://cdn.ncbi.nlm.nih.gov/pmc/blobs/51b2/10835941/13323b55fbb5/13256_2024_4349_Fig1_HTML.jpg']
+# messages = [
+#     {
+#         "role": "user",
+#         "content": [
+#             {
+#                 "type": "image",
+#                 "image": image_data[0],
+#             },
+#             {"type": "text", "text": "Describe this image."},
+#         ],
+#     }
+# ]
+
+# Multiple Choice Query
+# image_data = ['https://cdn.ncbi.nlm.nih.gov/pmc/blobs/51b2/10835941/13323b55fbb5/13256_2024_4349_Fig1_HTML.jpg']
+# messages = [
+#     {
+#         "role": "user",
+#         "content": [
+#             {
+#                 "type": "image",
+#                 "image": image_data[0],
+#             },
+#             {"type": "text", "text": "What orientation was the MRI in image B taken in?\nA. Axial\nB. Coronal\nC. Sagittal\nD. Oblique\n\nPlease reason step-by-step, and put your final answer within \\boxed{}."},
+#         ],
+#     }
+# ]
+
+prompt = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True)
+
+if image_data:
+    mm_prompt = {
+        "prompt": prompt,
+        "multi_modal_data": {"image": image_data}
+    }
+else:
+    mm_prompt = {"prompt": prompt}
+
+# Generate response
+outputs = llm.generate([mm_prompt], sampling_params)
+
+# Print the generated response
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"Prompt: {prompt}")
+    print(f"Generated text: {generated_text}")
+    print("-" * 50)
+```
+
+### Suggested Hyperparameters
+
+We suggest using the same settings used in evaluation to reproduce results:
+
+Format multiple choice questions with the following template:
+
+```
+{optional image(s)}
+{question}
+{options, 1 on each line}
+
+Please reason step-by-step, and put your final answer within \boxed{}.
+```
+
+Example Prompt:
+
+```
+{image(s)}
+What orientation was the MRI in image B taken in?
+A: Axial
+B: Coronal
+C: Sagittal
+D: Oblique
+
+Please reason step-by-step, and put your final answer within \boxed{}.
+```
+
+- Use the default system prompt ("You are a helpful assistant.")
+- Extract the answer by looking at the content within the last `\boxed{}`.
+- Temperature of 0.6
+- Top-p of 0.95
+- min_pixels = 262144
+- max_pixels = 262144
+
+### Known Issues
+
+- The model is finetuned for multiple-choice VQA. The model may follow instructions for other tasks but is not extensively tested or post-trained to do so.
+
+We hope to address these concerns moving forward in future iterations!
+
+## Citation
+
+If you find our work helpful, feel free to give us a cite.
+
+```bibtex
+@article{ossowski2025octomed,
+  title={OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning},
+  author={Ossowski, Timothy and Zhang, Sheng and Liu, Qianchu and Qin, Guanghui and Tan, Reuben and Naumann, Tristan and Hu, Junjie and Poon, Hoifung},
+  journal={arXiv preprint arXiv:2511.23269},
+  year={2025}
+}
+```