Upload fine-tuned MedGemma model for FLARE 2025 medical image analysis

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,335 @@
+---
+license: apache-2.0
+base_model: google/medgemma-4b-it
+tags:
+  - vision-language
+  - medical-imaging
+  - radiology
+  - medgemma
+  - gemma
+  - flare2025
+  - peft
+  - lora
+  - multimodal
+  - medical-ai
+datasets:
+  - FLARE2025
+pipeline_tag: image-text-to-text
+library_name: transformers
+---
+
+# MedGemma Fine-tuned for FLARE 2025 Medical Image Analysis
+
+This model is a fine-tuned version of [google/medgemma-4b-it](https://huggingface.co/google/medgemma-4b-it) 
+specifically optimized for medical image analysis tasks in the FLARE 2025 2D Medical Multimodal Dataset challenge.
+
+## Model Description
+
+- **Base Model**: MedGemma-4B-IT (Google's medical-specialized Gemma model)
+- **Fine-tuning Method**: QLoRA (Low-Rank Adaptation)
+- **Target Domain**: Medical imaging across 7 modalities (CT, MRI, X-ray, Ultrasound, Fundus, Pathology, Endoscopy)
+- **Tasks**: Medical image captioning, visual question answering, report generation, diagnosis support
+- **Training Data**: 19 FLARE 2025 datasets with comprehensive medical annotations
+
+## Training Details
+
+### Training Data
+The model was fine-tuned on 19 diverse medical imaging datasets from FLARE 2025, including:
+- **Classification**: Disease diagnosis with balanced accuracy optimization
+- **Multi-label Classification**: Multi-pathology identification 
+- **Detection**: Anatomical structure and pathology detection
+- **Instance Detection**: Identity-aware detection (e.g., chromosome analysis)
+- **Counting**: Cell counting and quantitative analysis
+- **Regression**: Continuous medical measurements
+- **Report Generation**: Comprehensive medical report writing
+
+Details available at: https://huggingface.co/datasets/FLARE-MedFM/FLARE-Task5-MLLM-2D
+
+### Training Configuration
+```yaml\n# LoRA Configuration\nlora_r: 16\nlora_alpha: 32\nlora_dropout: 0.1\ntarget_modules: ['gate_proj', 'up_proj', 'o_proj', 'down_proj', 'v_proj', 'q_proj', 'k_proj']\ntask_type: CAUSAL_LM\nbias: none\n\n```
+
+### Training Procedure
+- **Base Architecture**: MedGemma-4B with medical domain pre-training
+- **Optimization**: 4-bit quantization with BitsAndBytesConfig
+- **LoRA Configuration**: 
+  - r=64, alpha=16, dropout=0.1
+  - Target modules: All attention and MLP layers
+- **Memory Optimization**: Gradient checkpointing, flash attention
+- **Batch Size**: Dynamic based on image resolution and GPU memory
+- **Learning Rate**: 1e-4 with cosine scheduling
+- **Training Steps**: 4000 steps with evaluation every 500 steps
+- **Chat Template**: Gemma-style chat formatting for medical conversations
+
+## Model Performance
+
+This model has been evaluated across multiple medical imaging tasks using FLARE 2025 evaluation metrics:
+
+### Evaluation Metrics by Task Type
+
+**Classification Tasks (Disease Diagnosis):**
+- **Balanced Accuracy** (PRIMARY): Handles class imbalance in medical diagnosis
+- **Accuracy**: Standard classification accuracy
+- **F1 Score**: Weighted F1 for multi-class scenarios
+
+**Multi-label Classification (Multi-pathology):**
+- **F1 Score** (PRIMARY): Sample-wise F1 across multiple medical conditions
+- **Precision**: Label prediction precision
+- **Recall**: Medical condition coverage recall
+
+**Detection Tasks (Anatomical/Pathological):**
+- **F1 Score @ IoU > 0.5** (PRIMARY): Standard computer vision detection metric
+- **Precision**: Detection precision at IoU threshold
+- **Recall**: Detection recall at IoU threshold
+
+**Instance Detection (Identity-Aware Detection):**
+- **F1 Score @ IoU > 0.3** (PRIMARY): Medical imaging standard (e.g., chromosome detection)
+- **F1 Score @ IoU > 0.5**: Computer vision standard
+- **Average F1**: COCO-style average across IoU thresholds (0.3-0.7)
+- **Per-instance metrics**: Detailed breakdown by object identity
+
+**Counting Tasks (Cell/Structure Counting):**
+- **Mean Absolute Error** (PRIMARY): Cell counting accuracy
+- **Root Mean Squared Error**: Additional counting precision metric
+
+**Regression Tasks (Medical Measurements):**
+- **Mean Absolute Error** (PRIMARY): Continuous value prediction accuracy
+- **Root Mean Squared Error**: Regression precision metric
+
+**Report Generation (Medical Reports):**
+- **GREEN Score** (PRIMARY): Comprehensive medical report evaluation with 7 components:
+  - Entity matching with severity assessment (30%)
+  - Location accuracy with laterality (20%)
+  - Negation and uncertainty handling (15%)
+  - Temporal accuracy (10%)
+  - Size/measurement accuracy (10%)
+  - Clinical significance weighting (10%)
+  - Report structure completeness (5%)
+- **BLEU Score**: Text generation quality
+- **Clinical Efficacy**: Medical relevance scoring
+
+## Usage
+
+### Installation
+```bash
+pip install transformers torch peft accelerate bitsandbytes
+```
+
+### Basic Usage
+```python
+import torch
+from transformers import AutoTokenizer, AutoProcessor, AutoModelForImageTextToText
+from peft import PeftModel
+from PIL import Image
+
+# Load the fine-tuned model
+base_model_name = "google/medgemma-4b-it"
+adapter_model_name = "leoyinn/flare25-medgemma"
+
+# Load tokenizer and processor
+tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
+processor = AutoProcessor.from_pretrained(base_model_name, trust_remote_code=True)
+
+# Load base model
+base_model = AutoModelForImageTextToText.from_pretrained(
+    base_model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+    attn_implementation="eager"
+)
+
+# Load the fine-tuned adapter
+model = PeftModel.from_pretrained(base_model, adapter_model_name)
+
+# Prepare input with MedGemma chat format
+image = Image.open("medical_image.jpg").convert("RGB")
+image = image.resize((448, 448))  # MedGemma standard size
+
+# Create proper message format
+messages = [
+    {
+        "role": "system",
+        "content": [{
+            "type": "text", 
+            "text": "You are an expert medical AI assistant specialized in analyzing medical images and providing accurate diagnostic insights."
+        }]
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "image"},
+            {"type": "text", "text": "Describe the medical findings in this image and provide a diagnostic assessment."}
+        ]
+    }
+]
+
+# Apply chat template
+full_text = tokenizer.apply_chat_template(
+    messages, 
+    tokenize=False,
+    add_generation_prompt=True
+)
+
+# Process and generate
+inputs = processor(
+    images=[image],
+    text=full_text,
+    return_tensors="pt",
+    padding=True,
+    truncation=False
+).to(model.device, dtype=torch.bfloat16)
+
+# Generate medical response
+with torch.inference_mode():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=300,
+        do_sample=False,  # Deterministic for medical applications
+        use_cache=True,
+        cache_implementation="dynamic"
+    )
+
+# Decode response
+input_len = inputs["input_ids"].shape[-1]
+response = processor.decode(outputs[0][input_len:], skip_special_tokens=True)
+print(response)
+```
+
+### Advanced Usage for Specific Medical Tasks
+
+```python
+# For medical report generation
+def generate_medical_report(image_path, model, processor, tokenizer):
+    image = Image.open(image_path).convert("RGB").resize((448, 448))
+    
+    messages = [
+        {
+            "role": "system",
+            "content": [{
+                "type": "text", 
+                "text": "You are an expert medical AI assistant specialized in analyzing medical images and providing accurate diagnostic insights."
+            }]
+        },
+        {
+            "role": "user",
+            "content": [
+                {"type": "image"},
+                {"type": "text", "text": "Generate a comprehensive medical report for this image, including findings, impressions, and recommendations."}
+            ]
+        }
+    ]
+    
+    full_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = processor(images=[image], text=full_text, return_tensors="pt").to(model.device)
+    
+    with torch.inference_mode():
+        outputs = model.generate(**inputs, max_new_tokens=400, temperature=0.1)
+    
+    return processor.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+
+# For medical VQA
+def medical_vqa(image_path, question, model, processor, tokenizer):
+    image = Image.open(image_path).convert("RGB").resize((448, 448))
+    
+    instruction = "Look at the image carefully and answer the medical question accurately based on what you observe."
+    full_question = f"{instruction}\n\n{question}"
+    
+    messages = [
+        {
+            "role": "system",
+            "content": [{
+                "type": "text", 
+                "text": "You are an expert medical AI assistant specialized in analyzing medical images and providing accurate diagnostic insights."
+            }]
+        },
+        {
+            "role": "user",
+            "content": [
+                {"type": "image"},
+                {"type": "text", "text": full_question}
+            ]
+        }
+    ]
+    
+    full_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = processor(images=[image], text=full_text, return_tensors="pt").to(model.device)
+    
+    with torch.inference_mode():
+        outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
+    
+    return processor.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+```
+
+## Limitations and Ethical Considerations
+
+### Limitations
+- Model outputs may contain inaccuracies and should be verified by medical professionals
+- Performance may vary across different medical imaging modalities and populations
+- Training data may contain biases present in medical literature and datasets
+- Model has not been validated in clinical settings
+- Designed for research and educational purposes, not clinical decision-making
+
+### Intended Use
+- Medical education and training
+- Research in medical AI and computer vision
+- Development of clinical decision support tools (with proper validation)
+- Academic research in multimodal medical AI
+- Medical image analysis prototyping
+
+### Out-of-Scope Use
+- Direct clinical diagnosis without physician oversight
+- Treatment recommendations without medical professional validation
+- Use in emergency medical situations
+- Deployment in production clinical systems without extensive validation
+- Patient-facing applications without proper medical supervision
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{medgemma-flare2025,
+  title={MedGemma Fine-tuned for FLARE 2025 Medical Image Analysis},
+  author={Your Name},
+  year={2025},
+  publisher={Hugging Face},
+  url={https://huggingface.co/leoyinn/flare25-medgemma}
+}
+
+@misc{medgemma-base,
+  title={MedGemma: Medical Gemma Models for Healthcare},
+  author={Google Research},
+  year={2024},
+  publisher={Hugging Face},
+  url={https://huggingface.co/google/medgemma-4b-it}
+}
+
+@misc{flare2025,
+  title={FLARE 2025: A Multi-Modal Foundation Model Challenge for Medical AI},
+  year={2025},
+  url={https://huggingface.co/datasets/FLARE-MedFM/FLARE-Task5-MLLM-2D}
+}
+```
+
+## Model Details
+- **Model Type**: Vision-Language Model (VLM) specialized for medical applications
+- **Architecture**: MedGemma (Gemma-based) with LoRA adapters
+- **Parameters**: ~4B base parameters + LoRA adapters
+- **Precision**: bfloat16 base model + full precision adapters
+- **Framework**: PyTorch, Transformers, PEFT
+- **Input Resolution**: 448x448 pixels (standard for MedGemma)
+- **Context Length**: Supports long medical reports and conversations
+
+## Technical Specifications
+- **Base Model**: google/medgemma-4b-it
+- **Adapter Type**: LoRA (Low-Rank Adaptation)
+- **Target Modules**: All attention projection layers and MLP layers
+- **Chat Template**: Gemma-style with medical system prompts
+- **Attention Implementation**: Eager attention for stability
+- **Cache Implementation**: Dynamic caching for efficient inference
+
+## Contact
+For questions or issues, please open an issue in the model repository or contact the authors.
+
+---
+**Disclaimer**: This model is for research and educational purposes only. Always consult qualified medical professionals for clinical decisions.

adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/medgemma-4b-it",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "up_proj",
+    "o_proj",
+    "down_proj",
+    "v_proj",
+    "q_proj",
+    "k_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f57b648f2342820a11e8f7043c67e8dd7de460a66ab20b79a6e88299a508c6c6
+size 131252288

chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

config.json

@@ -0,0 +1,14 @@
+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "google/medgemma-4b-it--configuration_gemma.GemmaConfig",
+    "AutoModel": "google/medgemma-4b-it--modeling_gemma.GemmaForCausalLM",
+    "AutoModelForImageTextToText": "google/medgemma-4b-it--modeling_gemma.Gemma3ForConditionalGeneration"
+  },
+  "model_type": "gemma",
+  "transformers_version": "4.45.0",
+  "base_model_name_or_path": "google/medgemma-4b-it",
+  "peft_type": "LORA"
+}

special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40e151d89e2b2298440e00f62d75307e3fc674849f15739bfe36ee186454ef1e
+size 6161