peeache/Florence-2-Final-Mix-Data-LoRA-128-256

Enhanced Explanations for Kvasir-VQA

This repository contains our process for generating textual and visual explanations on top of the original SimulaMet/Kvasir-VQA-x1 dataset. The work enhances standard VQA answers with grounded reasoning, clinical language, and region-linked visual cues.

---

Textual Explanation Augmentation

We extended the original SimulaMet/Kvasir-VQA-x1 dataset with additional signals:

• Natural VQA answers from SimulaMet/Kvasir-VQA-x1.
• Ground-truth explanations from SimulaMet-HOST/Kvasir-VQA.
• Visual descriptions generated by Gemma 27B, which captured contextual details of the images.

By combining these three sources for each image and question pair, we created enhanced explanations grounded in both natural responses and domain-specific cues.

---

Visual Explanation Augmentation

To complement textual reasoning, we linked region-based visual cues to answers:

• Used pseudo masks generated via prompt-guided segmentation (e.g., ClipSeg).
• Integrated existing polyp and instrument masks from Kvasir-SEG.
• Linked masks to related answers using metadata from SimulaMet/Kvasir-VQA-x1.

This allowed the model to ground its predictions in specific image regions (e.g., polyps, instruments, anatomical landmarks).

---

Training Details

We trained the Florence-2 model with LoRA fine-tuning:

• LoRA config: r=128, a=256
• Tokens used:
  • <MedVQA> {question} → Standard VQA task
  • <MedVQA_EXPLAIN> {question} Explain in Detail → Textual explanation task
  • <REFERRING_EXPRESSION_SEGMENTATION> → Segmentation task (masks converted to Florence-supported location tokens)

⚠️ Note: This version of the model is also trained on the test split.

---

Caption-Based Post-Processing

In addition to VQA answers and explanations, we appended an auto-generated caption using the <MORE_DETAILED_CAPTION> token.

• Interestingly, the model learned to produce better grounded captions after training, even though captioning was never explicitly part of the training objective.
• These captions serve as a natural clinical narrative to enrich explanations.

---

Example JSON Entry

Below is an example of the final output format combining all signals:

{
    "val_id": 1313,
    "img_id": "cl8k2u1qk1ezn0832fzc5hr77",
    "question": "How many polyps are visible in the image?",
    "answer": "one polyp identified",
    "textual_explanation": "One polyp is identified in the image. It is a Paris Ip type polyp, measuring between 5-10mm in size, and appears as a rounded, pale pink mass with a slightly textured surface and small white spots scattered across its surface.\nOverall explaination of image: The image shows a single polypoid lesion in the gastrointestinal tract. The lesion appears pink in color and has a smooth, rounded shape with small white spots scattered across its surface. It is located in the center-right and lower-right regions of the image.",
    "visual_explanation": [
      {
        "type": "segmentation_mask",
        "data": "visuals/_mask_1313.jpg",
        "description": "Highlighted mask showing the region of interest supporting the answer."
      }
    ],
    "confidence_score": 0.9377866668833627
  }

Example Mask for the above:

---

Confidence Calculation

For each generated explanation, we also estimate a confidence score based on the model’s decoding stability:

• At every decoding step, we compute the top-k probability mass (sum of probabilities of the k most likely tokens).
• This top-k mass reflects how concentrated the model’s belief is in its most likely continuations.
• We average these values across all generated tokens to get the final stability confidence score.

This score lies between 0 and 1, with higher values indicating that the model was consistently confident in its token predictions during explanation generation.

Summary

• Textual explanations = Fusion of natural VQA, ground-truth HOST, and visual descriptions.
• Visual explanations = Masks + segmentation linked to VQA metadata.
• Training = Florence-2 with LoRA, multi-task prompting.
• Post-processing = Appended auto-generated captions for better clinical context.