Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,130 +1,206 @@
+---
+base_model: Qwen/Qwen2-VL-2B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Qwen/Qwen2-VL-2B-Instruct
+- lora
+---
 
-# chart-vision-qwen
+# Model Card for Model ID
 
-**Qwen2-VL-2B-Instruct fine-tuned on ChartQA** using LoRA adapters for chart question answering.
+<!-- Provide a quick summary of what the model is/does. -->
 
-*This is Submitted as a deliverable for Orange Problem, for NLP with DL course (UE23AM343BB1)
 
-**Team:** Langrangers (PES University)
-- Aaron Thomas Mathew — PES1UG23AM005
-- Aman Kumar Mishra — PES1UG23AM040
-- Preetham VJ — PES1UG23AM913
 
-**GitHub Repository:** [Aman-K-Mishra/orange-chartqa-slm](https://github.com/Aman-K-Mishra/orange-chartqa-slm)
+## Model Details
 
----
+### Model Description
 
-## Model Description
+<!-- Provide a longer summary of what this model is. -->
 
-Given a chart image (bar chart, line chart, pie chart, etc.) and a natural language question, this model predicts the answer. It was fine-tuned from [Qwen/Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) using LoRA (Low-Rank Adaptation) on the full ChartQA training split (28,299 samples).
 
-| Property | Value |
-|---|---|
-| Base model | Qwen2-VL-2B-Instruct |
-| Fine-tuning method | LoRA (PEFT) |
-| Dataset | HuggingFaceM4/ChartQA |
-| Training samples | 28,299 |
-| Trainable parameters | 4.36M (0.20% of 2.21B) |
-| Hardware | Tesla T4 (15.6 GB VRAM) |
-| Epochs | 1 |
 
----
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
 
 ## Training Details
 
-### LoRA Configuration
-| Parameter | Value | Reason |
-|---|---|---|
-| Rank (`r`) | 16 | rank=8 too small for chart reasoning; rank=32 risks OOM |
-| Alpha | 32 | Standard `alpha = 2×rank` rule |
-| Dropout | 0.05 | Light regularisation to prevent adapter overfitting |
-| Target modules | q_proj, k_proj, v_proj, o_proj | Most impactful attention projections for VLMs |
-
-### Training Hyperparameters
-| Parameter | Value | Reason |
-|---|---|---|
-| Batch size | 1 | OOM fix for T4 with max_length=768 |
-| Gradient accumulation | 16 steps | Effective batch = 16 |
-| Learning rate | 2e-4 | Standard for LoRA fine-tuning |
-| Max sequence length | 768 | Compromise: 512 too short, 1024 causes OOM |
-| Quantization | 8-bit (BitsAndBytes) | Full precision ≈ 16 GB; 8-bit ≈ 8 GB, safe for T4 |
-| Image resolution | 256–512 patches (28×28) | Matches Qwen2-VL patch size; T4-safe |
-| LR scheduler | Cosine annealing | Smooth decay over full epoch |
+### Training Data
 
----
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-## How to Use
+[More Information Needed]
 
-### Installation
+### Training Procedure
 
-```bash
-pip install transformers peft bitsandbytes accelerate datasets pillow
-```
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-### Load Adapters and Run Inference
+#### Preprocessing [optional]
 
-```python
-from transformers import AutoProcessor, Qwen2VLForConditionalGeneration, BitsAndBytesConfig
-from peft import PeftModel
-from PIL import Image
-import torch
+[More Information Needed]
 
-BASE_MODEL_ID = "Qwen/Qwen2-VL-2B-Instruct"
-ADAPTER_REPO  = "preethamvj/chart-vision-qwen"
 
-# Load base model in 8-bit
-bnb_config = BitsAndBytesConfig(load_in_8bit=True)
-model = Qwen2VLForConditionalGeneration.from_pretrained(
-    BASE_MODEL_ID,
-    quantization_config=bnb_config,
-    device_map="auto",
-    torch_dtype=torch.float16
-)
+#### Training Hyperparameters
 
-# Load LoRA adapters
-model = PeftModel.from_pretrained(model, ADAPTER_REPO)
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-# Optional: merge adapters into base weights for faster inference
-model = model.merge_and_unload()
+#### Speeds, Sizes, Times [optional]
 
-processor = AutoProcessor.from_pretrained(
-    BASE_MODEL_ID,
-    min_pixels=256 * 28 * 28,
-    max_pixels=512 * 28 * 28
-)
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-# Inference
-image = Image.open("your_chart.png").convert("RGB")
-question = "What is the highest value in the chart?"
+[More Information Needed]
 
-messages = [{
-    "role": "user",
-    "content": [
-        {"type": "image", "image": image},
-        {"type": "text",  "text": question}
-    ]
-}]
+## Evaluation
 
-text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
-inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")
+<!-- This section describes the evaluation protocols and provides the results. -->
 
-with torch.no_grad():
-    output = model.generate(**inputs, max_new_tokens=64)
+### Testing Data, Factors & Metrics
 
-answer = processor.decode(output[0], skip_special_tokens=True)
-print("Answer:", answer.split("assistant")[-1].strip())
-```
+#### Testing Data
 
----
+<!-- This should link to a Dataset Card if possible. -->
 
-## Intended Use
+[More Information Needed]
 
-This model is intended for chart question answering tasks — reading values, trends, comparisons, and facts from chart images. It is not designed for general visual question answering outside the chart domain.
+#### Factors
 
----
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
 
-## Limitations
+[More Information Needed]
+### Framework versions
 
-- Trained for only 1 epoch due to compute constraints (T4 GPU)
-- Loss shows high variance across steps, suggesting the learning rate may benefit from tuning in future runs
-- Performance may degrade on chart types not well-represented in ChartQA (e.g., highly complex infographics)
+- PEFT 0.18.1

adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2-VL-2B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12da95933985d3e564831a7ee15f976b966def3691c3bb8fbf62ad3e8022e381
+size 17469600

chat_template.jinja

@@ -0,0 +1,7 @@
+{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
+You are a helpful assistant.<|im_end|>
+{% endif %}<|im_start|>{{ message['role'] }}
+{% if message['content'] is string %}{{ message['content'] }}<|im_end|>
+{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
+{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
+{% endif %}

processor_config.json

@@ -0,0 +1,65 @@
+{
+  "image_processor": {
+    "data_format": "channels_first",
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.48145466,
+      0.4578275,
+      0.40821073
+    ],
+    "image_processor_type": "Qwen2VLImageProcessorFast",
+    "image_std": [
+      0.26862954,
+      0.26130258,
+      0.27577711
+    ],
+    "merge_size": 2,
+    "patch_size": 14,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 401408,
+      "shortest_edge": 200704
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen2VLProcessor",
+  "video_processor": {
+    "data_format": "channels_first",
+    "default_to_square": true,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": false,
+    "image_mean": [
+      0.48145466,
+      0.4578275,
+      0.40821073
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.26862954,
+      0.26130258,
+      0.27577711
+    ],
+    "max_frames": 768,
+    "max_pixels": 401408,
+    "merge_size": 2,
+    "min_frames": 4,
+    "min_pixels": 200704,
+    "patch_size": 14,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 12845056,
+      "shortest_edge": 3136
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen2VLVideoProcessor"
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff8cce547abc110590d19c6b5b6e0c6a7b4c8d1012d78b9c42131bae7f494a02
+size 11420367

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "max_pixels": 401408,
+  "min_pixels": 200704,
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "processor_class": "Qwen2VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}