Initial upload of We-Math Phi-4 (multimodal) with model card

README.md

@@ -1,90 +0,0 @@
-## We-Math Phi-4 (Multimodal)
-
-A fine-tuned variant of `Phi-4-multimodal-instruct` for image-based math reasoning on the We-Math dataset.
-
-- **Base model**: `Phi-4-multimodal-instruct`
-- **Repository**: `kalkiai3000/we-math-phi4`
-- **Modalities**: Vision + Text → Text
-- **Intended use**: Solve math problems that include an image (diagrams, handwritten notes, printed problems) with short, stepwise answers.
-
-### Training
-- **Data**: We-Math processed with captions; 1,000 matched samples
-  - Train: 800, Eval: 200 (random split)
-- **Prompting format** (Phi-4 MM): `<|user|><|image_1|>Please solve this math problem: {question}\nImage description: {caption}<|end|><|assistant|>{answer}<|end|>`
-- **Hyperparameters** (Trainer):
-  - `num_train_epochs`: 20
-  - `per_device_train_batch_size`: 1 (effective batch size 8 via `gradient_accumulation_steps=8`)
-  - `per_device_eval_batch_size`: 1
-  - `warmup_steps`: 50
-  - `fp16`: true, `gradient_checkpointing`: true
-  - `eval_strategy`: steps (every 25), `save_steps`: 50, `save_total_limit`: 3
-  - `remove_unused_columns`: false, `label_names`: ["labels"]
-- **Implementation notes**:
-  - Attention implementation forced to eager (flash attention disabled)
-  - KV cache disabled during training (`use_cache=False`)
-  - Minor compatibility shims for recent Transformers cache utilities
-- **Hardware**: NVIDIA A100 GPUs
-- **Seed**: 42
-- **Local paths**: finetuned at `/data-mount-large/models/phi4-wemath-finetuned`, merged at `/data-mount-large/models/phi4-wemath-merged`
-
-### Evaluation
-- **Eval split size**: 200
-- **Metric**: Cross-entropy loss on eval split
-- **Observed eval losses**:
-  - Best eval loss ~ **0.6948** (around step 550)
-  - Later eval loss values trend ~0.78 as training proceeds
-- **Best model selection**: `load_best_model_at_end=True` using `eval_loss` → the exported model corresponds to the best checkpoint.
-- **Approximate perplexity**: exp(0.6948) ≈ **2.00**; exp(0.7794) ≈ **2.18**
-
-### Data processing
-- Images loaded from `images/`; captions loaded from `captions-sentence.json`
-- Each conversation example matched to an image filename; question and answer extracted from structured messages
-- Images resized to max side 1024 and converted to RGB if needed
-
-### Usage
-Minimal example with the Hub model:
-
-```python
-import torch
-from PIL import Image
-from transformers import AutoProcessor, AutoModelForCausalLM
-
-model_id = "kalkiai3000/we-math-phi4"
-processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    trust_remote_code=True,
-    torch_dtype=torch.float16,
-    device_map="auto",
-    attn_implementation="eager",
-)
-model.config.use_cache = False
-
-image = Image.open("/path/to/problem_image.png").convert("RGB")
-question = "What is the area of the triangle?"
-prompt = f"<|user|><|image_1|>Please solve this math problem: {question}<|end|><|assistant|>"
-
-inputs = processor(text=prompt, images=image, return_tensors="pt")
-inputs = {k: (v.to(model.device) if hasattr(v, "to") else v) for k, v in inputs.items()}
-inputs["input_mode"] = torch.tensor([1], dtype=torch.long, device=model.device)
-
-with torch.no_grad():
-    output = model.generate(
-        **inputs,
-        max_new_tokens=128,
-        temperature=0.7,
-        do_sample=True,
-        pad_token_id=processor.tokenizer.eos_token_id,
-        num_logits_to_keep=1,
-    )
-
-print(processor.tokenizer.decode(output[0], skip_special_tokens=True))
-```
-
-### Limitations and bias
-- Trained on a small subset (1k) of We-Math; may not generalize broadly
-- Sensitive to prompt structure; include `<|image_1|>` and the Phi-4 conversation tokens
-- Hallucinations possible; verify results for critical use cases
-
-### Citation
-If you use this model, please cite the base model and this fine-tune.

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' and 'tools' in message and message['tools'] is not none %}{{ '<|' + message['role'] + '|>' + message['content'] + '<|tool|>' + message['tools'] + '<|/tool|>' + '<|end|>' }}{% else %}{{ '<|' + message['role'] + '|>' + message['content'] + '<|end|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>' }}{% else %}{{ eos_token }}{% endif %}"
+}

config.json

@@ -207,8 +207,8 @@
     "r": 320
   },
   "tie_word_embeddings": true,
-  "torch_dtype": "float16",
-  "transformers_version": "4.55.4",
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
   "use_cache": false,
   "vision_lora": {
     "dp": 0.0,

generation_config.json

@@ -6,5 +6,5 @@
     199999
   ],
   "pad_token_id": 199999,
-  "transformers_version": "4.55.4"
+  "transformers_version": "4.51.3"
 }

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d814457a67f12032e31f17b55ce51771f25798e53419a0d4927102378f74a8d4
+size 4989600976

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fbd8beaa84c1972e3f5b6e3b6fa58d536cbee9c3395a480bf7dafae4e04e2645
+size 4988391504

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bf416eb90c4ae287f9fa1ba428a7e84c5b982f58e98876777e6b7017e027c00
+size 4856491072

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46c7d79e1dc207decab5ed00e2a9704d57c8ae7c394d1c884a2aa281aca4c5e5
+size 4983106416

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50fc6423cec56ec15294e98818dea7b67502a815ab28d49afc29e78bec575479
+size 2480555192

tokenizer_config.json

@@ -118,7 +118,9 @@
     "AutoProcessor": "processing_phi4mm.Phi4MMProcessor"
   },
   "bos_token": "<|endoftext|>",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' and 'tools' in message and message['tools'] is not none %}{{ '<|' + message['role'] + '|>' + message['content'] + '<|tool|>' + message['tools'] + '<|/tool|>' + '<|end|>' }}{% else %}{{ '<|' + message['role'] + '|>' + message['content'] + '<|end|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>' }}{% else %}{{ eos_token }}{% endif %}",
   "clean_up_tokenization_spaces": false,
+  "dynamic_hd": 36,
   "eos_token": "<|endoftext|>",
   "extra_special_tokens": {},
   "model_max_length": 131072,