Update README.md

README.md

@@ -3,235 +3,133 @@ license: other
 license_name: phi4-model-license
 license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/LICENSE
 language:
-  - en
-  - ur
-  - de
-  - es
-  - tr
-  - fr
-  - it
+- en
+- ur
+- de
+- es
+- tr
+- fr
+- it
 base_model:
-  - microsoft/Phi-4-multimodal-instruct
+- microsoft/Phi-4-multimodal-instruct
 tags:
-  - phi
-  - phi4-multimodal
-  - quantized
-  - visual-question-answering
-  - speech-translation
-  - speech-summarization
-  - audio
-  - vision
-  - gguf
+- phi
+- phi4-multimodal
+- quantized
+- visual-question-answering
+- speech-translation
+- speech-summarization
+- audio
+- vision
+- gguf
 library_name: other
+pipeline_tag: image-to-text
 ---
 
 # Phi-4 Multimodal – Quantized GGUF + Omni Projector
 
-This repository provides **GGUF-converted weights** for running  
-`microsoft/Phi-4-multimodal-instruct` locally using `llama.cpp`.
+This repository provides **pre-converted GGUF weights** for running **[microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)** with a **quantized language model** and a **multimodal projector (mmproj)** on top of a specialized llama.cpp fork.
 
-It includes:
+- **GitHub (code + server setup):** [Ahmed-Shayan-Arsalan/Phi4-multimodal-Quantisized-Llama.cpp](https://github.com/Ahmed-Shayan-Arsalan/Phi4-multimodal-Quantisized-Llama.cpp)
 
-- A **quantized language model (LLM)**  
-- A separate **multimodal projector (mmproj)** containing vision (+ optional audio) encoders  
-
-No additional training was performed.  
-This is a **pure format + quantization conversion** of the original Microsoft model.
+The goal is to make Phi‑4 multimodal practical to run locally for text, vision, and audio tasks. All weights here are format conversions of the original Microsoft model and do not introduce new training data.
 
 ---
 
-# Files
+## Files in This Repository
+
+* **phi4-mm-Q4_K_M.gguf**: Quantized Phi‑4 multimodal **language model** (LLM).
+    * **Quantization:** Q4_K_M (4‑bit group-wise).
+    * **Usage:** Your main `-m` model in llama.cpp.
+* **phi4-mm-omni.gguf**: **Multimodal projector (mmproj)**.
+    * **Contents:** Vision encoder (SigLIP/Navit-style) and audio Conformer encoder.
+    * **Precision:** Stored in **F16 / F32** to preserve multimodal quality.
+    * **Usage:** Your `--mmproj` or `-mm` model in llama.cpp.
+* *(Optional variants)*: `phi4-mm-f16.gguf` (unquantized ref), `phi4-mm-vision-q8.gguf` (alternative quantization).
 
-You should have:
+---
 
-- `phi4-mm-Q4_K_M.gguf`  
-  → Quantized multimodal **language model** (Q4_K_M)
+## Intended Use
 
-- `phi4-mm-omni.gguf`  
-  → **Multimodal projector (mmproj)**  
-  Contains:
-  - Vision encoder (image understanding)
-  - Optional audio Conformer encoder (if enabled in your runtime)
+These GGUF files are designed for:
+* **Local inference** with llama.cpp or compatible runtimes.
+* **Research and experimentation** on multimodal reasoning.
+* **Prototyping** agents that consume text, images, and audio.
 
-You need:
-- ✅ One LLM GGUF  
-- ✅ One mmproj GGUF  
+**Not intended for:**
+* Training from scratch.
+* Any use violating the original [Microsoft Phi-4 License](https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/LICENSE).
 
 ---
 
-# How the MMProj Was Exported
-
-To export the vision + (optionally) audio encoder into a separate GGUF:
+## How These GGUFs Were Created
 
+### 1. Download the Base Model
 ```bash
+git lfs install
+git clone [https://huggingface.co/microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) phi-4-multimodal
+
+# Phi-4-Multimodal Deployment Guide (llama.cpp)
+
+## 2. Export the text LLM to GGUF
 python convert_hf_to_gguf.py \
   /path/to/phi-4-multimodal \
-  --mmproj \
   --outtype f16 \
-  --outfile phi4-mm-omni.gguf
-```
-
-### What This Does
-
-A custom `MmprojModel` / Phi-specific path in `convert_hf_to_gguf.py`:
-
-- Reads tensors from:
-  ```
-  model.embed_tokens_extend.image_embed.*
-  ```
-  and maps them to **CLIP-style names** expected by `llama.cpp`.
+  --outfile phi4-mm-f16.gguf
 
-- Optionally maps audio tensors from:
-  ```
-  model.embed_tokens_extend.audio_embed.*
-  ```
-  to the **Conformer layout** expected by the runtime.
+## 3. Quantize the LLM
+./build/bin/llama-quantize \
+  phi4-mm-f16.gguf \
+  phi4-mm-Q4_K_M.gguf \
+  Q4_K_M
 
-- Writes a valid GGUF file usable as:
-  ```
-  --mmproj
-  ```
-  or
-  ```
-  -mm
-  ```
-  in `llama.cpp`.
-
-⚠️ No training occurs.  
-This is strictly format conversion + optional quantization.
-
----
+## 4. Export the Multimodal Projector (mmproj)
+To extract the vision and audio encoders into a separate GGUF:
 
-# Building llama.cpp
-
-Make sure you have built your `llama.cpp` fork:
+python convert_hf_to_gguf.py \
+  /path/to/phi-4-multimodal \
+  --mmproj \
+  --outtype f16 \
+  --outfile phi4-mm-omni.gguf
 
-```bash
-cmake -B build
-cmake --build build --config Release
-```
+Technical Note: A custom MmprojModel path in the conversion script maps tensors from model.embed_tokens_extend.* to the CLIP-style and Conformer layouts expected by the llama.cpp runtime.
 
 ---
 
-# Running the Model
+## How to Use (llama.cpp)
 
-Assuming:
+### Server Mode (Recommended)
+This exposes an OpenAI-style HTTP API supporting multimodal prompts.
 
-- `phi4-mm-Q4_K_M.gguf` → LLM  
-- `phi4-mm-omni.gguf` → mmproj  
-
----
-
-## Server Mode (Recommended)
-
-```bash
 ./build/bin/llama-server \
   -m /path/to/phi4-mm-Q4_K_M.gguf \
   -mm /path/to/phi4-mm-omni.gguf \
   --host 0.0.0.0 \
   --port 8080
-```
-
-### The Server:
 
-- Exposes an **OpenAI-style HTTP API**
-- Supports **multimodal prompts**
-- Enables text + image (+ audio if runtime supports it)
+Vision: Send image_url parts or MTMD markers.
+Audio: Send audio content according to the multimodal documentation.
 
-### Typical Usage
-
-- Send text prompts to:
-  ```
-  /v1/chat/completions
-  ```
-
-- For vision:
-  - Use `image_url` parts
-  - Or use MTMD markers described in:
-    ```
-    llama.cpp/tools/server/README.md
-    ```
-
-- For audio (if supported in your build):
-  - Send audio content following the multimodal documentation of your runtime
-
----
-
-## CLI Mode
-
-```bash
+### CLI Mode
 ./build/bin/llama-cli \
   -m /path/to/phi4-mm-Q4_K_M.gguf \
   -mm /path/to/phi4-mm-omni.gguf \
   --color \
   --prompt "Explain this image in detail:"
-```
-
-Add image/audio flags as required by your specific `llama.cpp` fork.
-
----
-
-# Example Capabilities
-
-With proper runtime support:
-
-## Text
-- Instruction following  
-- Multi-turn chat  
-- Coding  
-- Reasoning  
-
-## Vision
-- Visual Question Answering (VQA)  
-- Image captioning  
-- Detailed scene description  
-- Chart / document understanding (within model limits)
-
-## Audio (if Conformer path enabled)
-- Automatic Speech Recognition (ASR)  
-- Speech translation (e.g. EN → FR)  
-- Speech summarization  
-
-Performance depends heavily on:
-- Hardware  
-- GPU backend (CUDA, Metal, etc.)  
-- Runtime implementation  
-
----
-
-# Limitations & Risks
-
-### General Model Limitations
-- May hallucinate
-- May misinterpret inputs
-- Not suitable for critical factual workflows without verification
-
-### Multimodal Limitations
-- Vision and audio understanding are powerful but imperfect
-- Do NOT use for:
-  - Medical decisions
-  - Legal advice
-  - Safety-critical systems
-  - Biometric identification
-
-### License Constraints
-
-You must comply with the original license:
-
-Base model:  
-`microsoft/Phi-4-multimodal-instruct`
-
-License:  
-https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/LICENSE
-
-This repository does **not** modify or override that license.
 
 ---
 
-# Acknowledgements
+## Example Capabilities
+- Text: Instruction following, reasoning, coding, multi‑turn chat.
+- Vision: Visual question answering (VQA), captioning, document/chart understanding.
+- Audio: Automatic speech recognition (ASR), translation (EN → FR), and summarization (where Conformer path is enabled).
 
-- Base model: Microsoft Phi-4 team  
-- Quantization & runtime: `llama.cpp` contributors  
-- Conversion tweaks & multimodal handling: custom Phi-4 GGUF pipeline  
+## Limitations & Risks
+- Hallucinations: May misinterpret content or hallucinate facts.
+- Verification: Not suitable for medical, legal, or safety-critical decisions without human verification.
+- Compliance: You must comply with the original Microsoft license.
 
-All credit for pretraining and model architecture goes to Microsoft and the Phi-4 team.
+## Acknowledgements
+- Base model: microsoft/Phi-4-multimodal-instruct
+- Serving stack: llama.cpp and its contributors.
+- Special thanks to the Microsoft Phi-4 team for the underlying pretraining.