Update README.md

README.md

@@ -1,10 +1,228 @@
 ---
 license: mit
 language:
-- en
+  - en
 tags:
-- math
-- llm
-- 4bit
-- quantize
----
+  - math
+  - llm
+  - 4bit
+  - quantize
+  - gptq
+  - qwen
+  - instruction
+---
+
+# Qwen2.5-Math-7B-Instruct-4bit
+
+## Model Description
+
+**Qwen2.5-Math-7B-Instruct-4bit** is a 4-bit quantized version of the [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct) model using GPTQ quantization (W4A16 - 4-bit weights, 16-bit activations).
+
+This model is optimized to:
+
+- Reduce model size by ~75% compared to the original model
+- Reduce GPU memory requirements during inference
+- Increase inference speed
+- Maintain high accuracy for mathematical tasks
+
+### Model Details
+
+- **Developed by:** Community
+- **Model type:** Causal Language Model (Quantized)
+- **Language(s):** English, Mathematics
+- **License:** MIT
+- **Finetuned from model:** [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct)
+- **Quantization method:** GPTQ (W4A16) via LLM Compressor
+- **Calibration dataset:** GSM8K (256 samples)
+
+### Model Sources
+
+- **Base Model:** [Qwen/Qwen2.5-Math-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct)
+- **Quantization Tool:** [vLLM LLM Compressor](https://docs.vllm.ai/projects/llm-compressor/)
+
+## Uses
+
+### Direct Use
+
+This model is designed for direct use in mathematical and reasoning tasks, including:
+
+- Solving arithmetic, algebra, and geometry problems
+- Mathematical reasoning and proofs
+- Analyzing and explaining mathematical concepts
+- Educational mathematics support
+
+### Example Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_path = "your-username/qwen2.5-math-7b-instruct-4bit"
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    device_map="auto",
+    dtype="float16",
+    trust_remote_code=True,
+    low_cpu_mem_usage=False,  # Important for compressed models
+)
+
+# Create prompt
+prompt = "<|im_start|>user\nSolve for x: 3x + 5 = 14<|im_end|>\n<|im_start|>assistant\n"
+
+# Generate
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(response)
+```
+
+### Downstream Use
+
+This model can be further fine-tuned for specific mathematical tasks or integrated into educational applications.
+
+### Out-of-Scope Use
+
+This model is NOT designed for:
+
+- Generating harmful or inappropriate content
+- Use in applications requiring absolute accuracy (such as critical financial calculations)
+- Tasks unrelated to mathematics or reasoning
+
+## Bias, Risks, and Limitations
+
+### Limitations
+
+- The model has been quantized and may have slightly lower accuracy compared to the original model
+- May encounter errors with some complex problems or edge cases
+- Model was primarily trained on English data
+
+### Recommendations
+
+Users should:
+
+- Verify results for important mathematical problems
+- Use the original model (full precision) if maximum accuracy is required
+- Understand that quantization may affect some tasks
+
+## How to Get Started with the Model
+
+### Installation
+
+```bash
+pip install transformers torch accelerate
+```
+
+### Quick Start
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "your-username/qwen2.5-math-7b-instruct-4bit"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    device_map="auto",
+    dtype="float16",
+    trust_remote_code=True,
+    low_cpu_mem_usage=False,
+)
+
+# Use the model
+prompt = "<|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=200)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Training Details
+
+### Quantization Procedure
+
+The model was quantized using:
+
+- **Method:** GPTQ (W4A16)
+- **Tool:** vLLM LLM Compressor
+- **Calibration dataset:** GSM8K (256 samples)
+- **Max sequence length:** 2048 tokens
+- **Target layers:** All Linear layers except `lm_head`
+
+### Quantization Hyperparameters
+
+- **Scheme:** W4A16 (4-bit weights, 16-bit activations)
+- **Block size:** 128
+- **Dampening fraction:** 0.01
+- **Calibration samples:** 256
+
+## Evaluation
+
+### Testing Data
+
+The model was evaluated on the GSM8K test set.
+
+### Metrics
+
+- **Accuracy:** Measured on GSM8K test set
+- **Model size:** ~3.5GB (compared to ~14GB of the original model)
+- **Compression ratio:** ~75% reduction
+- **Memory usage:** Significantly reduced compared to the original model
+
+### Results
+
+The compressed model maintains high accuracy for mathematical tasks while significantly reducing size and memory requirements.
+
+## Technical Specifications
+
+### Model Architecture
+
+- **Base Architecture:** Qwen2.5 (Transformer-based)
+- **Parameters:** 7B (quantized to 4-bit)
+- **Context Length:** 8192 tokens (original model), 2048 tokens (optimized for quantization)
+- **Quantization:** GPTQ W4A16
+
+### Compute Infrastructure
+
+#### Hardware
+
+- **Training/Quantization:** NVIDIA RTX 3060 12GB (or equivalent)
+- **Minimum Inference:** GPU with at least 8GB VRAM
+
+#### Software
+
+- **Quantization Tool:** vLLM LLM Compressor
+- **Framework:** PyTorch, Transformers
+- **Python:** >=3.12
+
+## Citation
+
+If you use this model, please cite:
+
+**Base Model:**
+
+```bibtex
+@article{qwen2.5,
+  title={Qwen2.5: A Large Language Model for Mathematics},
+  author={Qwen Team},
+  year={2024}
+}
+```
+
+**Quantization Method:**
+
+```bibtex
+@article{gptq,
+  title={GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},
+  author={Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},
+  journal={arXiv preprint arXiv:2210.17323},
+  year={2022}
+}
+```
+
+## Model Card Contact
+
+To report issues or ask questions, please open an issue on the repository.
+
+## Acknowledgments
+
+- Qwen Team for the original Qwen2.5-Math-7B-Instruct model
+- vLLM team for the LLM Compressor tool
+- Hugging Face for infrastructure and support