Update README.md

README.md

@@ -1,3 +1,100 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+tags:
+- code
+---
+# Fine-tuned Qwen2.5-Coder-7B for Function Writing
+
+## Model Description
+
+This model is a fine-tuned version of Qwen2.5-Coder-7B, specifically optimized for function writing tasks. The base model Qwen2.5-Coder-7B is part of the Qwen2.5-Coder family, which was trained on 5.5 trillion tokens including source code, text-code grounding, and synthetic data.
+
+### Base Model Details
+
+* **Type**: Causal Language Model
+* **Architecture**: Transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
+* **Parameters**: 7.61B (6.53B Non-Embedding)
+* **Layers**: 28
+* **Attention Heads**: 28 for Q and 4 for KV
+* **Context Length**: Up to 131,072 tokens
+
+## Fine-tuning Specifications
+
+The model was fine-tuned using LoRA (Low-Rank Adaptation) with the following configuration:
+
+### Training Parameters
+
+* **Training Data**: 30,000 examples
+* **Batch Size**: 1 per device
+* **Gradient Accumulation Steps**: 24
+* **Learning Rate**: 1e-6
+* **Number of Epochs**: 2
+* **Warmup Ratio**: 0.05
+* **Maximum Sequence Length**: 4,096 tokens
+* **Weight Decay**: 0.01
+* **Maximum Gradient Norm**: 0.5
+* **Learning Rate Scheduler**: Cosine
+
+### LoRA Configuration
+
+* **Rank (r)**: 32
+* **Alpha**: 32
+* **Dropout**: 0.05
+* **Target Modules**: q_proj, v_proj, o_proj, gate_proj, up_proj
+* **Training Mode**: BF16 mixed precision
+* **RS-LoRA**: Enabled
+
+### Training Infrastructure
+
+* **Quantization**: 4-bit quantization (NF4)
+* **Attention Implementation**: Flash Attention 2
+* **Memory Optimization**: Gradient checkpointing enabled
+
+## Usage
+
+This model is optimized for function writing tasks and can be loaded using the Hugging Face Transformers library. Here's a basic example:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load the model and tokenizer
+model = AutoModelForCausalLM.from_pretrained(
+    "path_to_your_model",
+    trust_remote_code=True,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained(
+    "path_to_your_model",
+    trust_remote_code=True
+)
+
+# Generate text
+input_text = "Write a function that..."
+inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=500)
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+```
+
+## Limitations
+
+* The model is specifically fine-tuned for function writing tasks and may not perform optimally for general code generation or other tasks
+* Maximum context length during fine-tuning was limited to 4,096 tokens
+* While the base model supports up to 128K tokens, using beyond 4,096 tokens may require additional validation
+
+## License
+
+This model inherits the Apache 2.0 license from its base model Qwen2.5-Coder-7B.
+
+## Citation
+
+If you use this model, please cite both the original Qwen2.5-Coder paper and acknowledge the fine-tuning work:
+
+```bibtex
+@article{hui2024qwen2,
+      title={Qwen2.5-Coder Technical Report},
+      author={Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and others},
+      journal={arXiv preprint arXiv:2409.12186},
+      year={2024}
+}
+```