File size: 3,799 Bytes
3a08249 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
---
license: llama2
base_model: codellama/CodeLlama-7b-Instruct-hf
tags:
- fine-tuned
- educational
- qa
- code
- llama
- peft
- lora
language:
- en
pipeline_tag: text-generation
library_name: peft
---
# CodeLLaMa7B-FineTuned-byMoomen
This model is a fine-tuned version of [codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf) using LoRA (Low-Rank Adaptation) for educational Q&A tasks.
## Model Details
- **Base Model**: codellama/CodeLlama-7b-Instruct-hf
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)
- **LoRA Rank**: 32
- **LoRA Alpha**: 64
- **Target Modules**: ['gate_proj', 'lm_head', 'k_proj', 'q_proj', 'up_proj', 'down_proj', 'v_proj', 'o_proj']
- **Training Focus**: Educational programming Q&A
- **Model Type**: Causal Language Model
## Usage
### Quick Start
```python
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
# Load model and tokenizer
model = AutoPeftModelForCausalLM.from_pretrained("Moomen123Msaadi/CodeLLaMa7B-FineTuned-byMoomen")
tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-Instruct-hf")
# Generate response
prompt = "Explain recursion in programming"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```
### Chat Format Usage
```python
# For educational Q&A conversations
messages = [
{"role": "system", "content": "You are a helpful educational assistant."},
{"role": "user", "content": "What is the difference between lists and tuples in Python?"}
]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(formatted_prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=300)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
```
### Memory-Efficient Loading
```python
# For systems with limited VRAM
from transformers import BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16
)
model = AutoPeftModelForCausalLM.from_pretrained(
"Moomen123Msaadi/CodeLLaMa7B-FineTuned-byMoomen",
quantization_config=quantization_config,
device_map="auto"
)
```
## Training Details
This model was fine-tuned using:
- **Parameter-Efficient Fine-Tuning (PEFT)** with LoRA
- **Educational conversation dataset** focused on programming concepts
- **Optimized for Q&A format** with system/user/assistant roles
## Intended Use
This model is designed for:
- 📚 Educational programming Q&A
- 💡 Concept explanations in computer science
- 🔧 Code debugging assistance
- 🎓 Technical tutoring and learning support
## Limitations
- Based on codellama/CodeLlama-7b-Instruct-hf, inherits its limitations
- Optimized for educational content, may not perform well on other tasks
- Requires base model for inference (LoRA adapters only)
- Performance depends on the quality of training data
## Model Architecture
This is a LoRA adapter that needs to be loaded with the base model. The adapter files are:
- `adapter_config.json`: LoRA configuration
- `adapter_model.safetensors`: Trained LoRA weights
## License
This model follows the same license as the base model: Llama 2 Custom License.
## Citation
If you use this model, please cite:
```bibtex
@misc{CodeLLaMa7B_FineTuned_byMoomen,
title={CodeLLaMa7B-FineTuned-byMoomen},
author={Moomen123Msaadi},
year={2024},
publisher={Hugging Face},
url={https://huggingface.co/Moomen123Msaadi/CodeLLaMa7B-FineTuned-byMoomen}
}
```
|