README.md

---
language: en
license: apache-2.0
tags:
- llama2
- lora
- code
- adapter
datasets:
- iamtarun/python_code_instructions_18k_alpaca
---

# LLaMA-2-7B CODE LoRA Adapter

This is a LoRA adapter for LLaMA-2-7B fine-tuned on code domain data.

## Model Details

- **Base Model**: meta-llama/Llama-2-7b-hf
- **Adapter Type**: LoRA (Low-Rank Adaptation)
- **Domain**: Code
- **Training Data**: Python code instructions
- **Training Examples**: 2000 (1600 train, 200 val, 200 test)
- **Epochs**: 2

## LoRA Configuration

- Rank (r): 16
- Alpha: 32
- Dropout: 0.05
- Target Modules: q_proj, v_proj, k_proj, o_proj

## Performance Metrics (100 test examples)

- Loss: 0.573
- Perplexity: 1.773
- BLEU: 32.76

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_8bit=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load adapter
model = PeftModel.from_pretrained(model, "Thamirawaran/llama2-7b-code-lora")

# Generate
prompt = 'Write a Python function to sort a list'
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Details

- Trained with FYP_MDLE library
- 8-bit quantization during training
- Gradient accumulation: 16 steps
- Learning rate: 2e-4
- Warmup steps: 20

## Citation

```bibtex
@misc{llama2-code-lora,
  author = {Team RAISE},
  title = {LLaMA-2-7B Code LoRA Adapter},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Thamirawaran/llama2-7b-code-lora}}
}
```

## License

Apache 2.0