pranav-pvnn/codellama-7b-python-ai-assistant-full-gguf

CodeLlama 7B Python AI Assistant (Merged GGUF)

This is a merged version of the QLoRA fine-tuned CodeLlama-7B model. The LoRA weights have been merged with the base model and converted to GGUF format for easy deployment.

Model Details

• Base Model: CodeLlama-7b-hf
• Original LoRA Adapter: pranav-pvnn/codellama-7b-python-ai-assistant
• Fine-tuning Method: QLoRA (4-bit quantization with LoRA)
• Format: GGUF (self-contained, no separate adapter needed)
• Training Framework: Unsloth

Available Quantizations

• codellama-7b-merged-f16.gguf - Full precision (FP16) - ~13 GB
• codellama-7b-merged-Q4_K_M.gguf - 4-bit quantization (recommended) - ~4 GB
• codellama-7b-merged-Q5_K_M.gguf - 5-bit quantization (higher quality) - ~5 GB
• codellama-7b-merged-Q8_0.gguf - 8-bit quantization (highest quality) - ~7 GB

Usage

With llama.cpp:

./llama-cli -m codellama-7b-merged-Q4_K_M.gguf -p "### Instruction:\nWrite a Python function to calculate factorial.\n### Response:\n"

With Python (llama-cpp-python):

from llama_cpp import Llama

llm = Llama(model_path="codellama-7b-merged-Q4_K_M.gguf")
prompt = "### Instruction:\nWrite a Python function to calculate factorial.\n### Response:\n"
output = llm(prompt, max_tokens=256)
print(output['choices'][0]['text'])

With Ollama:

1. Create a Modelfile:

FROM ./codellama-7b-merged-Q4_K_M.gguf

2. Create the model:

ollama create my-codellama -f Modelfile
ollama run my-codellama "Write a Python function to sort a list"

Training Details

• Quantization: 4-bit QLoRA
• LoRA Rank: 64
• Learning Rate: 2e-4
• Epochs: 4
• Max Seq Length: 2048
• Training Data: Custom Python programming examples (~2,000 examples)
• GPU: NVIDIA Tesla T4

Prompt Format

### Instruction:
[Your instruction here]
### Response:

License

Same as base model (Llama 2 license)

Acknowledgements

• Base Model: Meta's CodeLlama
• Training Framework: Unsloth