---
license: mit
datasets:
- sahil2801/CodeAlpaca-20k
- TokenBender/code_instructions_122k_alpaca_style
base_model:
- mistralai/Mistral-7B-Instruct-v0.3
tags:
- code
- python
- sql
- data-science
---

# Code Specialist 7B  

<p align="left">
  <a href="https://huggingface.co/Ricardouchub/code-specialist-7b">
    <img src="https://img.shields.io/badge/HuggingFace-Code_Specialist_7B-FFD21E?style=flat-square&logo=huggingface&logoColor=black" alt="Hugging Face"/>
  </a>
  <a href="https://www.python.org/">
    <img src="https://img.shields.io/badge/Python-3.10+-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python"/>
  </a>
  <a href="https://huggingface.co/docs/transformers">
    <img src="https://img.shields.io/badge/Transformers-4.56+-purple?style=flat-square&logo=huggingface&logoColor=white" alt="Transformers"/>
  </a>
  <a href="https://github.com/Ricardouchub">
    <img src="https://img.shields.io/badge/Author-Ricardo_Urdaneta-000000?style=flat-square&logo=github&logoColor=white" alt="Author"/>
  </a>
</p>

---

## Description  

**Code Specialist 7B** is a fine-tuned version of **Mistral-7B-Instruct-v0.3**, trained through **Supervised Fine-Tuning (SFT)** using datasets focused on **Python and SQL**.  
The goal of this training was to enhance the model’s performance in **data analysis, programming problem-solving, and technical reasoning**.

The model preserves the **7B parameter Transformer decoder-only** architecture while introducing a code-oriented fine-tuning, resulting in improved robustness for function generation, SQL queries, and technical answers.

---

## Base Model  

- [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)  
- Architecture: Transformer (decoder-only)  
- Parameters: ~7B  

---

## Datasets Used for SFT  

- [CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k)  
- [Code Instructions 122k (Alpaca-style)](https://huggingface.co/datasets/TokenBender/code_instructions_122k_alpaca_style)  

Both datasets were **filtered to include only Python and SQL examples**, following **Alpaca/Mistral-style** instruction formatting.

Example prompt format:  

```
[INST] Write a Python function that adds two numbers. [/INST]  
def add(a, b):  
    return a + b  
```

---

## Training Details  

| **Aspect**        | **Detail** |
|--------------------|-------------|
| **Method**         | QLoRA with final weight merge |
| **Frameworks**     | `transformers`, `trl`, `peft`, `bitsandbytes` |
| **Hardware**       | GPU with 12 GB VRAM (4-bit quantization for training) |

### Main Hyperparameters

| **Parameter** | **Value** |
|----------------|-----------|
| `per_device_train_batch_size` | 2 |
| `gradient_accumulation_steps` | 4 |
| `learning_rate` | 2e-4 |
| `num_train_epochs` | 1 |
| `max_seq_length` | 1024 |

---

## Usage  

```python
from transformers import AutoTokenizer, AutoModelForCausalLM  

model_id = "Ricardouchub/Code-Specialist-7B"  
tok = AutoTokenizer.from_pretrained(model_id)  
mdl = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")  

prompt = "[INST] Write a Python function that calculates the average of a list. [/INST]"  
inputs = tok(prompt, return_tensors="pt").to(mdl.device)  

out = mdl.generate(**inputs, max_new_tokens=256)  
print(tok.decode(out[0], skip_special_tokens=True))  
```

---

## Initial Benchmarks  

- **Simple evaluation (Python tasks):** Improved results on small programming and data-related tasks, including **data analysis, SQL query generation, and Python snippets**, compared to the base model.  
- Further evaluation on **HumanEval** or **MBPP** is recommended for reproducible metrics.

---

## Author  

**Ricardo Urdaneta**  
- [LinkedIn](https://www.linkedin.com/in/ricardourdanetacastro/)  
- [GitHub](https://github.com/Ricardouchub)  

---

## Limitations  

- The model does **not guarantee 100% accuracy** on complex programming tasks.  
- It may produce inconsistent results for ambiguous or incomplete prompts.  

---

## License  

This model is released under the same license as **Mistral-7B-Instruct-v0.3** — **MIT License**.