README.md

---
license: mit
base_model: deepseek-ai/deepseek-coder-6.7b-base
tags:
  - code
  - leetcode
  - python
  - fine-tuned
  - lora
  - qlora
datasets:
  - LongQ/leetcode_python
language:
  - en
pipeline_tag: text-generation
---

# DeepSeek-Coder-6.7B LeetCode Fine-Tuned

A fine-tuned version of [DeepSeek-Coder-6.7B-Base](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base) specialized for solving LeetCode-style algorithmic problems in Python.

## Model Details

| Attribute | Value |
|-----------|-------|
| **Base Model** | deepseek-ai/deepseek-coder-6.7b-base |
| **Fine-tuning Method** | QLoRA (4-bit quantization + LoRA) |
| **Training Data** | [LongQ/leetcode_python](https://huggingface.co/datasets/LongQ/leetcode_python) (2,369 problems) |
| **Epochs** | 3 |
| **Hardware** | NVIDIA T4 (16GB VRAM) |
| **Training Time** | ~5 hours |

## Performance

Evaluated on 100 LeetCode problems with automated code execution:

| Metric | Base Model | Fine-Tuned | Improvement |
|--------|------------|------------|-------------|
| **Overall Accuracy** | 24% | 34% | +42% |
| **Easy Problems** | 30.3% | 52% | +72% |
| **Medium Problems** | 32.4% | 27.8% | -14% |
| **Hard Problems** | 9.1% | 28.6% | +214% |

### Key Findings

- **Significant gains on Easy and Hard problems** — model learned both fundamental patterns and complex algorithms
- **Slight regression on Medium** — possible overfitting to extremes of difficulty distribution
- **Domain-specific data matters** — initial training on general coding data degraded performance

## Intended Use

- Solving algorithmic coding challenges
- LeetCode practice and learning
- Code generation for competitive programming
- Educational tool for understanding algorithmic solutions

## Limitations

- Optimized specifically for LeetCode-style problems, may not generalize to other coding tasks
- Python-only (not trained on other languages)
- May produce syntactically correct but logically incorrect solutions
- Struggles with problems requiring complex data structure implementations (LinkedList, Trees)

## How to Use

### With Hugging Face Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Jerry-lin23/deepseek-leetcode-fp16",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Jerry-lin23/deepseek-leetcode-fp16")

prompt = """### Problem:
Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target.

### Starter Code:
```python
class Solution:
    def twoSum(self, nums: List[int], target: int) -> List[int]:
```

### Solution:
```python
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.2)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### With Ollama (Local Deployment)

1. Convert to GGUF format
2. Create a Modelfile:
```
FROM ./deepseek-leetcode-q8.gguf
PARAMETER temperature 0.2
PARAMETER top_p 0.95
```
3. Import: `ollama create deepseek-leetcode -f Modelfile`
4. Run: `ollama run deepseek-leetcode`

## Training Details

### LoRA Configuration

```python
LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
```

### Training Arguments

```python
SFTConfig(
    num_train_epochs=3,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    warmup_ratio=0.03,
    fp16=True,
    gradient_checkpointing=True,
    max_seq_length=2048,
    dataset_text_field="text"
)
```

## Citation

```bibtex
@misc{deepseek-leetcode-finetuned,
  author = {Jerry Lin},
  title = {DeepSeek-Coder-6.7B LeetCode Fine-Tuned},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Jerry-lin23/deepseek-leetcode-fp16}
}
```

## Links

- 📊 [Benchmark Repository](https://github.com/jerrylin-23/DeepSeek-LeetCode-Oriented-Training)
- 🤗 [Base Model](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base)
- 📚 [Training Dataset](https://huggingface.co/datasets/LongQ/leetcode_python)