|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- pytorch |
|
|
- peft |
|
|
- lora |
|
|
- code-generation |
|
|
- deepseek-coder |
|
|
- fine-tuned |
|
|
datasets: |
|
|
- custom-code-dataset |
|
|
model-index: |
|
|
- name: BriskFO_Coderv1 |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
# BriskFO_Coderv1 |
|
|
|
|
|
## Model Description |
|
|
|
|
|
This is a **PEFT/LoRA adapter** fine-tuned on DeepSeek Coder 1.3B Instruct model. It was trained for 300 steps on a custom code generation dataset. |
|
|
|
|
|
## Model Type |
|
|
|
|
|
This is a **PEFT (Parameter-Efficient Fine-Tuning)** model, specifically using **LoRA (Low-Rank Adaptation)**. It contains only the adapter weights, not the full model. |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Base Model**: `deepseek-ai/deepseek-coder-1.3b-instruct` |
|
|
- **Training Steps**: 300 |
|
|
- **Learning Rate**: 2e-4 |
|
|
- **Batch Size**: 16 |
|
|
- **Gradient Accumulation**: 4 |
|
|
- **Sequence Length**: 34958 |
|
|
- **Training Method**: PEFT/LoRA |
|
|
|
|
|
## Files |
|
|
|
|
|
This repository contains: |
|
|
- `adapter_model.bin` / `adapter_model.safetensors` - LoRA adapter weights |
|
|
- `adapter_config.json` - PEFT configuration |
|
|
- `tokenizer.json`, `tokenizer_config.json` - Tokenizer files |
|
|
- `special_tokens_map.json` - Special tokens mapping |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install transformers peft accelerate torch |
|
|
``` |
|
|
|
|
|
### Loading the Model |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
from peft import PeftModel, PeftConfig |
|
|
|
|
|
# Load the base model |
|
|
base_model_id = "deepseek-ai/deepseek-coder-1.3b-instruct" |
|
|
adapter_model_id = "abel252/BriskFO_Coderv1" |
|
|
|
|
|
# Load tokenizer |
|
|
tokenizer = AutoTokenizer.from_pretrained(adapter_model_id) |
|
|
|
|
|
# Load base model |
|
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
|
base_model_id, |
|
|
torch_dtype="auto", |
|
|
device_map="auto" |
|
|
) |
|
|
|
|
|
# Load PEFT adapter |
|
|
model = PeftModel.from_pretrained(base_model, adapter_model_id) |
|
|
|
|
|
# For inference, you can merge the adapter with the base model (optional) |
|
|
# model = model.merge_and_unload() |
|
|
``` |
|
|
|
|
|
### Inference Example |
|
|
|
|
|
```python |
|
|
# Prepare input |
|
|
prompt = "Write a Python function to calculate fibonacci numbers" |
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
|
|
|
|
|
# Generate |
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=256, |
|
|
temperature=0.7, |
|
|
top_p=0.95, |
|
|
do_sample=True |
|
|
) |
|
|
|
|
|
# Decode |
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(response) |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
This model is released under the Apache 2.0 license. |
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
- Base model: [DeepSeek Coder](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct) |
|
|
- Fine-tuning framework: [PEFT](https://github.com/huggingface/peft) |
|
|
|
