---
base_model: AlfredPros/CodeLlama-7b-Instruct-Solidity
library_name: peft
pipeline_tag: text-generation
tags:
  - peft
  - lora
  - code-llama
  - solidity
  - smart-contracts
  - causal-lm
license: llama2
language:
  - en
---

# CodeLlama-7B Solidity — LoRA Adapter (final, 5 epochs)

This repository contains a **LoRA / PEFT adapter** (not a full model) fine-tuned on top of
[`AlfredPros/CodeLlama-7b-Instruct-Solidity`](https://huggingface.co/AlfredPros/CodeLlama-7b-Instruct-Solidity).

This is the **final checkpoint** of the run, saved at global step **1,485,045** — the full
**5 epochs** (`max_steps` reached, `should_training_stop = true`). Training is complete; this is
the adapter to use for inference. Optimizer / scheduler / RNG state are also included if you wish
to continue training beyond 5 epochs.

## Model Details

| Property | Value |
|---|---|
| Base model | `AlfredPros/CodeLlama-7b-Instruct-Solidity` (CodeLlama-7B, Solidity-tuned) |
| Adapter type | LoRA |
| PEFT version | 0.14.0 |
| Task type | `CAUSAL_LM` |
| Rank (`r`) | 64 |
| `lora_alpha` | 16 |
| `lora_dropout` | 0.1 |
| Target modules | `q_proj`, `v_proj` |
| Bias | none |
| Tokenizer | `CodeLlamaTokenizerFast` |
| Adapter size | ~134 MB (`adapter_model.safetensors`) |

### Final training metrics

| Metric | Value |
|---|---|
| Global step | 1,485,045 / 1,485,045 (complete) |
| Epochs | 5.0 / 5 |
| Final train loss | ~0.26 |
| Final mean token accuracy | ~0.93 |
| Train batch size | 1 |
| Save / log steps | 5,000 |
| Eval steps | 500 |

## Files in this repository

| File | Purpose |
|---|---|
| `adapter_config.json` | LoRA/PEFT configuration |
| `adapter_model.safetensors` | LoRA adapter weights (~134 MB) |
| `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` | Tokenizer |
| `optimizer.pt` | Optimizer state (only needed to continue training) |
| `scheduler.pt` | LR scheduler state (only needed to continue training) |
| `scaler.pt` | Mixed-precision grad scaler state |
| `rng_state.pth` | RNG state for reproducible resume |
| `trainer_state.json` | Full trainer progress / loss history |
| `training_args.bin` | Serialized `TrainingArguments` from the run |

> **Note:** The ~13 GB base model weights are **not** included here — they are pulled separately
> from `AlfredPros/CodeLlama-7b-Instruct-Solidity`. The training **dataset and training script are
> also not included**.

## How to use (inference)

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "AlfredPros/CodeLlama-7b-Instruct-Solidity"
ADAPTER = "Mukesh0606/solidity-codellama-lora-r64-final"   # or a local path to this folder

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

prompt = "// Write a secure ERC20 token contract in Solidity\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

### Merge the adapter into a standalone model (optional)

```python
merged = model.merge_and_unload()   # folds LoRA weights into the base
merged.save_pretrained("codellama-7b-solidity-merged")
tokenizer.save_pretrained("codellama-7b-solidity-merged")
```

## Continue training beyond 5 epochs (optional)

This run finished at its planned `max_steps`. To train further you would raise
`num_train_epochs` / `max_steps` in your `TrainingArguments` and resume:

```python
from transformers import Trainer

# Re-create the SAME base model, LoRA config, tokenizer, dataset, and updated TrainingArguments.
trainer = Trainer(model=peft_model, args=training_args, train_dataset=..., ...)
trainer.train(resume_from_checkpoint="path/to/checkpoint-1485045")
```

Requirements:
- Same base model (`AlfredPros/CodeLlama-7b-Instruct-Solidity`).
- Same LoRA configuration (already in `adapter_config.json`).
- The **original training dataset and preprocessing** (not stored here).
- A GPU with enough memory for 7B + LoRA training (original run used `train_batch_size=1`).

> If you load the adapter manually via `PeftModel.from_pretrained` for further training rather than
> via `resume_from_checkpoint`, pass `is_trainable=True`. `adapter_config.json` has
> `inference_mode: true`, which is correct for inference; the `Trainer` resume path switches back to
> training mode automatically.

## Hardware notes

- **Inference:** 7B runs on ~16 GB GPU in fp16, or ~6 GB with 4-bit quantization (bitsandbytes).
- **Training/resume:** needs a capable GPU; original run used batch size 1.

## Framework versions

- PEFT 0.14.0
- Transformers (Hugging Face `Trainer`)
- Base: CodeLlama-7B-Instruct (Solidity-tuned)

## License

Inherits the base model's license (Llama 2 Community License via CodeLlama). Review the base
model card before commercial use.