Mukesh0606/solidity-codellama-lora-r64-final

CodeLlama-7B Solidity — LoRA Adapter (final, 5 epochs)

This repository contains a LoRA / PEFT adapter (not a full model) fine-tuned on top of AlfredPros/CodeLlama-7b-Instruct-Solidity.

This is the final checkpoint of the run, saved at global step 1,485,045 — the full 5 epochs (max_steps reached, should_training_stop = true). Training is complete; this is the adapter to use for inference. Optimizer / scheduler / RNG state are also included if you wish to continue training beyond 5 epochs.

Model Details

Property	Value
Base model	AlfredPros/CodeLlama-7b-Instruct-Solidity (CodeLlama-7B, Solidity-tuned)
Adapter type	LoRA
PEFT version	0.14.0
Task type	CAUSAL_LM
Rank (r)	64
lora_alpha	16
lora_dropout	0.1
Target modules	q_proj, v_proj
Bias	none
Tokenizer	CodeLlamaTokenizerFast
Adapter size	~134 MB (adapter_model.safetensors)

Final training metrics

Metric	Value
Global step	1,485,045 / 1,485,045 (complete)
Epochs	5.0 / 5
Final train loss	~0.26
Final mean token accuracy	~0.93
Train batch size	1
Save / log steps	5,000
Eval steps	500

Files in this repository

File	Purpose
adapter_config.json	LoRA/PEFT configuration
adapter_model.safetensors	LoRA adapter weights (~134 MB)
tokenizer.json, tokenizer_config.json, special_tokens_map.json	Tokenizer
optimizer.pt	Optimizer state (only needed to continue training)
scheduler.pt	LR scheduler state (only needed to continue training)
scaler.pt	Mixed-precision grad scaler state
rng_state.pth	RNG state for reproducible resume
trainer_state.json	Full trainer progress / loss history
training_args.bin	Serialized TrainingArguments from the run

Note: The ~13 GB base model weights are not included here — they are pulled separately from AlfredPros/CodeLlama-7b-Instruct-Solidity. The training dataset and training script are also not included.

How to use (inference)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "AlfredPros/CodeLlama-7b-Instruct-Solidity"
ADAPTER = "Mukesh0606/solidity-codellama-lora-r64-final"   # or a local path to this folder

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

prompt = "// Write a secure ERC20 token contract in Solidity\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Merge the adapter into a standalone model (optional)

merged = model.merge_and_unload()   # folds LoRA weights into the base
merged.save_pretrained("codellama-7b-solidity-merged")
tokenizer.save_pretrained("codellama-7b-solidity-merged")

Continue training beyond 5 epochs (optional)

This run finished at its planned max_steps. To train further you would raise num_train_epochs / max_steps in your TrainingArguments and resume:

from transformers import Trainer

# Re-create the SAME base model, LoRA config, tokenizer, dataset, and updated TrainingArguments.
trainer = Trainer(model=peft_model, args=training_args, train_dataset=..., ...)
trainer.train(resume_from_checkpoint="path/to/checkpoint-1485045")

Requirements:

• Same base model (AlfredPros/CodeLlama-7b-Instruct-Solidity).
• Same LoRA configuration (already in adapter_config.json).
• The original training dataset and preprocessing (not stored here).
• A GPU with enough memory for 7B + LoRA training (original run used train_batch_size=1).

If you load the adapter manually via PeftModel.from_pretrained for further training rather than via resume_from_checkpoint, pass is_trainable=True. adapter_config.json has inference_mode: true, which is correct for inference; the Trainer resume path switches back to training mode automatically.

Hardware notes

• Inference: 7B runs on ~16 GB GPU in fp16, or ~6 GB with 4-bit quantization (bitsandbytes).
• Training/resume: needs a capable GPU; original run used batch size 1.

Framework versions

• PEFT 0.14.0
• Transformers (Hugging Face Trainer)
• Base: CodeLlama-7B-Instruct (Solidity-tuned)

License

Inherits the base model's license (Llama 2 Community License via CodeLlama). Review the base model card before commercial use.