Create README.md

README.md

@@ -0,0 +1,154 @@
+---
+license: llama2
+datasets:
+- AlfredPros/smart-contracts-instructions
+language:
+- en
+tags:
+- blockchain
+- solidity
+- smart contract
+---
+# Code LLaMA 7b Instruct Solidity
+
+A finetuned 7 billion parameters Code LLaMA - Instruct model to generate Solidity smart contract using 4-bit QLoRA finetuning provided by PEFT library.
+
+# Training Dataset
+
+Dataset used to finetune the model is AlfredPros' Smart Contracts Instructions (https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions). 
+A dataset containing 6,003 GPT-generated human instruction and Solidity source code data pairs. This dataset has been processed for training LLMs.
+
+# Training Parameters
+
+## Bitsandbytes quantization configurations
+- Load in 4-bit: true
+- 4-bit quantization type: NF4
+- 4-bit compute dtype: float16
+- 4-bit use double quantization: true
+
+## Supervised finetuning trainer parameters
+- Number of train epochs: 1
+- FP16: true
+- FP16 option level: O1
+- BF16: false
+- Per device train batch size: 1
+- Gradient accumulation steps: 1
+- Gradient checkpointing: true
+- Max gradient normal: 0.3
+- Learning rate: 2e-4
+- Weight decay: 0.001
+- Optimizer: paged AdamW 32-bit
+- Learning rate scheduler type: cosine
+- Warmup ratio: 0.03
+
+# Training Loss
+```
+Step	Training Loss
+ 100	0.330900
+ 200	0.293000
+ 300	0.276500
+ 400	0.290900
+ 500	0.306100
+ 600	0.302600
+ 700	0.337200
+ 800	0.295000
+ 900	0.297800
+1000	0.299500
+1100	0.268900
+1200	0.257800
+1300	0.264100
+1400	0.294400
+1500	0.293900
+1600	0.287600
+1700	0.281200
+1800	0.273400
+1900	0.266600
+2000	0.227500
+2100	0.261600
+2200	0.275700
+2300	0.290100
+2400	0.290900
+2500	0.316200
+2600	0.296500
+2700	0.291400
+2800	0.253300
+2900	0.321500
+3000	0.269500
+3100	0.295600
+3200	0.265800
+3300	0.262800
+3400	0.274900
+3500	0.259800
+3600	0.226300
+3700	0.325700
+3800	0.249000
+3900	0.237200
+4000	0.251400
+4100	0.247000
+4200	0.278700
+4300	0.264000
+4400	0.245000
+4500	0.235900
+4600	0.240400
+4700	0.235200
+4800	0.220300
+4900	0.202700
+5000	0.240500
+5100	0.258500
+5200	0.236300
+5300	0.267500
+5400	0.236700
+5500	0.265900
+5600	0.244900
+5700	0.297900
+5800	0.281200
+5900	0.313800
+6000	0.249800
+6003	0.271939
+```
+
+# Example Usage
+```py
+from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
+import torch
+import accelerate
+
+use_4bit = True
+bnb_4bit_compute_dtype = "float16"
+bnb_4bit_quant_type = "nf4"
+use_double_nested_quant = True
+compute_dtype = getattr(torch, bnb_4bit_compute_dtype)
+
+# BitsAndBytesConfig int-4 config
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=use_4bit,
+    bnb_4bit_use_double_quant=use_double_nested_quant,
+    bnb_4bit_quant_type=bnb_4bit_quant_type,
+    bnb_4bit_compute_dtype=compute_dtype,
+    load_in_8bit_fp32_cpu_offload=True
+)
+
+# Load model in 4-bit
+tokenizer = AutoTokenizer.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity")
+model = AutoModelForCausalLM.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity", quantization_config=bnb_config, device_map="balanced_low_0")
+
+# Make input
+input='Make a smart contract to create a whitelist of approved wallets. The purpose of this contract is to allow the DAO (Decentralized Autonomous Organization) to approve or revoke certain wallets, and also set a checker address for additional validation if needed. The current owner address can be changed by the current owner.'
+
+prompt = f"""### Instruction:
+Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:
+
+### Task:
+{input}
+
+### Solution:
+"""
+
+# Tokenize the input
+input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
+# Run the model to infere an output
+outputs = model.generate(input_ids=input_ids, max_new_tokens=256, do_sample=True, top_p=0.9, temperature=0.001, pad_token_id=1)
+
+# Display the generated output
+print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):])
+```