# codellama2-finetuned-nl2bash-fin

Finetuned on the [AnishJoshi/nl2bash-custom](https://huggingface.co/datasets/AnishJoshi/nl2bash-custom) dataset for generating bash code based on natural language descriptions.


## Model Details

- **Model Name:** CodeLlama2-Finetuned-NL2Bash
- **Base Model:** CodeLlama2
- **Task:** Natural Language to Bash Script Conversion
- **Framework:** PyTorch
- **Fine-tuning Dataset:** Custom dataset of natural language commands and corresponding Bash scripts, available [here](https://huggingface.co/datasets/AnishJoshi/nl2bash-custom)

### Model Description

- **Developed by:** Anish Joshi
- **Model type:** CausalLM
- **Finetuned from model:** Codellama2

## Files Included

- `adapter_config.json`: Configuration file for the adapter layers.
- `adapter_model.safetensors`: Weights of the adapter layers in the Safetensors format.
- `optimizer.pt`: State of the optimizer used during training.
- `rng_state.pth`: State of the random number generator.
- `scheduler.pt`: State of the learning rate scheduler.
- `special_tokens_map.json`: Mapping for special tokens used by the tokenizer.
- `tokenizer.json`: Tokenizer model including the vocabulary.
- `tokenizer_config.json`: Configuration settings for the tokenizer.
- `trainer_state.json`: State of the trainer including training metrics.
- `training_args.bin`: Training arguments used for fine-tuning.
- `README.md

## Usage

# Load model directly
```
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("AnishJoshi/codellama2-finetuned-nl2bash-fin")
model = AutoModelForCausalLM.from_pretrained("AnishJoshi/codellama2-finetuned-nl2bash-fin")
```

## Training Details

Training details available at [Finetuning Notebook](https://github.com/AnishJoshi13/Bash-Scripting-Assistant/blob/master/notebooks/finetuning.ipynb)


#### Training Hyperparameters

Training arguments and configuration are set using TrainingArguments and LoraConfig. The model is fine-tuned using the following parameters:

- `output_dir: codellama2-finetuned-nl2bash` - Directory to save the fine-tuned model.
- `per_device_train_batch_size`: 2 - Batch size per device.
- `gradient_accumulation_steps`: 16 - Number of gradient accumulation steps.
- `optim`: paged_adamw_32bit - Optimizer type.
- `learning_rate`: 2e-4 - Learning rate.
- `lr_scheduler_type`: cosine - Learning rate scheduler type.
- `save_strategy`: epoch - Strategy to save checkpoints.
- `logging_steps`: 10 - Number of steps between logging.
- `num_train_epochs`: 1 - Number of training epochs.
- `max_steps`: 100 - Maximum number of training steps.
- `fp16`: True - Use 16-bit floating-point precision.
- `push_to_hub`: False - Whether to push the model to Hugging Face Hub.
- `report_to`: none - Reporting destination.

## Evaluation

Evaulation metrics and calculations available at [Evaluation Notebook](https://github.com/AnishJoshi13/Bash-Scripting-Assistant/blob/master/notebooks/evaluation.ipynb)