README.md

---

base_model: unsloth/llama-3-8b-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:unsloth/llama-3-8b-Instruct
- grpo
- lora
- transformers
- trl
- unsloth
license: apache-2.0
language:
- en
---


# CLI Agent — Llama 3 8B GRPO Fine-tune

A LoRA adapter fine-tuned on Meta-Llama-3-8B-Instruct using GRPO (Group Relative Policy Optimization) to generate correct Linux shell commands from natural language task descriptions.

## Model Details

### Model Description

- **Developed by:** Jose Alvarez, Carson Chiem, Prisha Bhattacharyya, Vishal Tyagi
- **Model type:** Causal Language Model (LoRA adapter)
- **Language(s) (NLP):** English
- **License:** Meta Llama 3 Community License
- **Finetuned from model:** unsloth/llama-3-8b-Instruct

### Model Sources

- **Repository:** https://github.com/Alvarez-Jose/unsloth-grpo-project

## Uses

### Direct Use

Given a natural language description of a CLI task, the model outputs the correct shell command with no explanation, no markdown, and no backticks.

Example:
- Input: "Count the number of lines in /tmp/data/log.txt"
- Output: `wc -l /tmp/data/log.txt`

### Out-of-Scope Use

- Not intended for general conversation
- Not suitable for tasks outside Linux CLI command generation
- Should not be used for destructive or malicious shell commands

## Bias, Risks, and Limitations

- Model may generate incorrect or harmful shell commands — always review before executing
- Trained on a limited set of ~60 task types, may not generalize to all CLI scenarios
- Performance degrades on complex multi-step tasks

## How to Get Started with the Model
```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="jalva182/cli-agent-model",
    max_seq_length=512,
    load_in_4bit=True,
)

messages = [
    {"role": "system", "content": "You are a CLI expert. Given a task, output exactly the shell commands required. No explanation, no markdown, no backticks."},
    {"role": "user", "content": "Count the number of lines in /tmp/data/log.txt"},
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
outputs = model.generate(input_ids=inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Details

### Training Data

60 validated CLI tasks covering file operations, text processing (grep, awk, sed), sorting, archives, system info, permissions, and environment variables. Each task includes setup commands, expected output, and a reward function for GRPO training.

### Training Hyperparameters

- **Training regime:** bf16 mixed precision
- **Method:** GRPO (Group Relative Policy Optimization)
- **Learning rate:** 3e-6 with linear scheduler
- **Warmup ratio:** 0.1
- **Batch size:** 2 (per device)
- **Gradient accumulation steps:** 2
- **Total steps:** 10000
- **LoRA rank:** 32, alpha: 64
- **KL coefficient:** 0.05
- **Number of generations:** 4
- **Max sequence length:** 512

### Speeds, Sizes, Times

- **Training time:** ~3h 13min
- **Checkpoint size:** ~524MB (LoRA adapter only)
- **Final train loss:** 0.0141
- **Final reward:** 8.0/8.0 on easy tasks, ~6.0 average

## Evaluation

### Metrics

Reward function scoring 0-8 per task:
- +5 for correct output match
- +3 for command success with partial match
- -2 for command failure or wrong output

### Results

- **Best reward:** 8.0
- **Average reward (final steps):** ~6.0
- **Train loss:** 0.0141

## Environmental Impact

- **Hardware Type:** H100 SXM 80GB
- **Hours used:** ~3.5 hours
- **Cloud Provider:** Vast.ai

## Technical Specifications

### Model Architecture

- Base: Meta-Llama-3-8B-Instruct
- Adapter: LoRA (rank=32, alpha=64, dropout=0.05)
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

### Software

- unsloth 2026.3.3
- trl 0.24.0
- transformers 4.56.1
- torch 2.6.0+cu124
- PEFT 0.18.1

## Model Card Authors

Jose Alvarez

## Model Card Contact

https://github.com/Alvarez-Jose/unsloth-grpo-project

### Framework versions

- PEFT 0.18.1