README.md

---
title: Tech Advisor
emoji: 🔧
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.18.0
python_version: '3.13'
app_file: app.py
pinned: false
license: apache-2.0
short_description: AWS DevOps Agent expert — fine-tuned Nemotron 4B
tags:
  - build-small-hackathon
  - nemotron
  - aws
  - fine-tuning
  - off-the-grid
  - track:backyard
  - sponsor:nvidia
  - achievement:offgrid
  - achievement:welltuned
---

# 🔧 Tech Advisor

An AI expert on **AWS DevOps Agent** — fine-tuned on the latest documentation so it knows everything about the service: features, pricing, integrations, getting started, and best practices. Powered by **NVIDIA Llama-3.1-Nemotron-Nano-4B** (~5B params, well under the 32B cap).

## What it does

1. **Answers AWS DevOps Agent questions** — features, pricing, integrations, architecture, getting started
2. **Provides accurate details** — trained on 66 pages of official documentation
3. **Runs fully off the grid** — no cloud APIs, no external calls, model runs locally on GPU

## Tech Stack

- **Model**: [NVIDIA Llama-3.1-Nemotron-Nano-4B-v1.1](https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1) — fine-tuned with QLoRA on AWS DevOps Agent docs
- **Frontend**: Gradio chat interface
- **Hardware**: Zero GPU (HF Spaces)
- **Training**: QLoRA (4-bit NF4, LoRA r=16) on AWS EC2 g5.xlarge (A10G)
- **Training Data**: 66 pages of AWS DevOps Agent User Guide → 1,230 instruction-response pairs
- **Badge**: Off the Grid — runs fully locally, no cloud APIs at inference 🔌

## How to use

1. Ask about AWS DevOps Agent (e.g., "What is an Agent Space?", "How much does it cost?")
2. Get detailed, accurate answers based on the latest documentation

## Training Pipeline

The model is fine-tuned on AWS documentation using QLoRA, then pushed to HF Hub.

```
training/
├── prepare_data.py      # Convert raw docs → training JSONL
├── train.py             # QLoRA fine-tuning script
├── push_to_hub.py       # Merge adapter + push to HF Hub
├── requirements.txt     # Training dependencies
└── data/
    ├── raw/             # AWS documentation markdown files
    └── train.jsonl      # Generated training pairs
```

**Steps:**
1. Add AWS docs to `training/data/raw/` (one markdown file per topic) ✅ Done
2. `python training/prepare_data.py` — generate training data
3. `python training/train.py` — fine-tune on AWS EC2 with GPU
4. `python training/push_to_hub.py` — push merged model to HF Hub
5. Update `MODEL_ID` in `app.py` and deploy

See [`training/README.md`](training/README.md) for full details.

## Setup (Deployment)

No API keys or secrets needed — the model runs fully locally on Zero GPU hardware.
Select "ZeroGPU" as the hardware in your Space settings.

## Git Remotes

This repo uses two remotes:

- `origin` → GitHub: `https://github.com/hugotp-ui/20260607smallmodelshackathon.git`
- `space` → HF Space: `https://huggingface.co/spaces/build-small-hackathon/tech-advisor`

```bash
git push origin main   # GitHub
git push space main    # Deploy to Hugging Face Space
```

## Current Progress

- [x] Project scaffolding (Gradio app, training pipeline, README)
- [x] Raw training data collected (66 AWS DevOps Agent docs — full User Guide)
- [x] Data preparation script generates 1,230 instruction-response pairs
- [x] QLoRA training script ready (4-bit NF4, LoRA r=16, 3 epochs)
- [x] HF Space created under `build-small-hackathon/tech-advisor`
- [x] GPU instance launched (g5.xlarge, A10G 24GB, us-west-2)
- [x] Fine-tuning complete (35 min, final loss 0.43)
- [x] Merged model pushed to HF Hub (`aslanconfig/tech-advisor-nemotron-4b`)
- [x] `MODEL_ID` updated in app.py to fine-tuned model
- [x] Deployed to HF Space
- [ ] Record demo video
- [ ] Social media post

## Training Cost

| Item | Duration | Cost |
|------|----------|------|
| EC2 g5.xlarge (A10G 24GB) | ~2 hours total (includes setup + training) | ~$2.02 |
| Training time only | 35 minutes | ~$0.59 |
| EBS storage (100GB gp3) | 2 hours | ~$0.01 |
| **Total** | | **~$2.03** |

g5.xlarge on-demand rate: $1.006/hr in us-west-2. The actual fine-tuning of a 4B model with QLoRA costs under $1 in compute.

## Training Infrastructure

- **Instance**: g5.xlarge (NVIDIA A10G, 24GB VRAM) in us-west-2
- **Instance ID**: `i-0249ed98db8a6480b`
- **AMI**: Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.7 (Ubuntu 22.04) — `ami-0ca70308d230e8a6e`
- **SSH**: `ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@54.69.43.246`
- **Runtime**: PyTorch 2.7 + CUDA 12.8 (pre-installed via `/opt/pytorch`)

## Reproducible Setup Commands

```bash
# 1. Create EC2 key pair
aws ec2 create-key-pair --key-name hackathon-gpu --query 'KeyMaterial' --output text > ~/.ssh/hackathon-gpu.pem
chmod 400 ~/.ssh/hackathon-gpu.pem

# 2. Create security group with SSH access
aws ec2 create-security-group --group-name hackathon-gpu-sg --description "SSH access for hackathon GPU training"
aws ec2 authorize-security-group-ingress --group-id <sg-id> --protocol tcp --port 22 --cidr 0.0.0.0/0

# 3. Launch g5.xlarge instance
aws ec2 run-instances \
  --image-id ami-0ca70308d230e8a6e \
  --instance-type g5.xlarge \
  --key-name hackathon-gpu \
  --security-group-ids <sg-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":100,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=hackathon-training}]'

# 4. Copy project files to instance
scp -i ~/.ssh/hackathon-gpu.pem -r ./ ubuntu@<instance-ip>:~/hackathon/

# 5. SSH in
ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@<instance-ip>

# 6. Build Docker training image
cd ~/hackathon
docker build -t hackathon-train -f Dockerfile .

# 7. Run training in Docker
docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \
  bash -c "cd /workspace && python training/train.py"

# 8. Push merged model to HF Hub
docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \
  bash -c "cd /workspace && huggingface-cli login --token <HF_TOKEN> && python training/push_to_hub.py"
```

### Dockerfile

```dockerfile
FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel

WORKDIR /workspace

RUN pip install --no-cache-dir packaging ninja
ENV TORCH_CUDA_ARCH_LIST="8.6"
RUN pip install --no-cache-dir --no-build-isolation causal-conv1d
RUN pip install --no-cache-dir --no-build-isolation mamba-ssm
RUN pip install --no-cache-dir transformers peft trl bitsandbytes datasets accelerate
```

## Demo

<!-- Link to demo video here -->

## Social Post

<!-- Link to social media post here -->