README.md

metadata

title: Tech Advisor
emoji: 🔧
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.18.0
python_version: '3.13'
app_file: app.py
pinned: false
license: apache-2.0
short_description: AWS DevOps Agent expert — fine-tuned Nemotron 4B
tags:
  - build-small-hackathon
  - nemotron
  - aws
  - fine-tuning
  - off-the-grid
  - track:backyard
  - sponsor:nvidia
  - achievement:offgrid
  - achievement:welltuned

🔧 Tech Advisor

An AI expert on AWS DevOps Agent — fine-tuned on the latest documentation so it knows everything about the service: features, pricing, integrations, getting started, and best practices. Powered by NVIDIA Llama-3.1-Nemotron-Nano-4B (~5B params, well under the 32B cap).

What it does

1. Answers AWS DevOps Agent questions — features, pricing, integrations, architecture, getting started
2. Provides accurate details — trained on 66 pages of official documentation
3. Runs fully off the grid — no cloud APIs, no external calls, model runs locally on GPU

Tech Stack

• Model: NVIDIA Llama-3.1-Nemotron-Nano-4B-v1.1 — fine-tuned with QLoRA on AWS DevOps Agent docs
• Frontend: Gradio chat interface
• Hardware: Zero GPU (HF Spaces)
• Training: QLoRA (4-bit NF4, LoRA r=16) on AWS EC2 g5.xlarge (A10G)
• Training Data: 66 pages of AWS DevOps Agent User Guide → 1,230 instruction-response pairs
• Badge: Off the Grid — runs fully locally, no cloud APIs at inference 🔌

How to use

1. Ask about AWS DevOps Agent (e.g., "What is an Agent Space?", "How much does it cost?")
2. Get detailed, accurate answers based on the latest documentation

Training Pipeline

The model is fine-tuned on AWS documentation using QLoRA, then pushed to HF Hub.

training/
├── prepare_data.py      # Convert raw docs → training JSONL
├── train.py             # QLoRA fine-tuning script
├── push_to_hub.py       # Merge adapter + push to HF Hub
├── requirements.txt     # Training dependencies
└── data/
    ├── raw/             # AWS documentation markdown files
    └── train.jsonl      # Generated training pairs

Steps:

1. Add AWS docs to training/data/raw/ (one markdown file per topic) ✅ Done
2. python training/prepare_data.py — generate training data
3. python training/train.py — fine-tune on AWS EC2 with GPU
4. python training/push_to_hub.py — push merged model to HF Hub
5. Update MODEL_ID in app.py and deploy

See training/README.md for full details.

Setup (Deployment)

No API keys or secrets needed — the model runs fully locally on Zero GPU hardware. Select "ZeroGPU" as the hardware in your Space settings.

Git Remotes

This repo uses two remotes:

• origin → GitHub: https://github.com/hugotp-ui/20260607smallmodelshackathon.git
• space → HF Space: https://huggingface.co/spaces/build-small-hackathon/tech-advisor

git push origin main   # GitHub
git push space main    # Deploy to Hugging Face Space

Current Progress

• Project scaffolding (Gradio app, training pipeline, README)
• Raw training data collected (66 AWS DevOps Agent docs — full User Guide)
• Data preparation script generates 1,230 instruction-response pairs
• QLoRA training script ready (4-bit NF4, LoRA r=16, 3 epochs)
• HF Space created under build-small-hackathon/tech-advisor
• GPU instance launched (g5.xlarge, A10G 24GB, us-west-2)
• Fine-tuning complete (35 min, final loss 0.43)
• Merged model pushed to HF Hub (aslanconfig/tech-advisor-nemotron-4b)
• MODEL_ID updated in app.py to fine-tuned model
• Deployed to HF Space
• Record demo video
• Social media post

Training Cost

Item	Duration	Cost
EC2 g5.xlarge (A10G 24GB)	~2 hours total (includes setup + training)	~$2.02
Training time only	35 minutes	~$0.59
EBS storage (100GB gp3)	2 hours	~$0.01
Total		~$2.03

g5.xlarge on-demand rate: $1.006/hr in us-west-2. The actual fine-tuning of a 4B model with QLoRA costs under $1 in compute.

Training Infrastructure

• Instance: g5.xlarge (NVIDIA A10G, 24GB VRAM) in us-west-2
• Instance ID: i-0249ed98db8a6480b
• AMI: Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.7 (Ubuntu 22.04) — ami-0ca70308d230e8a6e
• SSH: ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@54.69.43.246
• Runtime: PyTorch 2.7 + CUDA 12.8 (pre-installed via /opt/pytorch)

Reproducible Setup Commands

# 1. Create EC2 key pair
aws ec2 create-key-pair --key-name hackathon-gpu --query 'KeyMaterial' --output text > ~/.ssh/hackathon-gpu.pem
chmod 400 ~/.ssh/hackathon-gpu.pem

# 2. Create security group with SSH access
aws ec2 create-security-group --group-name hackathon-gpu-sg --description "SSH access for hackathon GPU training"
aws ec2 authorize-security-group-ingress --group-id <sg-id> --protocol tcp --port 22 --cidr 0.0.0.0/0

# 3. Launch g5.xlarge instance
aws ec2 run-instances \
  --image-id ami-0ca70308d230e8a6e \
  --instance-type g5.xlarge \
  --key-name hackathon-gpu \
  --security-group-ids <sg-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":100,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=hackathon-training}]'

# 4. Copy project files to instance
scp -i ~/.ssh/hackathon-gpu.pem -r ./ ubuntu@<instance-ip>:~/hackathon/

# 5. SSH in
ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@<instance-ip>

# 6. Build Docker training image
cd ~/hackathon
docker build -t hackathon-train -f Dockerfile .

# 7. Run training in Docker
docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \
  bash -c "cd /workspace && python training/train.py"

# 8. Push merged model to HF Hub
docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \
  bash -c "cd /workspace && huggingface-cli login --token <HF_TOKEN> && python training/push_to_hub.py"

Dockerfile

FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel

WORKDIR /workspace

RUN pip install --no-cache-dir packaging ninja
ENV TORCH_CUDA_ARCH_LIST="8.6"
RUN pip install --no-cache-dir --no-build-isolation causal-conv1d
RUN pip install --no-cache-dir --no-build-isolation mamba-ssm
RUN pip install --no-cache-dir transformers peft trl bitsandbytes datasets accelerate

Demo

Social Post