--- title: Tech Advisor emoji: 🔧 colorFrom: blue colorTo: green sdk: gradio sdk_version: 6.18.0 python_version: '3.13' app_file: app.py pinned: false license: apache-2.0 short_description: AWS DevOps Agent expert — fine-tuned Nemotron 4B tags: - build-small-hackathon - nemotron - aws - fine-tuning - off-the-grid - track:backyard - sponsor:nvidia - achievement:offgrid - achievement:welltuned --- # 🔧 Tech Advisor An AI expert on **AWS DevOps Agent** — fine-tuned on the latest documentation so it knows everything about the service: features, pricing, integrations, getting started, and best practices. Powered by **NVIDIA Llama-3.1-Nemotron-Nano-4B** (~5B params, well under the 32B cap). ## What it does 1. **Answers AWS DevOps Agent questions** — features, pricing, integrations, architecture, getting started 2. **Provides accurate details** — trained on 66 pages of official documentation 3. **Runs fully off the grid** — no cloud APIs, no external calls, model runs locally on GPU ## Tech Stack - **Model**: [NVIDIA Llama-3.1-Nemotron-Nano-4B-v1.1](https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1) — fine-tuned with QLoRA on AWS DevOps Agent docs - **Frontend**: Gradio chat interface - **Hardware**: Zero GPU (HF Spaces) - **Training**: QLoRA (4-bit NF4, LoRA r=16) on AWS EC2 g5.xlarge (A10G) - **Training Data**: 66 pages of AWS DevOps Agent User Guide → 1,230 instruction-response pairs - **Badge**: Off the Grid — runs fully locally, no cloud APIs at inference 🔌 ## How to use 1. Ask about AWS DevOps Agent (e.g., "What is an Agent Space?", "How much does it cost?") 2. Get detailed, accurate answers based on the latest documentation ## Training Pipeline The model is fine-tuned on AWS documentation using QLoRA, then pushed to HF Hub. ``` training/ ├── prepare_data.py # Convert raw docs → training JSONL ├── train.py # QLoRA fine-tuning script ├── push_to_hub.py # Merge adapter + push to HF Hub ├── requirements.txt # Training dependencies └── data/ ├── raw/ # AWS documentation markdown files └── train.jsonl # Generated training pairs ``` **Steps:** 1. Add AWS docs to `training/data/raw/` (one markdown file per topic) ✅ Done 2. `python training/prepare_data.py` — generate training data 3. `python training/train.py` — fine-tune on AWS EC2 with GPU 4. `python training/push_to_hub.py` — push merged model to HF Hub 5. Update `MODEL_ID` in `app.py` and deploy See [`training/README.md`](training/README.md) for full details. ## Setup (Deployment) No API keys or secrets needed — the model runs fully locally on Zero GPU hardware. Select "ZeroGPU" as the hardware in your Space settings. ## Git Remotes This repo uses two remotes: - `origin` → GitHub: `https://github.com/hugotp-ui/20260607smallmodelshackathon.git` - `space` → HF Space: `https://huggingface.co/spaces/build-small-hackathon/tech-advisor` ```bash git push origin main # GitHub git push space main # Deploy to Hugging Face Space ``` ## Current Progress - [x] Project scaffolding (Gradio app, training pipeline, README) - [x] Raw training data collected (66 AWS DevOps Agent docs — full User Guide) - [x] Data preparation script generates 1,230 instruction-response pairs - [x] QLoRA training script ready (4-bit NF4, LoRA r=16, 3 epochs) - [x] HF Space created under `build-small-hackathon/tech-advisor` - [x] GPU instance launched (g5.xlarge, A10G 24GB, us-west-2) - [x] Fine-tuning complete (35 min, final loss 0.43) - [x] Merged model pushed to HF Hub (`aslanconfig/tech-advisor-nemotron-4b`) - [x] `MODEL_ID` updated in app.py to fine-tuned model - [x] Deployed to HF Space - [ ] Record demo video - [ ] Social media post ## Training Cost | Item | Duration | Cost | |------|----------|------| | EC2 g5.xlarge (A10G 24GB) | ~2 hours total (includes setup + training) | ~$2.02 | | Training time only | 35 minutes | ~$0.59 | | EBS storage (100GB gp3) | 2 hours | ~$0.01 | | **Total** | | **~$2.03** | g5.xlarge on-demand rate: $1.006/hr in us-west-2. The actual fine-tuning of a 4B model with QLoRA costs under $1 in compute. ## Training Infrastructure - **Instance**: g5.xlarge (NVIDIA A10G, 24GB VRAM) in us-west-2 - **Instance ID**: `i-0249ed98db8a6480b` - **AMI**: Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.7 (Ubuntu 22.04) — `ami-0ca70308d230e8a6e` - **SSH**: `ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@54.69.43.246` - **Runtime**: PyTorch 2.7 + CUDA 12.8 (pre-installed via `/opt/pytorch`) ## Reproducible Setup Commands ```bash # 1. Create EC2 key pair aws ec2 create-key-pair --key-name hackathon-gpu --query 'KeyMaterial' --output text > ~/.ssh/hackathon-gpu.pem chmod 400 ~/.ssh/hackathon-gpu.pem # 2. Create security group with SSH access aws ec2 create-security-group --group-name hackathon-gpu-sg --description "SSH access for hackathon GPU training" aws ec2 authorize-security-group-ingress --group-id --protocol tcp --port 22 --cidr 0.0.0.0/0 # 3. Launch g5.xlarge instance aws ec2 run-instances \ --image-id ami-0ca70308d230e8a6e \ --instance-type g5.xlarge \ --key-name hackathon-gpu \ --security-group-ids \ --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":100,"VolumeType":"gp3"}}]' \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=hackathon-training}]' # 4. Copy project files to instance scp -i ~/.ssh/hackathon-gpu.pem -r ./ ubuntu@:~/hackathon/ # 5. SSH in ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@ # 6. Build Docker training image cd ~/hackathon docker build -t hackathon-train -f Dockerfile . # 7. Run training in Docker docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \ bash -c "cd /workspace && python training/train.py" # 8. Push merged model to HF Hub docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \ bash -c "cd /workspace && huggingface-cli login --token && python training/push_to_hub.py" ``` ### Dockerfile ```dockerfile FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel WORKDIR /workspace RUN pip install --no-cache-dir packaging ninja ENV TORCH_CUDA_ARCH_LIST="8.6" RUN pip install --no-cache-dir --no-build-isolation causal-conv1d RUN pip install --no-cache-dir --no-build-isolation mamba-ssm RUN pip install --no-cache-dir transformers peft trl bitsandbytes datasets accelerate ``` ## Demo ## Social Post