CLAUDE.md

# 💻 DeepShell ModUI — Developer Reference (CLAUDE.md)

> This file is the AI assistant context document for Claude and other AI tools working on this codebase.
> Keep this in sync with README.md after major changes.

**Live Demo**: https://deepshell.cloud
**HF Spaces**: https://huggingface.co/spaces/muralipala/deepshell
**GitHub**: https://github.com/muralipala1504/deepshell_modui (private)
**Cloudflare Worker**: deepshell-proxy → proxies deepshell.cloud to HF Spaces

## 🏗️ Architecture Overview

- **Frontend**: Vanilla JS, no frameworks, served by FastAPI
- **Backend**: FastAPI + SSE + `/translate` + `/tts` endpoints
- **Translation**: LibreTranslate (self-hosted, en→hi)
- **TTS**: Piper (hi_IN-rohan-medium) + Web Speech API (EN)
- **LLM**: Groq → Cerebras → Ollama
- **Deployment**: HuggingFace Spaces (single container) + docker-compose (self-hosted)
- **Modes**: Assistant (fast answers) + Trainer (structured learning) — Dry Run removed (no live terminal in free version)

## 📁 Key Files

| File | Purpose |
|------|---------|
| `app.js` | Frontend — SSE, TTS, Piper audio, lang selector |
| `index.html` | UI — chat, modes, theme, lang dropdown |
| `deepshell-backend/deepshell/__main__.py` | FastAPI — SSE + /translate + /tts + security middleware |
| `deepshell-backend/deepshell/llm.py` | LLM clients + prompts |
| `Dockerfile` | Docker Compose image (Piper bundled) |
| `Dockerfile.hf` | HuggingFace Spaces image (Piper + LibreTranslate bundled) |
| `docker-compose.yml` | Self-hosted — DeepShell + LibreTranslate sidecar |
| `start.sh` | Start DeepShell + LibreTranslate (webapp mode) |
| `stop.sh` | Stop all services |
| `start_hf.sh` | HF Spaces startup script |
| `requirements-translation.txt` | LibreTranslate version pin |
| `about.html`    | About page — mission, features, builder, special thanks |
| `contact.html`  | Contact page — email, GitHub, YouTube links |

## ⚠️ Important Deployment Note
- HF Spaces builds using `Dockerfile` (not `Dockerfile.hf`)
- Always keep both in sync before deploying: `cp Dockerfile.hf Dockerfile`
- `dockerfile: Dockerfile.hf` added to README.md header but HF ignores it — direct copy is the working fix

## 🔧 Environment Variables

| Variable | Default | Purpose |
|----------|---------|---------|
| `GROQ_API_KEY` | required | Groq LLM API key |
| `PROVIDER` | groq | LLM provider (groq/cerebras/ollama) |
| `PORT` | 8001 (7860 on HF) | Server port |
| `PIPER_BINARY` | ~/piper/piper | Piper TTS binary path |
| `PIPER_VOICE_DIR` | ~/piper/voices | Piper voice models directory |
| `LIBRETRANSLATE_URL` | http://localhost:5000/translate | Translation service URL |
| `CEREBRAS_API_KEY` | optional | Cerebras failover API key |
| `ADMIN_API_KEY` | required | Locks /chat/history endpoint — generate with `python3 -c "import secrets; print(secrets.token_hex(32))"` |
| `MAX_TTS_LENGTH` | 500 | Max TTS input characters |
| `MAX_TRANSLATE_LENGTH` | 1000 | Max translation input characters |
| `CORS_ORIGINS` | blank | Extra allowed origins (comma-separated) — defaults cover deepshell.cloud + HF Space |
| `DEBUG` | false | Enable debug mode — NEVER true in production |

## 🔒 Security Architecture (Added 2026-05-29)

### What is hardened

| Layer | Measure |
|-------|---------|
| `/health` | Returns timestamp only — no debug info, no internal config exposed |
| `/chat/history` | Locked behind `X-Admin-Key` header — returns 401 without valid `ADMIN_API_KEY` |
| `/translate` | Rate limited 30/minute per IP + input validation + language whitelist |
| `/tts` | Rate limited 20/minute per IP + input validation (max 500 chars) + language whitelist |
| `/chat/run-agent-stream` | Rate limited 10/minute per IP |
| `/chat/run-agent` | Rate limited 20/minute per IP |
| CORS | Locked to deepshell.cloud + www.deepshell.cloud + muralipala-deepshell.hf.space only |
| IP Logging | All requests log IP + endpoint + status — prompt content NEVER logged |
| Input validation | All endpoints validate via Pydantic v2 field_validator — lang whitelist, length caps |
| Language injection | Unsupported lang values rejected with 422 |
| Payload abuse | Oversized payloads rejected with 422 |
| Real IP detection | CF-Connecting-IP → X-Forwarded-For → client.host fallback chain for accurate rate limiting behind Cloudflare |

### ADMIN_API_KEY setup

```bash
# Generate
python3 -c "import secrets; print(secrets.token_hex(32))"

# Local dev
export ADMIN_API_KEY="<generated_key>"

# HF Spaces
# Settings → Variables and Secrets → New Secret → ADMIN_API_KEY
```

### Access /chat/history with admin key

```bash
curl -s https://deepshell.cloud/chat/history \
  -H "X-Admin-Key: <your_admin_key>" | python3 -m json.tool
```

### Security test commands

```bash
# Health — timestamp only
curl -s https://deepshell.cloud/health | python3 -m json.tool

# History without key — must return 401
curl -s https://deepshell.cloud/chat/history | python3 -m json.tool

# Rate limit test — 429 after 20 hits
for i in {1..25}; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST https://deepshell.cloud/tts \
    -H "Content-Type: application/json" \
    -d '{"text": "test", "lang": "hi"}')
  echo "Request $i: $STATUS"
done

# Oversized payload — must return 422
curl -s -X POST https://deepshell.cloud/tts \
  -H "Content-Type: application/json" \
  -d "{\"text\": \"$(python3 -c "print('A'*600)")\", \"lang\": \"hi\"}" | python3 -m json.tool

# Language injection — must return 422
curl -s -X POST https://deepshell.cloud/tts \
  -H "Content-Type: application/json" \
  -d '{"text": "test", "lang": "en; rm -rf /"}' | python3 -m json.tool
```

## 🌐 TTS Pipeline

### English
- Web Speech API (browser-native)
- Instantaneous response
- Triggered via Voice ON button

### Hindi (Piper)
- Sentence by sentence for low latency
- 300ms delay on first sentence to prevent cutoff
- Fallback to Web Speech API if backend fails
- Slight delay vs EN — expected (translate → Piper generate → audio blob → play)
- ✅ Fixed (2026-06-05): Code blocks now stripped before sending to Piper — prose only
- Fix: `app.js` — replaced backtick strip with full code block regex + inline code → "code snippet"
- ✅ Fixed (2026-06-15): Inline code spans (`command`) in Trainer responses were being replaced with the spoken phrase "code snippet" — repeated 5-9x per response (once per inline backtick in "Breaking it down" / "Pro tip" sections), making EN/HI audio sound broken.
- Fix: `app.js:97` — changed `.replace(/`[^`]+`/g, "code snippet")` to `.replace(/`([^`]+)`/g, "$1")`, so TTS now speaks the actual command/text inside inline code instead of a placeholder. Fenced ```bash``` blocks remain fully stripped (line 96, unchanged).

## 🌐 Lang Dropdown Status (2026-06-05)

| Lang | Status |
|------|--------|
| EN | ✅ Live — instant |
| HI | ✅ Live — slight delay (pipeline: translate + Piper TTS) |
| TA | ⏳ Coming Soon |
| TE | ⏳ Coming Soon |
| AR | ⏳ Coming Soon |

## 🚀 Deployment Paths

### Webapp (development)
```bash
./start.sh   # starts DeepShell + LibreTranslate
./stop.sh
```

### Docker Compose (self-hosted)
```bash
export GROQ_API_KEY=...
export ADMIN_API_KEY=...
docker compose up -d
```

### HuggingFace Spaces
- Branch: `hf-deploy`
- Push: `git push hf hf-deploy:main`
- Single container: Dockerfile.hf (Piper + LibreTranslate bundled)
- Port: 7860
- Secrets: GROQ_API_KEY + CEREBRAS_API_KEY + ADMIN_API_KEY set in HF Space settings
- URL: https://muralipala-deepshell.hf.space

### Custom Domain (deepshell.cloud)
- Registrar: Hostinger (complimentary domain)
- DNS: Cloudflare (nameservers: ingrid.ns.cloudflare.com, wilson.ns.cloudflare.com)
- Cloudflare Worker: deepshell-proxy proxies all traffic to muralipala-deepshell.hf.space
- SSL: Auto-provisioned by Cloudflare (free)
- No HF Pro needed
- Real user IPs passed via CF-Connecting-IP header for accurate rate limiting

## 🔁 Standard Deploy Workflow

```bash
# 1. Make changes on main
git add .
git commit -m "your message"
git push origin main

# 2. Merge to hf-deploy and push to HF
git checkout hf-deploy
git merge main
cp Dockerfile.hf Dockerfile
git add Dockerfile
git commit -m "sync Dockerfile for HF deploy"
git push hf hf-deploy:main
git checkout main
```

## 📧 Email Routing
- `contact@deepshell.cloud` → forwarded to muralidharmpala@gmail.com
- Configured via Cloudflare Email Routing (free)
- MX + SPF + DKIM records auto-added by Cloudflare

## 🎬 SysTelligence YouTube Videos (DeepShell Playlist)

### Trainer Mode Series (2026-06-05)
| Video | URL |
|-------|-----|
| LVM Extend Online | https://youtu.be/nswBYOLCRAM |
| Docker Compose Nginx + PostgreSQL | https://youtu.be/PS6pOot_q8E |
| Kubernetes CrashLoopBackOff | https://youtu.be/CAvrppozzQQ |
| Ansible User Management | https://youtu.be/FRRpX0yyUKw |

- Recorded via OBS (Firefox Classic theme, Window Capture)
- EN TTS only for Trainer series — HI fixed after recording
- All added to "DeepShell" playlist on SysTelligence channel

### Next (2026-06-06)
- HN (Hacker News) style videos planned — topics TBD

## 🗂️ Pending / Next Sprint

- [ ] HN videos — record + upload (2026-06-06)
- [ ] TA, TE, AR TTS — future sprints

---

---
title: DeepShell
emoji: 💻
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
license: mit
short_description: AI-powered DevOps assistant with Hindi TTS
pinned: false
---


## 🔍 SEO & Google Search Console
| Item | Detail |
|------|--------|
| GSC Property | https://www.deepshell.cloud/ |
| Verification | HTML meta tag in `index.html` (line 5) |
| Sitemap | https://www.deepshell.cloud/sitemap.xml (3 pages) |
| Robots | Cloudflare managed robots.txt (Googlebot allowed, AI scrapers blocked) |
| Sitemap endpoint | `GET /sitemap.xml` in `__main__.py` |
| Robots endpoint | `GET /robots.txt` in `__main__.py` (overridden by CF managed) |

> ⚠️ Do NOT remove the GSC meta tag from `index.html` — breaks GSC verification
> ⚠️ Do NOT disable CF managed robots.txt — protects against AI scrapers