| # klaus-3 GPU services | |
| Deploy these to `~/workspace/veil-pgd` on klaus-3. They provide every | |
| model the Mac orchestrator needs, behind HTTP, so the Mac stays torch-free. | |
| ## Ports | |
| - `8081` qwen-3.5-4b (llama.cpp, OpenAI-compatible `/v1`) | |
| - `8082` gemma-4-4b (llama.cpp, OpenAI-compatible `/v1`) | |
| - `8090` vision service (FastAPI: CLIP + all-MiniLM embeddings + LPIPS) | |
| ## Setup | |
| ```bash | |
| cd ~/workspace/veil-pgd | |
| python3 -m venv .venv && . .venv/bin/activate | |
| # install torch matching the box CUDA first, then: | |
| pip install -r services/requirements-klaus3.txt | |
| ``` | |
| ## VRAM (8GB laptop 3070) | |
| Kill the old gemma-4-12b llama-server first to free VRAM. Then two Q4_K_M 4B | |
| VLMs (~2.6GB each) + CLIP (~1GB) coexist (~6GB). Do not also load the 12B. | |
| ## Run | |
| ```bash | |
| # 1. surrogate VLMs | |
| bash services/serve_vlms.sh | |
| # 2. vision service (CLIP + embeddings + LPIPS) | |
| . .venv/bin/activate | |
| uvicorn services.vision_service:app --host 0.0.0.0 --port 8090 | |
| ``` | |
| ## Verify | |
| ```bash | |
| curl localhost:8081/v1/models | |
| curl localhost:8082/v1/models | |
| curl localhost:8090/health | |
| ``` | |