| ---
|
| title: Gemma 4 12B QAT — Multimodal API
|
| emoji: 🧠
|
| colorFrom: blue
|
| colorTo: green
|
| sdk: docker
|
| pinned: false
|
| license: apache-2.0
|
| ---
|
|
|
| # Gemma 4 12B QAT — Ollama Multimodal API
|
|
|
| Self-hosted inference for `gemma4:12b-it-qat` via Ollama, wrapped in a
|
| FastAPI layer. Handles **text**, **image**, and **text + image** requests.
|
|
|
| ## Endpoints
|
|
|
| | Method | Path | Description |
|
| |--------|------|-------------|
|
| | `GET` | `/health` | Liveness + model status |
|
| | `POST` | `/chat` | Text-only chat (`application/json`) |
|
| | `POST` | `/chat/image` | Image ± text (`multipart/form-data`) |
|
| | `POST` | `/v1/chat/completions` | OpenAI-compatible passthrough |
|
|
|
| ## Quick usage
|
|
|
| ### Text
|
|
|
| ```bash
|
| curl -X POST https://<space-url>/chat \
|
| -H "Content-Type: application/json" \
|
| -d '{
|
| "messages": [{"role":"user","content":"Explain transformers briefly."}]
|
| }'
|
| ```
|
|
|
| ### Image + text
|
|
|
| ```bash
|
| curl -X POST https://<space-url>/chat/image \
|
| -F "prompt=What is in this image?" \
|
| -F "file=@photo.jpg"
|
| ```
|
|
|
| ### OpenAI SDK (Next.js)
|
|
|
| ```ts
|
| import OpenAI from "openai";
|
|
|
| const client = new OpenAI({
|
| baseURL: "https://<space-url>/v1",
|
| apiKey: "ollama", // any non-empty string
|
| });
|
|
|
| const res = await client.chat.completions.create({
|
| model: "gemma4:12b-it-qat",
|
| messages: [{ role: "user", content: "Hello!" }],
|
| });
|
| ```
|
|
|
| ## Hardware note
|
|
|
| - **Free CPU tier (2 vCPU / 16 GB)** — works, ~20–30 tok/s, cold start ~5 min
|
| - **T4 GPU tier** — recommended for production; ~80–120 tok/s
|
|
|
| Model is pulled on first boot and cached at `/app/models`.
|
| Enable a **persistent storage** volume on your Space to avoid re-downloading
|
| on every restart.
|
|
|
| ## Environment variables
|
|
|
| | Variable | Default | Notes |
|
| |----------|---------|-------|
|
| | `MODEL_TAG` | `gemma4:12b-it-qat` | Any Ollama model tag |
|
| | `OLLAMA_MODELS` | `/app/models` | Cache directory |
|
|
|
