README.md

metadata

title: Gemma 4 12B QAT — Multimodal API
emoji: 🧠
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
license: apache-2.0

Gemma 4 12B QAT — Ollama Multimodal API

Self-hosted inference for gemma4:12b-it-qat via Ollama, wrapped in a FastAPI layer. Handles text, image, and text + image requests.

Endpoints

Method	Path	Description
GET	/health	Liveness + model status
POST	/chat	Text-only chat (application/json)
POST	/chat/image	Image ± text (multipart/form-data)
POST	/v1/chat/completions	OpenAI-compatible passthrough

Quick usage

Text

curl -X POST https://<space-url>/chat \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role":"user","content":"Explain transformers briefly."}]
  }'

Image + text

curl -X POST https://<space-url>/chat/image \
  -F "prompt=What is in this image?" \
  -F "file=@photo.jpg"

OpenAI SDK (Next.js)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://<space-url>/v1",
  apiKey: "ollama",          // any non-empty string
});

const res = await client.chat.completions.create({
  model: "gemma4:12b-it-qat",
  messages: [{ role: "user", content: "Hello!" }],
});

Hardware note

• Free CPU tier (2 vCPU / 16 GB) — works, ~20–30 tok/s, cold start ~5 min
• T4 GPU tier — recommended for production; ~80–120 tok/s

Model is pulled on first boot and cached at /app/models. Enable a persistent storage volume on your Space to avoid re-downloading on every restart.

Environment variables

Variable	Default	Notes
MODEL_TAG	gemma4:12b-it-qat	Any Ollama model tag
OLLAMA_MODELS	/app/models	Cache directory