google

/

gemma-3-27b-it

like 1.97k

Follow

Google 55.4k

I deployed a Gemma-3-27b model

https://huggingface.co/google/gemma-3-27b-it

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: gemma-3-27b
  namespace: onecloud-dev
spec:
  predictor:
    containers:
    - args:
      - --model=google/gemma-3-27b-it
      - --tokenizer=google/gemma-3-27b-it
      - --tensor-parallel-size=8
      - --gpu-memory-utilization=0.9
      - --max-model-len=8192
      - --trust-remote-code
      - --enforce-eager
      env:
      - name: HF_TOKEN
        valueFrom:
          secretKeyRef:
            key: HF_TOKEN
            name: hf-secret
            optional: false
      image: vllm/vllm-openai:v0.9.2
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 5
        httpGet:
          path: /health
          port: 8000
        initialDelaySeconds: 900
        periodSeconds: 40
        timeoutSeconds: 10
      name: kserve-container
      ports:
      - containerPort: 8000
        name: http
        protocol: TCP
      readinessProbe:
        failureThreshold: 5
        httpGet:
          path: /health
          port: 8000
        initialDelaySeconds: 900
        periodSeconds: 15
        timeoutSeconds: 10
      resources:
        limits:
          cpu: "180"
          memory: 636Gi
          nvidia.com/gpu: "8"
        requests:
          cpu: "180"
          memory: 636Gi
          nvidia.com/gpu: "8"
      volumeMounts:
      - mountPath: /dev/shm
        name: shared-memory
    imagePullSecrets:
    - name: pull-secret
    labels:
      release: kserve-demo-gemma-3-27b-dev__v0.0.1
      type: model
    maxReplicas: 1
    minReplicas: 1
    scaleMetric: cpu
    scaleTarget: 75
    serviceAccountName: gemma-3-27b
    volumes:
    - emptyDir:
        medium: Memory
      name: shared-memory

it works well for image-to-text.

curl -X POST https://localhost:8000/v1/chat/completions     -H "Authorization: Bearer $TOKEN"     -H "Content-Type: application/json" -d'{
         "model": "google/gemma-3-27b-it",
         "messages": [
                 {
                         "role": "user",
                         "content": [
                                 {
                                         "type": "text",
                                         "text": "Describe this image in one sentence."
                                 },
                                 {
                                         "type": "image_url",
                                         "image_url": {
                                                 "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
                                         }
                                 }
                         ]
                 }
         ],
         "max_tokens":500,
         "stream":false
 }'
{"id":"chatcmpl-73c969e0-713d-98d7-82b2-c85ad6ae7b81","object":"chat.completion","created":1753881300,"model":"google/gemma-3-27b-it","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"Here's a breathtaking view of the Statue of Liberty Island in a captivating scene depicts the Statue of Liberty stands proudly on Liberty Island, with the iconic Statue of Liberty Island, framed by the cityscape in the Manhattan skyline. The in the background. The stands water, the cityscape in a beautiful, with a clear blue sky. The a clear, bright day. The sun shining a sunlit by the backdrop. The Statue of bright, with a sunny is in the blue, and a beautiful New York skyline, with a the against New York City. The is beautiful a is visible. The a of the and. \n\nHere's a the and of Liberty against the backdrop a bright blue sky, of and The of the clear.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":106}],"usage":{"prompt_tokens":276,"total_tokens":425,"completion_tokens":149,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

However, for text-to-text, it is a bit stupid

  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "google/gemma-3-27b-it",
  "messages": [
    {
      "role": "user",
      "content": "Please answer in one clear sentence: What is the capital of France?"
    }
  ],
  "max_tokens": 100,
  "stream": false
}'

{"id":"chatcmpl-0f352925-796d-9ccf-8a20-ff35a679b50d","object":"chat.completion","created":1753881805,"model":"google/gemma-3-27b-it","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"France is the capital?\nParis","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":106}],"usage":{"prompt_tokens":23,"total_tokens":31,"completion_tokens":8,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

Do you know why? Thanks.