KAGGLE_INFERENCE_GUIDE.md

Kaggle Inference Guide for Hopcoder-Mini-9B

Running Hopcoder-Mini-9B on Kaggle requires careful VRAM management. This guide covers the two deployment approaches (Transformers + 4-bit, or GGUF via Ollama/llama.cpp), common timeout failures, and proven workarounds.

---

Approach 1: Transformers + BitsAndBytes (4-bit QLoRA)

Recommended Kaggle Environment Settings

Setting	Value	Why
Accelerator	GPU T4 x1 (single is enough)	4-bit model ≈ 7 GB; T4 has 15 GB
Internet	Enabled	Required for from_pretrained() download
GPU type	Any T4; avoid P100 (no bf16)	T4 supports bfloat16 compute
Timeout	Default (9h)	Inference is fast; training needs long timeout

Minimum Working Inference Script

import os
import torch
from transformers import (
    AutoModelForImageTextToText,
    AutoProcessor,
    BitsAndBytesConfig,
)

# ── CRITICAL: Verify transformers version ──────────────────────
import transformers
_tfver = tuple(int(x) for x in transformers.__version__.split(".")[:3])
assert _tfver >= (5, 12, 1), (
    f"transformers {transformers.__version__} too old. "
    f"Run: pip install --upgrade 'transformers>=5.12.1' and restart kernel."
)

# ── Load model with 4-bit quantization ─────────────────────────
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForImageTextToText.from_pretrained(
    "TaimoorSiddiqui/Hopcoder-Mini-9B",
    quantization_config=quant_config,
    device_map="auto",         # auto-partitions across available GPUs
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",  # T4 does NOT support flash_attention_2
)

processor = AutoProcessor.from_pretrained(
    "TaimoorSiddiqui/Hopcoder-Mini-9B",
    trust_remote_code=True,
)

# ── Generate ───────────────────────────────────────────────────
messages = [
    {"role": "user", "content": "Write a Python function to check if a number is prime."},
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, return_tensors="pt").to(model.device)

# ── CRITICAL: Set reasonable generation limits ─────────────────
out = model.generate(
    **inputs,
    max_new_tokens=512,       # Start with 512; increase only if needed
    temperature=0.6,
    top_p=0.95,
    top_k=20,
    do_sample=True,
    pad_token_id=processor.tokenizer.pad_token_id,
    eos_token_id=processor.tokenizer.eos_token_id,
)

decoded = processor.decode(out[0], skip_special_tokens=True)
print(decoded)

Common Timeout Scenarios & Fixes

Symptom	Root Cause	Fix
Cell runs for 5+ minutes, no output	device_map="auto" on CPU-only instance	Switch to GPU accelerator
OutOfMemoryError during from_pretrained	Using load_in_8bit or no quantization on T4	Use load_in_4bit with bnb_4bit_use_double_quant=True
Generation hangs after first token	use_cache accidentally set to False in code	Do NOT set model.config.use_cache = False during inference
Returns empty string or truncated text	eos_token_id mismatch or pad token in EOS list	Verify generation_config.json has eos_token_id: 248046 only
KeyError: 'qwen3_5'	transformers version < 5.12.1	pip install --upgrade 'transformers>=5.12.1' then restart kernel
ImportError: bitsandbytes	bitsandbytes not installed or version < 0.46.1	pip install --upgrade 'bitsandbytes>=0.46.1' then restart
ValueError: attn_implementation="flash_attention_2"	T4 GPUs don't support Flash Attention 2	Change to "sdpa" or omit the parameter
Token-by-token generation is very slow (~1 tok/s)	No KV cache, or context too long (>32K tokens)	Set max_new_tokens=512, keep input < 4K tokens

VRAM Budget on T4 (15 GB total)

Component	VRAM Usage	Room Remaining
Model weights (4-bit NF4 + double quant)	~7 GB	8 GB
KV cache (512 new tokens)	~1 GB	7 GB
KV cache (4096 context window)	~2 GB	5 GB
Activation / optimizer states (inference)	~1 GB	4 GB
Headroom		4 GB (should not OOM with these settings)

---

Approach 2: GGUF via llama.cpp Server (Ollama-style)

If you've converted the model to GGUF and uploaded it alongside your Kaggle notebook:

Step 1: Upload GGUF files as Kaggle dataset

1. Upload hopcoder-mini-9b-Q4_K_M.gguf (5.6 GB) as a new Kaggle dataset
2. Attach the dataset to your notebook

Step 2: Install and run llama.cpp server

# Clone llama.cpp (already includes qwen3.5 support at b9846+)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4

# Start the server
./build/bin/llama-server \
    -m /kaggle/input/your-dataset/hopcoder-mini-9b-Q4_K_M.gguf \
    -c 8192 \
    --port 8080 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    -ngl 999  # offload all layers to GPU if CUDA is available
&

Step 3: Query via API

import requests
import json

response = requests.post(
    "http://localhost:8080/v1/chat/completions",
    headers={"Content-Type": "application/json"},
    json={
        "messages": [
            {"role": "user", "content": "Explain quantum computing in 3 sentences."}
        ],
        "max_tokens": 512,
        "temperature": 0.6,
        "top_p": 0.95,
    },
    stream=True,
)

for line in response.iter_lines():
    if line:
        data = json.loads(line.decode().replace("data: ", ""))
        if "choices" in data and data["choices"][0].get("delta", {}).get("content"):
            print(data["choices"][0]["delta"]["content"], end="", flush=True)

---

Quick Checklist Before Deployment

• transformers >= 5.12.1 installed (run pip show transformers)
• bitsandbytes >= 0.46.1 installed (run pip show bitsandbytes)
• Kaggle accelerator set to GPU (not CPU)
• trust_remote_code=True on all from_pretrained() calls
• attn_implementation="sdpa" (not flash_attention_2)
• max_new_tokens capped at 512–2048
• model.config.use_cache is not set to False
• Input text is under 4K tokens for fast response

---

Performance Benchmarks (T4, 4-bit, single GPU)

Context Length	max_new_tokens	Tokens/sec	Total Response Time
500 tokens	256	~12 tok/s	~21 seconds
1K tokens	512	~8 tok/s	~64 seconds
4K tokens	512	~4 tok/s	~128 seconds
4K tokens	2048	~3 tok/s	~683 seconds (11 min)

Recommendation: For interactive use, keep input < 1K tokens and max_new_tokens <= 512. For batch processing, you can increase both.