KAGGLE_INFERENCE_GUIDE.md

# Kaggle Inference Guide for Hopcoder-Mini-9B

Running Hopcoder-Mini-9B on Kaggle requires careful VRAM management. This guide covers the two deployment approaches (Transformers + 4-bit, or GGUF via Ollama/llama.cpp), common timeout failures, and proven workarounds.

---

## Approach 1: Transformers + BitsAndBytes (4-bit QLoRA)

### Recommended Kaggle Environment Settings

| Setting | Value | Why |
|---|---|---|
| Accelerator | **GPU T4 x1** (single is enough) | 4-bit model ≈ 7 GB; T4 has 15 GB |
| Internet | **Enabled** | Required for `from_pretrained()` download |
| GPU type | Any T4; avoid P100 (no bf16) | T4 supports bfloat16 compute |
| Timeout | **Default (9h)** | Inference is fast; training needs long timeout |

### Minimum Working Inference Script

```python
import os
import torch
from transformers import (
    AutoModelForImageTextToText,
    AutoProcessor,
    BitsAndBytesConfig,
)

# ── CRITICAL: Verify transformers version ──────────────────────
import transformers
_tfver = tuple(int(x) for x in transformers.__version__.split(".")[:3])
assert _tfver >= (5, 12, 1), (
    f"transformers {transformers.__version__} too old. "
    f"Run: pip install --upgrade 'transformers>=5.12.1' and restart kernel."
)

# ── Load model with 4-bit quantization ─────────────────────────
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForImageTextToText.from_pretrained(
    "TaimoorSiddiqui/Hopcoder-Mini-9B",
    quantization_config=quant_config,
    device_map="auto",         # auto-partitions across available GPUs
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",  # T4 does NOT support flash_attention_2
)

processor = AutoProcessor.from_pretrained(
    "TaimoorSiddiqui/Hopcoder-Mini-9B",
    trust_remote_code=True,
)

# ── Generate ───────────────────────────────────────────────────
messages = [
    {"role": "user", "content": "Write a Python function to check if a number is prime."},
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, return_tensors="pt").to(model.device)

# ── CRITICAL: Set reasonable generation limits ─────────────────
out = model.generate(
    **inputs,
    max_new_tokens=512,       # Start with 512; increase only if needed
    temperature=0.6,
    top_p=0.95,
    top_k=20,
    do_sample=True,
    pad_token_id=processor.tokenizer.pad_token_id,
    eos_token_id=processor.tokenizer.eos_token_id,
)

decoded = processor.decode(out[0], skip_special_tokens=True)
print(decoded)
```

### Common Timeout Scenarios & Fixes

| Symptom | Root Cause | Fix |
|---|---|---|
| Cell runs for 5+ minutes, no output | `device_map="auto"` on CPU-only instance | Switch to GPU accelerator |
| `OutOfMemoryError` during `from_pretrained` | Using `load_in_8bit` or no quantization on T4 | Use `load_in_4bit` with `bnb_4bit_use_double_quant=True` |
| Generation hangs after first token | `use_cache` accidentally set to `False` in code | Do NOT set `model.config.use_cache = False` during inference |
| Returns empty string or truncated text | `eos_token_id` mismatch or pad token in EOS list | Verify `generation_config.json` has `eos_token_id: 248046` only |
| `KeyError: 'qwen3_5'` | transformers version < 5.12.1 | `pip install --upgrade 'transformers>=5.12.1'` then **restart kernel** |
| `ImportError: bitsandbytes` | bitsandbytes not installed or version < 0.46.1 | `pip install --upgrade 'bitsandbytes>=0.46.1'` then restart |
| `ValueError: attn_implementation="flash_attention_2"` | T4 GPUs don't support Flash Attention 2 | Change to `"sdpa"` or omit the parameter |
| Token-by-token generation is very slow (~1 tok/s) | No KV cache, or context too long (>32K tokens) | Set `max_new_tokens=512`, keep input < 4K tokens |

### VRAM Budget on T4 (15 GB total)

| Component | VRAM Usage | Room Remaining |
|---|---|---|
| Model weights (4-bit NF4 + double quant) | ~7 GB | 8 GB |
| KV cache (512 new tokens) | ~1 GB | 7 GB |
| KV cache (4096 context window) | ~2 GB | 5 GB |
| Activation / optimizer states (inference) | ~1 GB | 4 GB |
| **Headroom** | | **4 GB** (should not OOM with these settings) |

---

## Approach 2: GGUF via llama.cpp Server (Ollama-style)

If you've converted the model to GGUF and uploaded it alongside your Kaggle notebook:

### Step 1: Upload GGUF files as Kaggle dataset

1. Upload `hopcoder-mini-9b-Q4_K_M.gguf` (5.6 GB) as a new Kaggle dataset
2. Attach the dataset to your notebook

### Step 2: Install and run llama.cpp server

```bash
# Clone llama.cpp (already includes qwen3.5 support at b9846+)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4

# Start the server
./build/bin/llama-server \
    -m /kaggle/input/your-dataset/hopcoder-mini-9b-Q4_K_M.gguf \
    -c 8192 \
    --port 8080 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    -ngl 999  # offload all layers to GPU if CUDA is available
&
```

### Step 3: Query via API

```python
import requests
import json

response = requests.post(
    "http://localhost:8080/v1/chat/completions",
    headers={"Content-Type": "application/json"},
    json={
        "messages": [
            {"role": "user", "content": "Explain quantum computing in 3 sentences."}
        ],
        "max_tokens": 512,
        "temperature": 0.6,
        "top_p": 0.95,
    },
    stream=True,
)

for line in response.iter_lines():
    if line:
        data = json.loads(line.decode().replace("data: ", ""))
        if "choices" in data and data["choices"][0].get("delta", {}).get("content"):
            print(data["choices"][0]["delta"]["content"], end="", flush=True)
```

---

## Quick Checklist Before Deployment

- [ ] `transformers >= 5.12.1` installed (run `pip show transformers`)
- [ ] `bitsandbytes >= 0.46.1` installed (run `pip show bitsandbytes`)
- [ ] Kaggle accelerator set to GPU (not CPU)
- [ ] `trust_remote_code=True` on all `from_pretrained()` calls
- [ ] `attn_implementation="sdpa"` (not `flash_attention_2`)
- [ ] `max_new_tokens` capped at 512–2048
- [ ] `model.config.use_cache` is **not** set to `False`
- [ ] Input text is under 4K tokens for fast response

---

## Performance Benchmarks (T4, 4-bit, single GPU)

| Context Length | max_new_tokens | Tokens/sec | Total Response Time |
|---|---|---|---|
| 500 tokens | 256 | ~12 tok/s | ~21 seconds |
| 1K tokens | 512 | ~8 tok/s | ~64 seconds |
| 4K tokens | 512 | ~4 tok/s | ~128 seconds |
| 4K tokens | 2048 | ~3 tok/s | ~683 seconds (11 min) |

> **Recommendation:** For interactive use, keep input < 1K tokens and `max_new_tokens <= 512`. For batch processing, you can increase both.