docs/troubleshooting.md

# Troubleshooting

## The model outputs `!!!!!!!!!!` (or one repeated token)

**This is the most likely failure you will hit.** It is a placement problem, not a broken
checkpoint.

Piko-9b is a hybrid model: 24 of its 32 layers use gated linear attention with a causal
convolution and a recurrent state kept in `float32` (`mamba_ssm_dtype`). That state does not
survive being split across CPU and GPU. When `device_map="auto"` cannot fit the model in VRAM it
silently offloads layers to CPU, the recurrent state is corrupted, and every logit collapses to
the same token.

It fails **silently** — `from_pretrained` succeeds, generation runs, and you get fluent-looking
garbage.

Observed on an RTX 5070 Ti (17.1 GB) with `device_map="auto"`, bfloat16:

```
prompt : "Write a Python function that reverses a string."
output : "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
```

Same weights, same machine, 4-bit and fully resident on the GPU:

```
output : "The capital of France is Paris."
```

**Fix — keep the whole model on one device:**

```python
model = AutoModelForMultimodalLM.from_pretrained(
    "Dexy2/Piko-9b",
    dtype=torch.bfloat16,
    device_map={"": 0},          # not "auto"
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
    ),
)
```

If you have ≥ 22 GB of VRAM you can drop the quantization config and keep `device_map={"": 0}`.

**How to detect it in your own code:**

```python
if len(set(response.replace(" ", ""))) < 5:
    raise RuntimeError("Degenerate output — model is probably split across devices")
```

## `ImportError: ... requires the Torchvision library`

`transformers` 5.x uses tensor-based fast image processors. Without `torchvision`,
`AutoProcessor.from_pretrained` raises before the model is even touched — so this hits you even
if you only want text.

```bash
pip install torchvision
```

Match the build to your torch install. If no wheel exists for your Python version (this happens
on pre-release Pythons such as 3.15), use Python 3.10–3.12.

## `AttributeError: module 'transformers' has no attribute 'AutoModelForMultimodalLM'`

Your `transformers` is too old. Piko-9b needs **≥ 5.5**, not the `>= 4.57` quoted in some earlier
documentation.

```bash
pip install -U "transformers>=5.5"
```

The `"transformers_version": "4.57.0.dev0"` recorded inside `config.json` is stale build metadata
and should not be used to pick a version.

## `TypeError: embedding(): argument 'indices' must be Tensor, not BatchEncoding`

In `transformers` 5.x, `apply_chat_template(..., return_tensors="pt")` returns a `BatchEncoding`,
not a bare tensor. Either index it, or ask for a dict:

```python
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
model.generate(**inputs, max_new_tokens=256)
```

## `The fast path is not available ... install flash-linear-attention / causal-conv1d`

A warning, not an error. The linear-attention layers fall back to a pure-PyTorch implementation
and everything still works, just more slowly.

```bash
pip install flash-linear-attention causal-conv1d
```

These require a recent torch; if the log says *"Skipping import of cpp extensions due to
incompatible torch version"*, the kernels are installed but unused, which is harmless.

This is **not** FlashAttention-2. Only 8 of the 32 layers use softmax attention at all, so
FlashAttention-2 has far less effect here than on a conventional transformer.

## CUDA out of memory

| Symptom | Action |
|---|---|
| OOM while loading | Use `--quantization 4bit`. Do **not** switch to `device_map="auto"` |
| OOM during generation on long prompts | KV cache for the 8 full-attention layers grows with length; shorten the prompt or lower `max_new_tokens` |
| OOM only when batching | Lower `--batch-size`; padding makes every sequence as long as the longest |

Do not set `max_memory` to force a CPU spill. That reintroduces the offload bug above.

## The model prints its reasoning

Piko-9b emits a `<think> … </think>` span before its answer. Strip it:

```python
answer = text.rsplit("</think>", 1)[-1].strip() if "</think>" in text else text.strip()
```

The examples in `examples/` do this by default; pass `--show-reasoning` to keep it.

## Output is identical every time / ignores `temperature`

`generation_config.json` sets no sampling parameters, so the shipped default is **greedy**.
Passing `temperature` alone does nothing — you must also pass `do_sample=True`:

```python
model.generate(**inputs, do_sample=True, temperature=0.7, top_p=0.95, max_new_tokens=512)
```

## `trust_remote_code=True` prompts or warnings

Not needed. This repository contains no `.py` files and no `auto_map`; the architecture is
native to `transformers`. Remove the flag.

## Batched outputs are wrong for short prompts

Check that padding stays on the left (`tokenizer_config.json` sets `padding_side: "left"`). Right
padding puts pad tokens between the prompt and the first generated token and corrupts
decoder-only generation.

## Image questions get vague or wrong answers

This is expected behaviour for this checkpoint, not a configuration error. The vision tower was
copied unchanged from `Qwen/Qwen3.5-9B` and was never trained or re-aligned against Piko's
fine-tuned language backbone. See [`reports/lineage_analysis.md`](../reports/lineage_analysis.md)
§4 and the measured results in the model card. If you need reliable document reading, use
`Qwen/Qwen3.5-9B` itself, where the tower and the backbone match.

## Windows notes

* Native Windows works, but `bitsandbytes` support is version-sensitive; if 4-bit fails to load,
  WSL2 with Ubuntu is the smoother path.
* Loading 21 GB from an external USB drive is I/O bound and can take 10–20 minutes. Copy the
  checkpoint to internal NVMe first — subsequent loads take seconds.
* Long paths: enable `LongPathsEnabled` or keep the checkout near the drive root.

## Linux notes

* Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` to reduce fragmentation on long runs.
* `TOKENIZERS_PARALLELISM=false` silences fork warnings during evaluation.