docs/troubleshooting.md

Troubleshooting

The model outputs !!!!!!!!!! (or one repeated token)

This is the most likely failure you will hit. It is a placement problem, not a broken checkpoint.

Piko-9b is a hybrid model: 24 of its 32 layers use gated linear attention with a causal convolution and a recurrent state kept in float32 (mamba_ssm_dtype). That state does not survive being split across CPU and GPU. When device_map="auto" cannot fit the model in VRAM it silently offloads layers to CPU, the recurrent state is corrupted, and every logit collapses to the same token.

It fails silently — from_pretrained succeeds, generation runs, and you get fluent-looking garbage.

Observed on an RTX 5070 Ti (17.1 GB) with device_map="auto", bfloat16:

prompt : "Write a Python function that reverses a string."
output : "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"

Same weights, same machine, 4-bit and fully resident on the GPU:

output : "The capital of France is Paris."

Fix — keep the whole model on one device:

model = AutoModelForMultimodalLM.from_pretrained(
    "Dexy2/Piko-9b",
    dtype=torch.bfloat16,
    device_map={"": 0},          # not "auto"
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
    ),
)

If you have ≥ 22 GB of VRAM you can drop the quantization config and keep device_map={"": 0}.

How to detect it in your own code:

if len(set(response.replace(" ", ""))) < 5:
    raise RuntimeError("Degenerate output — model is probably split across devices")

ImportError: ... requires the Torchvision library

transformers 5.x uses tensor-based fast image processors. Without torchvision, AutoProcessor.from_pretrained raises before the model is even touched — so this hits you even if you only want text.

pip install torchvision

Match the build to your torch install. If no wheel exists for your Python version (this happens on pre-release Pythons such as 3.15), use Python 3.10–3.12.

AttributeError: module 'transformers' has no attribute 'AutoModelForMultimodalLM'

Your transformers is too old. Piko-9b needs ≥ 5.5, not the >= 4.57 quoted in some earlier documentation.

pip install -U "transformers>=5.5"

The "transformers_version": "4.57.0.dev0" recorded inside config.json is stale build metadata and should not be used to pick a version.

TypeError: embedding(): argument 'indices' must be Tensor, not BatchEncoding

In transformers 5.x, apply_chat_template(..., return_tensors="pt") returns a BatchEncoding, not a bare tensor. Either index it, or ask for a dict:

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
model.generate(**inputs, max_new_tokens=256)

The fast path is not available ... install flash-linear-attention / causal-conv1d

A warning, not an error. The linear-attention layers fall back to a pure-PyTorch implementation and everything still works, just more slowly.

pip install flash-linear-attention causal-conv1d

These require a recent torch; if the log says "Skipping import of cpp extensions due to incompatible torch version", the kernels are installed but unused, which is harmless.

This is not FlashAttention-2. Only 8 of the 32 layers use softmax attention at all, so FlashAttention-2 has far less effect here than on a conventional transformer.

CUDA out of memory

Symptom	Action
OOM while loading	Use --quantization 4bit. Do not switch to device_map="auto"
OOM during generation on long prompts	KV cache for the 8 full-attention layers grows with length; shorten the prompt or lower max_new_tokens
OOM only when batching	Lower --batch-size; padding makes every sequence as long as the longest

Do not set max_memory to force a CPU spill. That reintroduces the offload bug above.

The model prints its reasoning

Piko-9b emits a <think> … </think> span before its answer. Strip it:

answer = text.rsplit("</think>", 1)[-1].strip() if "</think>" in text else text.strip()

The examples in examples/ do this by default; pass --show-reasoning to keep it.

Output is identical every time / ignores temperature

generation_config.json sets no sampling parameters, so the shipped default is greedy. Passing temperature alone does nothing — you must also pass do_sample=True:

model.generate(**inputs, do_sample=True, temperature=0.7, top_p=0.95, max_new_tokens=512)

trust_remote_code=True prompts or warnings

Not needed. This repository contains no .py files and no auto_map; the architecture is native to transformers. Remove the flag.

Batched outputs are wrong for short prompts

Check that padding stays on the left (tokenizer_config.json sets padding_side: "left"). Right padding puts pad tokens between the prompt and the first generated token and corrupts decoder-only generation.

Image questions get vague or wrong answers

This is expected behaviour for this checkpoint, not a configuration error. The vision tower was copied unchanged from Qwen/Qwen3.5-9B and was never trained or re-aligned against Piko's fine-tuned language backbone. See reports/lineage_analysis.md §4 and the measured results in the model card. If you need reliable document reading, use Qwen/Qwen3.5-9B itself, where the tower and the backbone match.

Windows notes

• Native Windows works, but bitsandbytes support is version-sensitive; if 4-bit fails to load, WSL2 with Ubuntu is the smoother path.
• Loading 21 GB from an external USB drive is I/O bound and can take 10–20 minutes. Copy the checkpoint to internal NVMe first — subsequent loads take seconds.
• Long paths: enable LongPathsEnabled or keep the checkout near the drive root.

Linux notes

• Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation on long runs.
• TOKENIZERS_PARALLELISM=false silences fork warnings during evaluation.