README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
base_model_relation: finetune
library_name: transformers
language:
  - en
pipeline_tag: text-generation
tags:
  - abstention
  - honesty
  - calibration
  - gguf
  - safetensors
  - qwen3.5
---

# IDK v1

A 4B abstention fine-tune of Qwen3.5-4B. Instead of fabricating when it doesn't know, it emits a leading `[IDK]` marker plus a short reason — and, given a search tool, it looks the answer up first. Runs on an 8 GB GPU.

> ⚠️ **Experimental model — use at your own risk.** IDK v1 is an early research release provided as-is, with no guarantees of correctness, safety, or fitness for any purpose. It is a 4B model and can still be wrong or fabricate. You are responsible for validating its outputs before relying on them.

## What it does

Given a question, the model takes one of three actions:

- **Answer** — when the prompt or its own knowledge supports a reliable answer.
- **Search** — when a findable external/current fact is missing, it calls a `web_search(query)` tool and grounds its answer on the result.
- **Decline** — when it can't answer reliably and no tool resolves it, it replies with a leading `[IDK]` and a brief reason, rather than guessing.

Its abstention tracks difficulty: it declines more as questions get harder, and (with search) reserves `[IDK]` for what genuinely isn't findable.

## Recommended serving

- **Reasoning: ON.** Every training example carries a reasoning trace; calibration and instruction-following are meaningfully better with thinking enabled.
- **Format: conversational chat.** The abstention behavior lives in the conversational format it was trained on; rigid `\boxed{}`-style output prompts suppress it.
- **System prompt** granting the decline affordance, e.g.:
  ```
  You may answer, or decline with [IDK] and a brief reason if you are not
  confident. Do not guess.
  ```
- **Search tool (recommended for factual use).** Declare an OpenAI-style `web_search(query)` function backed by SearXNG, Serper, or any search API. Without it, the model can only answer or decline.
- **Sampler:** `top_k 40`, `top_p 0.95`, `min_p 0.05`, `repeat_penalty 1.1`.
- **Temperature:** minimal impact — tested across the full `0.2–1.0` range with little behavioral change, so anything in that band is fine.
- **Quantization:** `BF16` for best calibration; `Q4_K_M` (~2.7 GB) runs on 8 GB GPUs and abstains slightly more.

Formats in this repo:
- **`safetensors`** (merged bf16, ~8 GB) — for 🤗 Transformers / vLLM; load by repo id (see below).
- **[`IDK-v1-Q4_K_M.gguf`](https://huggingface.co/BrokenCompute/IDK-v1/blob/main/IDK-v1-Q4_K_M.gguf)** (~2.7 GB) — runs on 8 GB GPUs; llama.cpp / LM Studio.
- **[`IDK-v1-BF16.gguf`](https://huggingface.co/BrokenCompute/IDK-v1/blob/main/IDK-v1-BF16.gguf)** (~8.4 GB) — full precision for llama.cpp / LM Studio.

### Loading (Transformers)

The base is a vision-language model, so load with **`AutoModelForImageTextToText`** (not `AutoModelForCausalLM`), and apply the chat template with thinking on:

```python
import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

tok = AutoTokenizer.from_pretrained("BrokenCompute/IDK-v1")
model = AutoModelForImageTextToText.from_pretrained(
    "BrokenCompute/IDK-v1", dtype=torch.bfloat16, device_map="auto")

msgs = [
    {"role": "system", "content": "You may answer, or decline with [IDK] and a brief reason if you are not confident. Do not guess."},
    {"role": "user", "content": "Which jurist said the First Amendment 'may finally have worked itself pure'?"},
]
text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False, enable_thinking=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))
```

## Training

- **Base:** Qwen3.5-4B (via Unsloth), 4-bit QLoRA supervised fine-tune. No RLHF/DPO in this checkpoint.
- **Data:** 8,763-example abstention corpus. Every example carries a reasoning trace; a subset teaches `web_search` tool use.
- **Recipe:** LoRA rank 32 / alpha 32, 2 epochs, lr 2e-4, max-seq 4096, seed 3407, final loss ~0.95.
- Trained on a single **RTX 5060 Ti (16 GB)**.

## Limitations

- It's a 4B — more honest about its limits than most, but "more honest" is not "correct." Not for medical, legal, or financial decisions.
- Over-refusal without tools; pair with search for factual work.
- English; focused on Health, Law, and Software-Engineering domains.
- **Multimodal base, text-only fine-tune.** Qwen3.5-4B is a vision-language model; this fine-tune trained only the text pathway. The vision tower is the base's, carried along unmodified and untested here — treat this as a text model.
- Reasoning-off increases over-refusal — keep reasoning on. Quantization matters far less: with reasoning on, `Q4_K_M` tracks `BF16` closely on most benchmarks.

## License

Built on Qwen3.5-4B (Apache-2.0). This fine-tune and model card are released under Apache-2.0.