README.md

---
language:
  - en
  - de
license: apache-2.0
library_name: transformers
tags:
  - qwen3.5
  - qwen3
  - qlora
  - dora
  - sft
  - german
  - 2b
  - text-generation
pipeline_tag: text-generation
base_model:
  - Qwen/Qwen3-Next-2B
---

# Quant Pico 2B

A 2B-parameter German-and-English instruction-tuned language model, fine-tuned from
Qwen3.5-2B (a.k.a. Qwen3-Next-2B) using QLoRA + DoRA across six sequential SFT
phases. Trained on a single Vast.ai RTX A4000 instance. Designed for local
inference on consumer GPUs (RTX 2060 / 3060 / 3090) and small-VRAM quantized
serving via llama.cpp / Ollama.

## Model details

- **Architecture:** Qwen3_5ForCausalLM (hybrid SSM + sparse full attention, `full_attention_interval=4`)
- **Parameters:** ~2.0B (hidden 2048, intermediate 6144, 24 layers)
- **Context length:** 262 144 tokens (inherited from base)
- **Tokenizer:** Qwen3.5 BPE, vocab 248 320, EOS `248044`
- **Precision:** bfloat16 (safetensors), plus f16 and Q4_K_M GGUF siblings
- **Chat template:** Standard Qwen3.5 chatml (vision/tool blocks included for
  compatibility; this is a text-only fine-tune)

## Training

- **Base model:** [`Qwen/Qwen3-Next-2B`](https://huggingface.co/Qwen/Qwen3-Next-2B)
  (branded "Qwen3.5 2B" in our internal naming)
- **Method:** QLoRA (4-bit base) + DoRA, r=96, alpha=192, all linear modules
- **Phases** (sequential SFT, ~42h wall-clock on A4000):
  1. Code
  2. Math
  3. Reasoning
  4. German
  5. Tool use
  6. Chat / mix
- **Identity tuning:** Yes — the model adopts the persona "Quant Pico 2B" via
  German-language system-prompt conditioning. See *Limitations* below.
- **NEFTune:** enabled
- **Optimiser:** paged_adamw_8bit
- **Hardware:** Vast.ai RTX A4000 (~$210 total)

> **Note on data disclosure.** Full per-phase dataset lists and token counts are
> recorded in the private training log. This card summarises the phase ordering
> and qualitative intent; the exact source datasets are not enumerated here.

## Intended use

- Local chat assistant on consumer hardware (German + English)
- Code and math assistance at the 2B-class capability ceiling
- Tool-calling experiments (template supports it; coverage is partial)

## Out-of-scope

- Production safety-critical applications
- High-stakes reasoning (medical, legal, financial)
- Long-context retrieval beyond what the base Qwen3.5-2B supports
  reliably in our benchmarks

## Limitations

- **Persona conditioning:** The model is biased toward responding as
  *"Quant Pico 2B, ein hilfsbereiter deutschsprachiger KI-Assistent"* when
  prompted. This is intentional, but downstream users may want to override
  the system prompt to neutralise it.
- **2B-class ceiling:** Code, math, and reasoning quality are bounded by
  the base model size. Do not expect frontier-model performance.
- **Hybrid-attention quirks:** The base uses linear (SSM-style) attention
  on most layers. Generation quality is sensitive to `temperature` and
  `top_p`; defaults baked into the Ollama Modelfile are `0.7 / 0.9`.
- **No RLHF or DPO:** This is pure SFT. There is no preference learning.

## Files in this repo

| File | Purpose |
|---|---|
| `final_model/model.safetensors` | HF-format weights, bfloat16 |
| `final_model/{config,generation_config,tokenizer,tokenizer_config}.json` | HF model + tokenizer metadata |
| `final_model/chat_template.jinja` | Qwen3.5 chatml template |
| `quant-pico-2b-f16.gguf` | llama.cpp / Ollama format, 16-bit |
| `quant-pico-2b-Q4_K_M.gguf` | llama.cpp / Ollama format, Q4 quant (~1.2 GB) |
| `Modelfile` | Ollama recipe (German system prompt, sampling defaults) |

## How to use

### With Hugging Face transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("20ZollCoder/quant-pico-2b", subfolder="final_model")
model = AutoModelForCausalLM.from_pretrained(
    "20ZollCoder/quant-pico-2b",
    subfolder="final_model",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "Du bist Quant Pico 2B, ein hilfsbereiter deutschsprachiger KI-Assistent."},
    {"role": "user", "content": "Erklaere mir in zwei Saetzen, was ein QLoRA ist."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

### With Ollama

```bash
# After pulling this repo
ollama create quant-pico-2b -f Modelfile
ollama run quant-pico-2b
```

### With llama.cpp

```bash
./llama-server -m quant-pico-2b-Q4_K_M.gguf -ngl 999 --port 8080
```

## Eval

No formal benchmarks are reported in this card. The model is in active
internal use; informal observations are summarised in the *Limitations*
section. A future revision will add lm-eval-harness numbers.

## Provenance

- Trained and packaged by `20ZollCoder` on a single Vast.ai A4000 instance,
  deployed and validated on a local MilanLinux RTX 2060 box.
- Build pipeline and phase scripts live in the `llm-finetuning-pipeline`
  reference suite; see the `local-llm-finetune-and-serve` skill
  (open-source documentation) for the patterns used.

## License

Apache 2.0, matching the base model. You may use, modify, and redistribute
under the terms of that license. Attribution to the original Qwen3.5-2B
authors (Alibaba) and to this fine-tune is appreciated but not required.