20ZollCoder/quant-pico-2b

Quant Pico 2B

A 2B-parameter German-and-English instruction-tuned language model, fine-tuned from Qwen3.5-2B (a.k.a. Qwen3-Next-2B) using QLoRA + DoRA across six sequential SFT phases. Trained on a single Vast.ai RTX A4000 instance. Designed for local inference on consumer GPUs (RTX 2060 / 3060 / 3090) and small-VRAM quantized serving via llama.cpp / Ollama.

Model details

• Architecture: Qwen3_5ForCausalLM (hybrid SSM + sparse full attention, full_attention_interval=4)
• Parameters: ~2.0B (hidden 2048, intermediate 6144, 24 layers)
• Context length: 262 144 tokens (inherited from base)
• Tokenizer: Qwen3.5 BPE, vocab 248 320, EOS 248044
• Precision: bfloat16 (safetensors), plus f16 and Q4_K_M GGUF siblings
• Chat template: Standard Qwen3.5 chatml (vision/tool blocks included for compatibility; this is a text-only fine-tune)

Training

• Base model: Qwen/Qwen3-Next-2B (branded "Qwen3.5 2B" in our internal naming)
• Method: QLoRA (4-bit base) + DoRA, r=96, alpha=192, all linear modules
• Phases (sequential SFT, ~42h wall-clock on A4000):
  1. Code
  2. Math
  3. Reasoning
  4. German
  5. Tool use
  6. Chat / mix
• Identity tuning: Yes — the model adopts the persona "Quant Pico 2B" via German-language system-prompt conditioning. See Limitations below.
• NEFTune: enabled
• Optimiser: paged_adamw_8bit
• Hardware: Vast.ai RTX A4000 (~$210 total)

Note on data disclosure. Full per-phase dataset lists and token counts are recorded in the private training log. This card summarises the phase ordering and qualitative intent; the exact source datasets are not enumerated here.

Intended use

• Local chat assistant on consumer hardware (German + English)
• Code and math assistance at the 2B-class capability ceiling
• Tool-calling experiments (template supports it; coverage is partial)

Out-of-scope

• Production safety-critical applications
• High-stakes reasoning (medical, legal, financial)
• Long-context retrieval beyond what the base Qwen3.5-2B supports reliably in our benchmarks

Limitations

• Persona conditioning: The model is biased toward responding as "Quant Pico 2B, ein hilfsbereiter deutschsprachiger KI-Assistent" when prompted. This is intentional, but downstream users may want to override the system prompt to neutralise it.
• 2B-class ceiling: Code, math, and reasoning quality are bounded by the base model size. Do not expect frontier-model performance.
• Hybrid-attention quirks: The base uses linear (SSM-style) attention on most layers. Generation quality is sensitive to temperature and top_p; defaults baked into the Ollama Modelfile are 0.7 / 0.9.
• No RLHF or DPO: This is pure SFT. There is no preference learning.

Files in this repo

File	Purpose
final_model/model.safetensors	HF-format weights, bfloat16
final_model/{config,generation_config,tokenizer,tokenizer_config}.json	HF model + tokenizer metadata
final_model/chat_template.jinja	Qwen3.5 chatml template
quant-pico-2b-f16.gguf	llama.cpp / Ollama format, 16-bit
quant-pico-2b-Q4_K_M.gguf	llama.cpp / Ollama format, Q4 quant (~1.2 GB)
Modelfile	Ollama recipe (German system prompt, sampling defaults)

How to use

With Hugging Face transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("20ZollCoder/quant-pico-2b", subfolder="final_model")
model = AutoModelForCausalLM.from_pretrained(
    "20ZollCoder/quant-pico-2b",
    subfolder="final_model",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "Du bist Quant Pico 2B, ein hilfsbereiter deutschsprachiger KI-Assistent."},
    {"role": "user", "content": "Erklaere mir in zwei Saetzen, was ein QLoRA ist."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

With Ollama

# After pulling this repo
ollama create quant-pico-2b -f Modelfile
ollama run quant-pico-2b

With llama.cpp

./llama-server -m quant-pico-2b-Q4_K_M.gguf -ngl 999 --port 8080

Eval

No formal benchmarks are reported in this card. The model is in active internal use; informal observations are summarised in the Limitations section. A future revision will add lm-eval-harness numbers.

Provenance

• Trained and packaged by 20ZollCoder on a single Vast.ai A4000 instance, deployed and validated on a local MilanLinux RTX 2060 box.
• Build pipeline and phase scripts live in the llm-finetuning-pipeline reference suite; see the local-llm-finetune-and-serve skill (open-source documentation) for the patterns used.

License

Apache 2.0, matching the base model. You may use, modify, and redistribute under the terms of that license. Attribution to the original Qwen3.5-2B authors (Alibaba) and to this fine-tune is appreciated but not required.