README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
  - deepreinforce-ai/Ornith-1.0-9B
  - Qwen/Qwen3.5-9B
base_model_relation: merge
tags:
  - piko
  - piko-9b
  - multimodal
  - vision-language
  - hybrid-attention
  - linear-attention
  - qwen3_5
  - ocr
  - document-understanding
---

<div align="center">

<img src="piko-logo.png" alt="Piko-9b" width="380">

**A 9.65B hybrid-attention vision-language model, composed from two open checkpoints.**

[![License](https://img.shields.io/badge/license-Apache--2.0-0072bc)](LICENSE)
[![Params](https://img.shields.io/badge/params-9.65B-0072bc)](#architecture)
[![Context](https://img.shields.io/badge/context-262K%20declared-0072bc)](#context-length)

</div>

---

## Model summary

Piko-9b is a 9.65-billion-parameter vision-language model that takes text and images and produces
text. It is built by splicing together two existing open checkpoints rather than trained from
scratch:

| Component | Parameters | Source | State |
|---|---:|---|---|
| Language backbone | 9,197,093,888 | Fine-tune of `deepreinforce-ai/Ornith-1.0-9B` | Fine-tuned |
| Vision tower + merger | 456,010,480 | `Qwen/Qwen3.5-9B` | **Copied verbatim** |

It emits a `<think>…</think>` reasoning trace before its answer, uses greedy decoding by default,
and needs about 8 GB of VRAM in 4-bit.

**On a 70-case regression suite it scores 65/70, against 63/70 for `Qwen/Qwen3.5-9B` measured
identically. That difference is not statistically significant.** Full detail
[below](#benchmark-results).

## What Piko-9b is

A composition, verified at the tensor level. All 775 tensors in the published checkpoint are
bitwise identical to a tensor in one of the two source checkpoints — 427 from the language
backbone, 348 from Qwen's vision tower, none unaccounted for.

The language backbone is genuinely fine-tuned: 0 of 14 sampled tensors match stock Qwen3.5-9B, with
1.5–5.0 % relative L2 drift. That fine-tuning came from a chain of merged QLoRA stages
(`wraithfast-phase10 → 12 → 13 → 14 → 15`) applied to Ornith-1.0-9B.

The vision tower is not fine-tuned at all. It was downloaded from `Qwen/Qwen3.5-9B` and attached
unchanged.

## Relationship to the base model

```
Qwen/Qwen3.5-9B-Base  (Apache-2.0)
   │
   ├── Qwen/Qwen3.5-9B ──────────── vision tower + merger, copied verbatim ──┐
   │                                                                        │
   └── deepreinforce-ai/Ornith-1.0-9B  (MIT)                                 │
            │ QLoRA r=32 α=64, merged after each stage                       │
            └── wraithfast-phase15-150k-full-ft ─── language backbone ───────┤
                                                                             │
                                                    Dexy2/Piko-9b  ──────────┘
```

### What this means in practice

The vision tower was trained by Qwen to project into **Qwen3.5-9B's** embedding space. The
backbone it is attached to has since drifted 1.5–5 % away from that space, and **no vision
training was performed to re-align it.**

That was expected to break image handling. Measured, it does not: OCR scores 10/10 and document
understanding 10/10 on the custom suite. This is the most surprising result in the release and it
is reported as an empirical finding, not as evidence that the composition was principled.

Full evidence: [`reports/lineage_analysis.md`](reports/lineage_analysis.md).

### Corrections to the previous model card

The earlier release described six training stages totalling 520,000 examples, and published nine
benchmark scores. Tensor comparison shows **none of those stages are present in these weights**,
and the nine scores were measured on a different, text-only checkpoint. Details:
[`reports/repository_audit.md`](reports/repository_audit.md) §5, [`CHANGELOG.md`](CHANGELOG.md).

## Intended uses

Supported by measurement on this checkpoint:

* **Document and receipt reading** — extracting fields from rendered documents, including
  OCR-error correction. 10/10 and 10/10 on the custom suite.
* **Table and chart reading** — 9/10.
* **General instruction following and reasoning** — 9/10 and 10/10.
* **Code generation** — 5/5, verified by executing the generated functions against assertions.
* **Long-context retrieval** — 5/5 at 2K, 8K and 32K filler tokens, depths 0.1–0.9.

## Out-of-scope uses

* **Anything safety-critical** — medical, legal, financial, or operational decisions.
* **Factual questions about entities it may not know.** It invented a plausible summary of a
  non-existent treaty when asked. See [Limitations](#limitations).
* **Adversarial or untrusted input without downstream validation.** No specific prompt-injection
  defence; this matters most in document pipelines, its strongest use case.
* **Video or audio.** Not supported — see [Supported modalities](#supported-modalities).
* **Deployments needing a stable self-identity without a system prompt.** Asked what it is, this
  checkpoint answers *"I am Wraith, an AI model."*

## Architecture

`Qwen3_5ForConditionalGeneration`, `model_type: qwen3_5`. A **hybrid attention stack**, not a
conventional transformer.

| Language model | | Vision encoder | |
|---|---:|---|---:|
| Hidden size | 4,096 | Depth | 27 |
| Layers | 32 | Hidden size | 1,152 |
| — linear attention | 24 | Attention heads | 16 |
| — full attention | 8 (every 4th) | Intermediate size | 4,304 |
| Attention heads | 16 | Patch size | 16 |
| Key/value heads | 4 (GQA) | Spatial merge | 2 |
| Head dim | 256 | Output size | 4,096 |
| Intermediate size | 12,288 | Position embeddings | 2,304 |
| Vocabulary | 248,320 | DeepStack indexes | `[]` (disabled) |

Linear attention: conv kernel 4, key head dim 128, 16 key heads, 32 value heads, value head dim
128, SSM state in `float32`.

Positional encoding: **mRoPE**, interleaved, sections `[11, 11, 10]`, `partial_rotary_factor` 0.25,
`rope_theta` 10,000,000. No RoPE scaling is applied.

Weights: 11 safetensors shards, 21.3 GB, **BF16** throughout.

### Parameter count

**9,653,104,368** total, read from the safetensors headers. Note that a 4-bit loaded model reports
5,724,972,272 — that is the packed element count, not the logical parameter count.

### Context length

`max_position_embeddings` is **262,144**, native to the architecture with no scaling trick.

**That is a configuration value, not a validated capability.** Measured: retrieval works at 2K, 8K
and ~32K filler tokens; a 14,429-token prompt was answered correctly in 3.5 s. Beyond ~32K is
untested, and a 32,768-token single forward pass exceeded the 15.92 GB test GPU.

### Supported modalities

| Modality | Metadata | Weights | Trained here | Validated |
|---|---|---|---|---|
| Text | Yes | Yes | Yes | **Yes** |
| Image | Yes | Yes (456 M, verbatim Qwen) | **No** | **Yes — works** |
| Video | Yes (`video_preprocessor_config.json`, `<\|video_pad\|>`) | Shares image tower | No | **Not tested** |
| Audio | Token IDs only | **No encoder** | No | **Not supported** |

The audio tokens (`<|audio_start|>`, `<|audio_pad|>`, `<|audio_end|>`) are inherited tokenizer
metadata. There is no audio encoder in the weights.

## Training and creation methodology

**The published weights contain no training performed under the Piko name.** They are a
tensor-level splice, dated 2026-07-24, of a language checkpoint last written 2026-07-16 and a
vision tower downloaded unchanged.

Six QLoRA adapters trained between 2026-07-20 and 2026-07-24 exist in the source workspace
(120k, 75k, 75k, 30k, 70k rows plus a vision remap, all r=32 α=64). Tensor comparison proves
**none of them are merged into these weights**: the LoRA deltas are non-zero, and the published
tensors match the pre-adapter checkpoint exactly.

What *is* in the weights is the WraithFast fine-tuning chain that produced
`wraithfast-phase15-150k-full-ft` from Ornith-1.0-9B — five merged QLoRA stages, each
`r=32, α=64, lora_dropout=0.0`, 1 epoch, cosine schedule, `paged_adamw_8bit`, on a single consumer
GPU.

### Training datasets

**Could not be fully verified.** Dataset manifests in the source workspace record category names
and row counts (code, math, science, general dialogue, identity rehearsal, news) but several
upstream sources are identified only by local filename, with no licence recorded.

One finding is worth stating plainly: the "OCR" datasets were **text-only**. The project's own
manifest says so —

> `"note": "OCR rows train interpretation of OCR/image text detection outputs, not direct pixel vision."`

— and none of that data is in these weights regardless.

### Data licensing and provenance

Unresolved. See [`NOTICE`](NOTICE) for what *is* established: the MIT licence covering the
language backbone's ancestor and the Apache-2.0 licence covering the vision tower.

## Evaluation methodology

70 hand-written cases, graded by deterministic Python checks. **No judge model** — every result is
reproducible and carries no grader bias. Image fixtures are rendered from code, so there is no
dataset licence and no network dependency.

Both models were run with identical prompts, precision, quantization, decoding, batch size, seed,
and grading code. The comparison tool refuses to emit a table if any of those differ.

| | |
|---|---|
| Hardware | RTX 5070 Ti, 15.92 GB |
| Precision | bfloat16 compute, 4-bit NF4 weights |
| Decoding | greedy, `max_new_tokens=512`, seed 0, batch 1 |
| torch / transformers | 2.10.0+cu128 / 5.5.0 |
| Date | 2026-07-29 |

Reproduce: [`docs/evaluation.md`](docs/evaluation.md).

## Benchmark results

### Custom regression suite — measured on these weights

| Benchmark | Piko-9b | Base model | Difference | Examples | Status |
| --------- | ------: | ---------: | ---------: | -------: | ------ |
| OCR | 100.0% | 100.0% | +0.0% | 10 | No significant difference |
| Document understanding | 100.0% | 100.0% | +0.0% | 10 | No significant difference |
| Reasoning | 100.0% | 100.0% | +0.0% | 10 | No significant difference |
| Long-context retrieval | 100.0% | 100.0% | +0.0% | 5 | No significant difference |
| Coding (executed) | 100.0% | 80.0% | +20.0% | 5 | No significant difference |
| Instruction following | 90.0% | 80.0% | +10.0% | 10 | No significant difference |
| Tables and charts | 90.0% | 90.0% | +0.0% | 10 | No significant difference |
| Hallucination / safety | 70.0% | 70.0% | +0.0% | 10 | No significant difference |
| **Overall** | **65/70 (92.9%)** | **63/70 (90.0%)** | **+2.9%** | 70 | No significant difference |

Base model = `Qwen/Qwen3.5-9B`, the source of Piko-9b's vision tower, run identically on the same
day and hardware.

**Status compares 95% Wilson score intervals.** At 5–10 examples per category the intervals are
wide and every difference overlaps. The +20 % on coding is 5 cases versus 4 of 5 — one example.
**Piko-9b is not shown to outperform Qwen3.5-9B.**

### Adjudication of the 5 Piko-9b failures

Reporting the raw score alongside what the grader actually caught:

| Case | Raw | Reality |
|---|---|---|
| `hl-01` | FAIL | **Genuine hallucination** — invented a full summary of a non-existent treaty |
| `if-04` | FAIL | **Genuine miss** — "Rain falls from the sky" contains an 'e' in "the" |
| `hl-02` | FAIL | Grader artefact — model correctly said the 2027 Nobel *"has not been awarded yet"* |
| `hl-05` | FAIL | Grader artefact — model correctly said the stdlib *"does not have"* that function |
| `tab-05` | FAIL | Truncation — JSON was correct but cut off at 512 tokens by the reasoning trace |

So 2 genuine failures, 3 artefacts. The headline number stays 65/70 as measured; the grader was
not retuned after the fact.

### Standard public benchmarks

| Benchmark | Piko-9b | Base model | Difference | Examples | Status |
| --------- | ------: | ---------: | ---------: | -------: | ------ |
| IFEval | — | — | — | — | **Not run** |
| MMLU-Pro | — | — | — | — | **Not run** |
| GSM8K | — | — | — | — | **Not run** |
| HumanEval | — | — | — | — | **Not run** |
| OCRBench | — | — | — | — | **Not run** |
| DocVQA | — | — | — | — | **Not run** |
| ChartQA | — | — | — | — | **Not run** |
| TextVQA | — | — | — | — | **Not run** |
| MMMU | — | — | — | — | **Not run** |

Each needs 1–3 hours per model on this hardware, and each needs a paired baseline run to mean
anything. The scripts are present and runnable; enable them in
[`evaluation/configs/piko_9b.yaml`](evaluation/configs/piko_9b.yaml).

> Any benchmark numbers you have seen for Qwen3.5-9B belong to **Qwen**, not to Piko-9b. The only
> numbers in this card measured on Piko-9b are in the custom suite table above.

## Inference

> **Keep the model on one device.** `device_map="auto"` on a GPU too small to hold it offloads
> layers to CPU, corrupts the hybrid linear-attention state, and the model emits a single repeated
> character with **no error raised**. Use `device_map={"": 0}` and a quantization that fits.
> [`docs/troubleshooting.md`](docs/troubleshooting.md)

`trust_remote_code` is **not** required. `torchvision` **is** required, even for text-only use.

```python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, BitsAndBytesConfig

model = AutoModelForMultimodalLM.from_pretrained(
    "Dexy2/Piko-9b",
    dtype=torch.bfloat16,
    device_map={"": 0},
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
    ),
)
model.eval()
processor = AutoProcessor.from_pretrained("Dexy2/Piko-9b")

messages = [
    {"role": "system", "content": "You are Piko-9, an AI assistant. Be accurate and concise."},
    {"role": "user", "content": [
        {"type": "image", "url": "receipt.png"},
        {"type": "text", "text": 'Return only JSON: {"merchant": str, "date": str, "total": float}'},
    ]},
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=768, do_sample=False)

text = processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
answer = text.rsplit("</think>", 1)[-1].strip() if "</think>" in text else text.strip()
print(answer)
```

**Budget tokens for reasoning.** The `<think>` span consumes `max_new_tokens`; one suite case
failed purely because the closing brace of correct JSON fell past a 512-token limit. Allow
768–1024 for structured output.

**Sampling is off by default.** `generation_config.json` sets no sampling parameters, so passing
`temperature` alone does nothing — pass `do_sample=True` as well.

Ready-made scripts: [`examples/`](examples/) — text, multimodal, batch, and an interactive CLI with
streaming. All validate VRAM before loading and refuse to enable offload.

## Hardware requirements

| Configuration | Weights | Practical VRAM | Status |
|---|---:|---:|---|
| bfloat16 | 19.8 GB | ~22 GB | Not tested (GPU too small) |
| 8-bit | 10.1 GB | ~12 GB | Not tested |
| **4-bit NF4** | 5.6 GB | **7.4 GB measured** | **Validated** |

Measured on an RTX 5070 Ti (15.92 GB), 4-bit, weights on NVMe:

| | |
|---|---|
| Cold load | 101–119 s (**10–25 min from external USB**) |
| Resident VRAM | 7.37 GB |
| Host RSS | 1.28 GB |
| Decode, batch 1 | 29–36 tok/s |
| Decode, batch 4 | 81–85 tok/s |
| Prefill | ~5,000–5,600 tok/s |
| Image preprocessing | 13 ms median |
| Peak VRAM at 8K context | 11.5 GB |

Decode throughput is **flat across context length** — the hybrid stack keeps a fixed-size state in
24 of 32 layers. Figures are a floor: `flash-linear-attention` and `causal-conv1d` were not
installed.

Full detail: [`reports/performance_report.md`](reports/performance_report.md),
[`docs/hardware.md`](docs/hardware.md).

## Quantization

4-bit NF4 is the validated path and **the vision tower survives it** — OCR 10/10 and document
understanding 10/10 were measured under NF4. 8-bit should work but was not tested.

No GGUF, AWQ, or GPTQ artefact has been produced. Scripts for AWQ and GPTQ are provided but
unexecuted; both exclude the linear-attention state parameters and leave the vision tower in bf16
by default, because text-only calibration can break the image path while every text metric stays
healthy. [`docs/quantization.md`](docs/quantization.md)

## Limitations

* **It hallucinates confidently.** Asked about a non-existent treaty, it produced a fluent
  invented summary. It scored 7/10 on a 10-prompt hallucination-and-safety probe — the same as
  the base model. Verify factual claims.
* **No demonstrated advantage over its base model.** 65/70 versus 63/70 is within noise. If you
  want Qwen3.5-9B's behaviour, use Qwen3.5-9B — it is the better-documented, better-supported
  choice with a matched vision tower.
* **The vision tower was never aligned to this backbone.** It works on the 30 synthetic document
  images tested. Photographs, handwriting, low-quality scans, and natural scenes were **not**
  tested.
* **Identity is not stable without a system prompt.** It answers *"I am Wraith."*
* **Alignment is inherited, not trained.** No safety tuning was performed for this checkpoint.
* **Context beyond ~32K is unverified** despite the 262K declaration.
* **Video and audio do not work** despite metadata suggesting otherwise.
* **CPU offload silently corrupts it** — the most likely way a deployment breaks.
* **Standard public benchmarks were not run**, so there is no comparison to the wider field.

## Bias, safety, and hallucination risks

The language backbone inherits whatever biases exist in Ornith-1.0-9B, in Qwen3.5-9B-Base beneath
it, and in the undocumented fine-tuning data. None of this was audited.

* **Hallucination** is the primary risk, and the primary use case — document extraction — is one
  where a fluent wrong answer is easy to miss. Validate structured output against a schema.
* **Refusals** were probed with 5 harmful prompts and passed all 5, plus one jailbreak string.
  That is a smoke test, not an assurance.
* **Prompt injection.** A document containing instructions is input the model may follow. Treat
  every output as untrusted data, never as commands. [`SECURITY.md`](SECURITY.md)
* **Language coverage.** Only English was tested, though the tokenizer is multilingual.

## Reproducibility

```bash
pip install -r requirements.txt
python evaluation/custom_suite/build_assets.py

python evaluation/run_smoke_eval.py --model Dexy2/Piko-9b          # ~5 min

python evaluation/custom_suite/run_custom_eval.py \
  --model Dexy2/Piko-9b --label piko-9b --quantization 4bit \
  --max-new-tokens 512 --output evaluation/results/custom_suite_piko-9b.json

python evaluation/custom_suite/run_custom_eval.py \
  --model Qwen/Qwen3.5-9B --label qwen3.5-9b-base --quantization 4bit \
  --max-new-tokens 512 --output evaluation/results/custom_suite_qwen3.5-9b-base.json

python evaluation/compare_results.py \
  --candidate evaluation/results/custom_suite_piko-9b.json \
  --baseline  evaluation/results/custom_suite_qwen3.5-9b-base.json \
  --output    evaluation/results/comparison.md
```

Every result file records model, revision, timestamp, hardware, OS, Python, torch, transformers,
precision, quantization, batch size, decoding parameters, seed, example count, and failures.

Verify provenance and configuration yourself:

```bash
python scripts/audit_repository.py  --model <local-path> --output reports/repository_audit.json
python scripts/analyze_lineage.py   --candidate <path> --language <path> --vision <path> \
                                    --output reports/lineage_analysis.json
```

## Documentation

| Page | Contents |
|---|---|
| [`reports/repository_audit.md`](reports/repository_audit.md) | Every configuration value, measured |
| [`reports/lineage_analysis.md`](reports/lineage_analysis.md) | Tensor-level provenance proof |
| [`reports/performance_report.md`](reports/performance_report.md) | Latency, throughput, memory |
| [`docs/installation.md`](docs/installation.md) | Environment setup |
| [`docs/inference.md`](docs/inference.md) | Text, image, batch, streaming, long context |
| [`docs/hardware.md`](docs/hardware.md) | VRAM by configuration |
| [`docs/quantization.md`](docs/quantization.md) | 4-bit, 8-bit, GGUF/AWQ/GPTQ status |
| [`docs/evaluation.md`](docs/evaluation.md) | Running and reading evaluations |
| [`docs/troubleshooting.md`](docs/troubleshooting.md) | Start here when something breaks |

## Citation

```bibtex
@software{piko9b_2026,
  title  = {Piko-9b: a composed 9.65B hybrid-attention vision-language model},
  author = {Dexy},
  year   = {2026},
  url    = {https://huggingface.co/Dexy2/Piko-9b},
  note   = {Language backbone derived from deepreinforce-ai/Ornith-1.0-9B (MIT);
            vision tower from Qwen/Qwen3.5-9B (Apache-2.0)}
}
```

Please also cite the upstream models — see [`CITATION.cff`](CITATION.cff).

## License

**Apache-2.0**, with upstream attribution preserved in [`NOTICE`](NOTICE):

| Component | Upstream | Licence |
|---|---|---|
| Language backbone | `deepreinforce-ai/Ornith-1.0-9B` | **MIT** |
| Vision tower + merger | `Qwen/Qwen3.5-9B` | **Apache-2.0** |
| Tokenizer and vocabulary | Qwen3.5 | Apache-2.0 |

MIT is compatible with redistribution under Apache-2.0 **provided the MIT notice is retained** —
that is what `NOTICE` is for. Training-data licensing for the fine-tuning stages could not be
established.

## Contact

Issues and questions: [model discussions](https://huggingface.co/Dexy2/Piko-9b/discussions) or
[github.com/itsdexy/Piko-9b](https://github.com/itsdexy/Piko-9b/issues). Security:
[`SECURITY.md`](SECURITY.md).