docs/ARCHIVE_AND_COMPARISON_PLAN.md

# Bonsai Archive And Comparison Plan

## Public Archives

| Artifact | Hub repository | Status | Local retention after verification |
| --- | --- | --- | --- |
| MLX BF16 config-repaired baseline | `TiGa-RCE/Bonsai-27B-MLX-BF16-Config-Repaired` | Reproducible archival baseline, not locally runnable | Can be removed after Hub hash verification and redownloaded when a conversion run needs it |
| oQ2e S32 smoke | `TiGa-RCE/Bonsai-27B-oQ2e-S32-Smoke` | Public pipeline/calibration evidence, not a quality release | Can be removed after Hub hash verification; retain only if a local oQe regression test needs it |

Do not publish the generic affine Q2 artifact as a model release. It is a
negative control: it loaded but failed direct generation and deterministic
screens. Keep its small logs and metadata; delete its 7.9 GB weights only after
the user explicitly approves a cleanup list.

## Upstream Models: Link, Do Not Mirror

| Upstream artifact | Role | Necessary now? |
| --- | --- | --- |
| `prism-ml/Bonsai-27B-gguf` Q1_0 | Native 1-bit reference | Yes, already cached and used through Prism's runner |
| `prism-ml/Bonsai-27B-mlx-1bit` | Native MLX Q1 reference | Keep metadata; runtime integration remains a separate branch |
| `peppi314/Bonsai-27B-dSpark-MLX-4bit` | Speculative-decoding sidecar, not a standalone model | No additional copy or mirror needed |
| `prism-ml/Ternary-Bonsai-27B-gguf` Q2_0 | Official retrained ternary quality-oriented comparison | Download later, only when the native Q1 versus ternary screen is scheduled |
| `prism-ml/Ternary-Bonsai-27B-mlx-2bit` | Official MLX ternary comparison | Download later, only after the compatible Prism MLX runtime is established |

## Controlled MLX Comparison Matrix

The next local derivatives should use the public BF16 archive as the sole
source and must have the same tokenizer, prompts, fixed decoding settings, and
evaluation scripts:

| Variant | Generation route | Comparison purpose | Promotion gate |
| --- | --- | --- | --- |
| MLX Q4 | Uniform affine MLX | Standard low-bit baseline | Structural load, direct generation, 100 HellaSwag + 100 ARC-Challenge |
| oQ4 | oQe MLX | Tests calibration benefit over Q4 | Same screens and a predeclared memory/latency measurement |
| MLX Q8 | Uniform affine MLX | High-fidelity local reference when memory permits | Same screens; no claim that it is FP16 |
| oQ8 | oQe MLX | Tests whether oQe retains any benefit at a high bit-width | Same screens and runtime measurement |
| Bonsai Q1 native | Prism llama.cpp runtime | Binary operating point | Same semantic screens, throughput recorded separately by runtime |
| Ternary Bonsai Q2 native | Prism runner | Official retrained ternary operating point | Same semantic screens, clearly labeled as a different trained checkpoint |

### Important Fairness Boundaries

- Q4 versus oQ4 and Q8 versus oQ8 are direct quantization-method comparisons.
- Q1 versus official ternary is an operating-point comparison, not proof that
  one quantizer is better: the official ternary checkpoint has different
  trained ternary weights.
- Accuracy and formatting screens can be reused across all compatible models.
  Throughput figures cannot be compared across oMLX and Prism llama.cpp without
  recording each runtime's context size, cache policy, decoding parameters, and
  hardware path.

## Prerequisite Before New oQ Runs

Repair the imatrix collector so it checkpoints and resumes. The current full
run stopped after 44 samples without preserving a reusable partial cache. Do
not start oQ4 or oQ8 quality calibration until that collector has an atomic
checkpoint and a resume test.