README.md

---
license: apache-2.0
---

# Note

You MUST load the chat template manually in llama-server or LMStudio.

Additionally, I suggest using Zed because it can actually handle Claude's tool calls, which this will use.

# Ornith-1.0-9B — Fable 5 Edition

A fine-tuned variant of Ornith-1.0-9B) blended with gate-pattern knowledge extracted from the [FABLE 5 traces](https://huggingface.co/datasets/Glint-Research/Fable-5-traces) via **HPC Pauli decomposition**.

## Method

### 1. Gate-pattern recovery (Heisenberg-Programming-on-Contour / ContourFuse)

We treat each FABLE 5 conversation as a trajectory through the embedding manifold. From the tokenized assistant turns we build a bigram graph over the top-30k tokens, then decompose each directed edge's *transition matrix* into Pauli components:

$$
\begin{bmatrix}1 & 1 \\ 1 & w_{ij}\end{bmatrix}
= I + X + c_Z(i,j) \cdot Z, \quad
c_Z(i,j) = \frac{1 - w_{ij}}{2}
$$

where $w_{ij} = f_{ij} / \sqrt{f_i \cdot f_j}$ is a frequency-normalized edge weight.

For each source token $i$ we construct a 12288-dimensional **Pauli state vector**:

$$
z_i = \left[\, \frac{E_i}{\tau} \;\Big|\; \frac{\mu_i}{\tau} \;\Big|\; \frac{\varepsilon_i}{\tau} \,\right]
$$

- $E_i$ — embedding of token $i$ (center)
- $\mu_i = \sum_j P(j|i)\, E_j$ — **X-component**: probability-weighted expected next embedding
- $\varepsilon_i = \sum_j c_Z(i,j)\, E_j$ — **Z-component**: Pauli-coupling-weighted neighbor sum
- $\tau = 0.003$ — temperature scaling

The gate projection weight $W_{\text{hpc}}$ is solved via ridge regression:
$E_n \cdot W_{\text{hpc}} \approx z$ where $E_n$ is the normalized embedding matrix.

$W_{\text{hpc}}$ is then norm-scaled to match the original gate_proj weight standard deviation (~0.012) and injected as an **additive correction**:

$$
W_{\text{inj}} = W_{\text{orig}} + \alpha \cdot W_{\text{hpc}}, \quad \alpha = 0.3
$$

Direct replacement of gate_proj fails because transformer layers are co-adapted; additive superposition preserves the original functionality while imprinting FABLE-derived structural priors.

### 2. QLoRA fine-tuning

The HPC-injected model is further trained on FABLE 5 assistant conversations via QLoRA:

- **Base**: HPC-injected Ornith-1.0-9B, loaded in 4-bit NF4 (BitsAndBytes)
- **Target modules**: `gate_proj`, `up_proj`, `down_proj` (all MLP projections)
- **LoRA rank**: 16, alpha: 32, dropout: 0.05
- **Training**: 1 epoch, AdamW (lr=2e-4), linear warmup, gradient accumulation ×8
- **Loss**: 1.23 (PPL ≈ 3.44) on training data

### 3. Merge & GGUF export

LoRA adapters are directly merged into the safetensor shards (element-wise addition of $B \cdot A \cdot \alpha/r$), producing a clean HuggingFace model, then converted to GGUF Q8_0 via `llama.cpp/convert_hf_to_gguf.py` (with `--no-mtp` to exclude the MTP prediction head).

## Files

| File | Description |
|------|-------------|
| `ornith-1.0-9b-fable.q8_0.gguf` | Q8_0 quantized GGUF — ready for llama.cpp / Ollama |
| `model-*.safetensors` | HuggingFace shards (merged, bfloat16) |
| `config.json` | Model configuration |

## Usage (llama.cpp)

```bash
./main -m ornith-1.0-9b-fable.q8_0.gguf \
       -p "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n" \
       -n 128
```

## Dataset

[Glint-Research/Fable-5-traces](https://huggingface.co/datasets/Glint-Research/Fable-5-traces)