Orchidia V0 — Structured Decision Model on Qwen3.5-4B (Experimental Baseline)

Experimental V0 baseline. Not production-ready. Research artifact for Clef-inspired structured decisions.

Latest (2026-10-28): the rank-16 LoRA variant v0_16k_lora_r16 is complete and passes all three pre-registered gates — natural-weighted OOD 87.52%, validation 87.88%, painting-geometry|noul|K=2 88.59%. It is now the best model in this repo and requires adapter_final/. Fresh start here: HANDOFF_2026-10-28.md → PROJECT_STATUS.md → experiments/v0_16k_lora_r16/RESULTS.md.

Orchidia is a Clef-inspired structured decision model. It takes a STATE (text; image/video-ready) plus a DYNAMIC DECISION SCHEMA (arbitrary questions, types, and choices) and produces structured probability distributions for every question — not free-text generation.

• Backbone: Qwen/Qwen3.5-4B (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a).
• V0 uses a FROZEN Qwen3.5-4B backbone. Only the Orchidia DynamicDecisionHead was trained. Full Qwen weights are NOT duplicated here; this repo contains the 10.5 MB head (2,628,609 params) plus configs/metrics/code to reconstruct V0 on top of the base model.
• Dataset: ZefanCai/Open-Jev (release-v2-redistributable). V0 trained on diverse/shuffled experiments at 2k, 8k, 16k (all frozen backbone, head-only).
• Dynamic: any number of questions, any K per question. Output example: {"intent": {"refund": 0.43, ...}, "urgency": {...}, "escalate": {...}}.
• Release status: experimental V0 baseline + first LoRA variant. Frozen 16k completed 2026-10-24; LoRA r16 completed 2026-10-28 with all gates passing. Next step: the one-time primary holdout evaluation (not yet run). 32k is not approved.

1. Model architecture

State + dynamic schema
→ Qwen3.5-4B trunk (Qwen3_5Model, hidden 2560, frozen)
→ question-conditioned representation (question text inside each input)
→ DynamicDecisionHead: LayerNorm(2560) + Linear(2560→1024) + GELU + Dropout(0.1) + Linear(1024→1)
→ per-option logits [sum(K)] (chunked forward, chunk=8)
→ per-question softmax → {qid: {choice: prob}}

There are NO fixed question classifiers. One shared scorer handles unseen questions and variable K. Loss: soft cross-entropy/KL for all kinds (choice/noul/score targets are distributions, including soft like [0,0.5,0.5]); ordinal EMD logged for score. Text-only V0 path; image/video hook preserved in code (explicit error until media batching is wired, no silent drop).

2. Training history

Experiment	Data	Steps	max_length	Precision	Checkpoint	Status
2k diverse	2,000 shuffled	~120	256	fp16	checkpoints/v0_2k_shuf_final/	Preserved baseline
8k diverse	8,000 shuffled	500	256	fp16	checkpoints/v0_8k_shuf/	Preserved
16k diverse	16,000 shuffled	1,000	512	bf16	checkpoints/v0_16k_shuf/	Best frozen baseline
LoRA r16 16k	16,000 shuffled	1,000	512	bf16	checkpoints/v0_16k_lora_r16/	Best overall (requires adapter)

All runs: frozen backbone, head-only, AdamW 1e-4, batch 2 × grad_accum 8 (effective 16), seed 0. 2k→8k warm-start (head-only); 8k→16k warm-start (head-only, interrupted at step 50, exact-resumed). The 16k run completed on A100 80GB (2026-10-24); final training val loss 0.6008. The LoRA run completed on A100 80GB (2026-10-28); final training val loss 0.2977. It warm-starts from the 8k head and trains 23.86M params (2.63M head + 21.23M adapters on q,k,v,o,gate,up,down). Intermediate step checkpoints were not uploaded.

Context-length note: 2k/8k trained at 256 tokens; 16k trained at 512. The 2k/8k 512 scores in comparisons are evaluation-only re-scores. 16k-vs-8k/2k deltas therefore confound data scale with training context.

3. Dataset information

ZefanCai/Open-Jev — typed decisions: kind ∈ {choice→categorical, noul→boolean, score→ordinal}, target distribution sums to 1.0. Rows grouped by group_id into {state, schema, labels}; splits are group-separated (verified 0 leakage train/val/test/ood). Counts: train 79116 / val 3723 / test 10356 / ood 15701. Do NOT redistribute Open-Jev data — this repo ships only Orchidia weights/configs/metrics. Use only state/question/kind/options as inputs (never target/metadata).

Source-bias warning (important): release-v2 is source-ordered. select(range(N)) without shuffle yields 100% customer-control-v1. V0 release uses deterministic shuffle(seed).select(N). Never use first-N.

4. Evaluation results (512-token protocol)

First-512 (source-ordered; NOT representative of OOD)

Model	Validation	Test	OOD	Calibration
2k	70.3%	87.5%	86.5%	74.6%
8k	73.2%	87.5%	88.1%	77.5%
16k	77.0%	92.0%	86.5%	81.1%

Shuffled-512 (seed 1, hard K up to 16)

Model	Validation	Test	OOD	Calibration
2k	63.1%	68.2%	71.9%	71.3%
8k	67.2%	67.4%	68.9%	73.6%
16k	74.4%	74.0%	70.7%	76.0%

Stratified analysis samples (deterministic, no overlap with historical 512)

Model	OOD strat 4096 (raw)	OOD strat (natural-weighted)	Val strat 2048 (raw)	Val strat (natural-weighted)
2k	72.2%	73.0%	66.5%	65.7%
8k	68.5%	69.2%	70.1%	69.3%
16k	69.0%	69.6%	74.3%	73.4%

Key finding: 16k is a clear win on validation (+3.8pp vs 8k, +7.2pp vs 2k, natural-weighted) but does not improve natural-prevalence-weighted OOD accuracy over the 2k baseline (16k 69.6% vs 2k 73.0%). The gap is driven by a single failure mode: painting-geometry-v1|noul|K=2 (foreground detection in synthetic images) where 16k scores 46.6% vs 2k 70.1% (−23.5pp). This source accounts for 44% of full-OOD row mass.

Calibration (loss/ECE/Brier) improves monotonically: 16k has best OOD loss (0.640) and Brier (0.396). The model is better calibrated but does not solve the domain-adaptation problem on painting-geometry-v1.

Full analysis: analysis_work/results/analysis/analysis_report.md. Code snapshot: code/v0_16k_shuf_v2/.

5. Known limitations

• V0 head-only, 16k diverse (~1,000 steps at 512): under-trained for full generalization; expect gains from backbone adaptation.
• Text-only V0; images/videos raise explicit error (hook exists, batching not wired).
• max_length 512 truncates very long states; 16k is the only model trained at 512.
• Calibration split unused; ECE/Brier reported only, no recalibration.
• OOD in Open-Jev is source-specific (reserved wording/layouts/families), not universal domain shift.
• Frozen backbone limits domain adaptation (see painting-geometry-v1 regression).
• Historical first-512 OOD/test slices are 100% customer-control-v1; do not quote as general OOD.

6. Exact base-model dependency

• Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (full SHA; short 851bf6e806ef).
• transformers>=5.10 required (16k session used 5.20.2).
• torch, datasets, huggingface_hub, scikit-learn, scipy. See requirements.txt.
• This repo does NOT contain Qwen weights. Reconstruction downloads Qwen from HF at inference time.

7. Installation instructions

pip install torch "transformers>=5.10" datasets huggingface_hub scikit-learn pyyaml tqdm
# Optional (H100 speed): pip install flash-attn  # then --attn flash_attention_2
# Optional for frozen models; REQUIRED for v0_16k_lora_r16: pip install peft accelerate
git clone https://huggingface.co/Seedyai/Orchidia && cd Orchidia
# Head downloads with the repo (checkpoints/v0_16k_shuf/head_final.pt).
# Qwen3.5-4B (~8GB) downloads on first inference via transformers.

No Open-Jev download needed for inference. For training/eval (local MiniClef-4B only): pip install -r requirements.txt and load_dataset("ZefanCai/Open-Jev", "release-v2-redistributable").

8. Inference example

Dynamic schema — any Q, any K (do not hard-code counts):

from inference.example import DynamicDecisionHead, predict_structured
# see inference/example.py for full runnable script
state = {"text": "Customer received the wrong product and wants a refund."}
schema = {"questions": [
  {"id": "intent", "type": "categorical", "question": "What is the customer intent?",
   "choices": ["refund", "shipping", "complaint", "other"]},
  {"id": "urgency", "type": "categorical", "question": "How urgent?",
   "choices": ["low", "medium", "high"]},
  {"id": "escalate", "type": "boolean", "question": "Should we escalate?",
   "choices": ["yes", "no"]},
]}
out = predict_structured(backbone, tokenizer, head, state, schema)
# {"intent": {"refund": 0.43, ...}, "urgency": {...}, "escalate": {...}}

Run: python inference/example.py --head checkpoints/v0_16k_shuf/head_final.pt --dtype bf16 (T4: --dtype fp16). Verified: Q=3 K=4/3/2 probs sum to 1.

9. Checkpoint loading example

Two formats, same weights:

import torch
from transformers import AutoProcessor, Qwen3_5Model
from inference.example import DynamicDecisionHead, load_head_state_dict

REV = "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-4B", revision=REV, trust_remote_code=True)
tokenizer = processor.tokenizer
backbone = Qwen3_5Model.from_pretrained("Qwen/Qwen3.5-4B", revision=REV, dtype=torch.bfloat16, trust_remote_code=True)
for p in backbone.parameters(): p.requires_grad = False
head = DynamicDecisionHead(hidden_size=2560)

# Either format works:
state, fmt = load_head_state_dict("checkpoints/v0_16k_shuf/head_final.pt")  # wrapped
# state, fmt = load_head_state_dict("checkpoints/v0_16k_shuf/hf/decision_head.pt")  # raw state_dict
head.load_state_dict(state)
head.eval()

The head has 2,628,609 params in 6 fp32 tensors. The backbone is frozen.

LoRA checkpoint (v0_16k_lora_r16) does not use the head-only loader above. Use the adapter-aware loader in the LoRA code snapshot:

from src.model import MiniClef4B
model = MiniClef4B.load_for_eval(
    head_ckpt="checkpoints/v0_16k_lora_r16/head_lora_final.pt",
    adapter_dir="checkpoints/v0_16k_lora_r16/adapter_final",
    dtype="bf16", max_length=512,
)

head_lora_final.pt alone is not enough, and hf/decision_head.pt is invalid for this checkpoint. See checkpoints/v0_16k_lora_r16/hf/ADAPTER_REQUIRED.md.

10. Reproducible code snapshot

code/v0_16k_shuf_v2/ is the versioned, self-contained code snapshot for the completed 16k experiment. It includes:

• Exact training code (src/) with restored preprocessing.py and __init__.py
• Machine-written run configs (configs/v0_16k_shuf_run_config.yaml, ..._resume.yaml)
• Cleaned human-readable config (configs/v0_16k_shuf.yaml) with completion metadata
• Updated inference example (inference/example.py) defaulting to 16k head, pinned revision, both formats
• Analysis code (analysis/) with deterministic samples, evaluation drivers, verification, and report
• MANIFEST.json, CHANGES.md documenting differences from the original snapshot

See code/v0_16k_shuf_v2/README.md for reproduction instructions.

11. Future roadmap

1. LoRA adaptation of Qwen3.5-4B — completed 2026-10-28, all pre-registered gates pass. See experiments/v0_16k_lora_r16/RESULTS.md.
2. Frozen-vs-LoRA comparison — completed (identical data/seed/steps, six scenarios, paired tests).
3. Next: one-time primary holdout evaluation of the fixed LoRA checkpoint (requires explicit approval; not yet run).
4. Optional verification before trusting the exact magnitude: adapter-merge re-score and a second-seed run.
5. 32k remains not approved. If more scale is pursued, prefer LoRA at 32k over frozen 32k; the frozen scaling curve was already flat-to-negative on OOD.
6. Further architecture/data improvements only after the above: longer context, per-source stratification, ordinal-aware loss weight for score, media path wiring.

12. Current status (2026-10-28) — start here for continuation

HANDOFF_2026-10-28.md has the restore/verify commands. PROJECT_STATUS.md is the canonical handoff. Read them first; this section is a pointer only.

• Completed: 2k diverse, 8k diverse, 16k diverse (2026-10-24), and LoRA r16 16k (2026-10-28). All preserved.
• LoRA r16 passes all three pre-registered gates: natural-weighted OOD 87.52% (≥ 73.0%), painting-geometry|noul|K=2 88.59% (≥ 60.0%), validation paired CI lower bound +11.33 pp (≥ −1.0 pp). The targeted frozen-16k regression is fixed.
• Best model: checkpoints/v0_16k_lora_r16/head_lora_final.pt + checkpoints/v0_16k_lora_r16/adapter_final/. Head-only inference from hf/decision_head.pt is invalid for this checkpoint.
• The primary holdout (9,363 rows / 808 groups) was never loaded (holdout_touched=false). It is unspent and eligible for exactly one evaluation after explicit approval.
• Frozen subsets remain the control set: 16k frozen natural-weighted OOD 69.6%, 2k frozen 73.0%. LoRA is +17.96 pp over frozen 16k on the same rows.
• Full metrics: checkpoints/v0_16k_lora_r16/analysis/ (summary, per-row results, gate report) and checkpoints/v0_16k_shuf/metrics/.
• Protocol warning: 2k/8k trained at 256, 16k and LoRA at 512; frozen-vs-frozen deltas still confound data scale with context length.
• 32k is not approved or launched.

See PROJECT_STATUS.md (canonical handoff: experiments, decisions, bugs, workflow). Local dev repo MiniClef-4B retains src/, scripts/, full logs; this HF repo is the minimal reconstructable release.

---

License/attribution: Orchidia code/configs/metrics as released here; Qwen3.5-4B weights follow their upstream license; Open-Jev data NOT included (see its repo for license; CC0 synthetic + third-party notes). This V0 is experimental.