reports/final_status.md

# OCC Stack: Final Status

## What Ships

### ✅ Done (production quality)
1. **Impact Oracle** (`oracle/oracle.py`) — Rule-based scoring for code/QA/debate. Detects hidden-test gaming, rewards abstention, Brier-score calibration, compute-cost penalty.
2. **Credit Ledger** (`ledger/ledger.py`) — Non-transferable, decaying, capability-scoped credits with full provenance.
3. **Resource Broker** (`broker/broker.py`) — Capability-based gating with 6 decision types and risk classes.
4. **GRPO/RL Hook** (`rl/reward.py`, `rl/grpo_hook.py`) — TRL-compatible reward function + offline comparator.
5. **Literature Review** (`reports/literature_review.md` + RS-OS paper comparison in `reports/report.md`).
6. **Blog Post** (`reports/blog_post.md`).

### ✅ Done (simulated benchmarks)
1. **Code Compute Allocation** (`benchmarks/benchmark_code.py`) — 52.3% compute savings at iso-accuracy.
2. **Retrieval QA** (`benchmarks/benchmark_retrieval_qa.py`, `_nli.py`) — OCC underperforms, honest negative result.
3. **Debate v2** (`benchmarks/benchmark_debate_v2.py`) — 43.2% savings at iso-accuracy, adversarial containment.
4. **Anti-Gaming** (`eval_runner.py`) — 100% hidden-test detection, credit exhaustion for spam.
5. **Ablations** — 10 ablations measuring each mechanism's contribution.

### ✅ Done (external validation)
- Debate v2 job `69fa273ab745af80fb373135`: **COMPLETED**. Results at `reports/debate_v2_results.json`.
  - OCC: 0.930 accuracy, 2,890 mean compute → **43.2% savings** vs equal turns (5,087)
  - Confidence-weighted voting with adversarial agents: dangerous (amplifies overconfident wrong answers)

### ⚠️ Blocked (real LLM)
- 4 GPU jobs attempted, 4 failed due to model capability or infrastructure:
  1. Qwen-Coder-0.5B: chat template mismatch → all answers wrong
  2. Qwen-Coder-0.5B v2: chat template fixed, model generates code but 0% pass rate (0.5B too weak for HumanEval)
  3. Qwen-Coder-0.5B v3: robust extraction, same 0% pass rate (model capability floor)
  4. StarCoder2-3B: model loading timed out before generation (3B download too slow on provisioned T4)

### ❌ Not Done
- GRPO training (needs GPU + TRL, not attempted due to sandbox rate-limiting)
- Retrieval QA with domain-tuned NLI
- Real LLM results for code benchmark

## Key Numbers

| Benchmark | OCC Result | Best Baseline | Savings |
|-----------|-----------|---------------|---------|
| Code allocation (sim) | 0.780 acc, 8,350 tokens | 0.780 acc, 17,500 tokens | 52.3% |
| Debate v2 (40% adversarial) | 0.930 acc, 2,890 tokens | 0.930 acc, 5,087 tokens | 43.2% |
| Anti-gaming detection | 100% | — | — |
| Retrieval QA | 0.710 acc | 0.790 (RAG+verifier) | OCC loses |

## Honest Bottom Line

OCC works for code allocation and debate — the mechanisms (tiered escalation, credit-based turn allocation) are sound and backed by published literature. The real-LLM validation is the missing piece, blocked by model choice (0.5B too weak) and infrastructure (3B download timing). The system design, anti-gaming properties, and literature positioning are solid enough for a workshop paper.

## Repository

https://huggingface.co/narcolepticchicken/occ-stack