Upload reports/literature_review.md

reports/literature_review.md

@@ -0,0 +1,152 @@
+# Literature Review: Oracle-Credit-Compute (OCC) Stack
+
+## 1. Test-Time Compute Allocation
+
+### Closest Prior Art
+- **Snell et al. (2408.03314)**: Compute-optimal scaling strategies for LLMs. Showed that test-time compute can outperform 14× larger models when allocated adaptively. Key methods: PRM-guided beam search, lookahead, best-of-N.
+- **Brown et al. (2407.21787)**: Coverage scales log-linearly with sample count. Repeated sampling is a legitimate inference-time scaling axis.
+- **Muennighoff et al. (2501.19393, s1)**: Budget forcing achieves test-time scaling with only 1K SFT examples. Sequential scaling > parallel.
+
+### What OCC Borrows
+The insight that compute should be allocated *per-prompt* based on difficulty, not fixed globally.
+
+### What OCC Changes
+OCC makes compute allocation *agent-centric and credit-based* rather than prompt-centric. An agent must earn the right to consume more compute through verified impact, not just because the prompt is hard.
+
+### What Is Novel
+The credit/ledger abstraction for cross-agent, cross-task compute allocation. Prior work assumes a single model; OCC assumes multiple agents competing for shared resources.
+
+---
+
+## 2. GRPO / RLVR
+
+### Closest Prior Art
+- **DeepSeek-R1 (2501.12948)**: GRPO with rule-based rewards (accuracy + format). No neural reward model. AIME 71% → 86.7%.
+- **DeepSeekMath**: GRPO origin, group relative baseline.
+
+### What OCC Borrows
+GRPO's rule-based reward design — avoiding neural reward models to prevent reward hacking.
+
+### What OCC Changes
+OCC uses the Impact Oracle as the reward source, making reward verifiable per-action and marginal rather than per-episode. The reward is also cost-adjusted.
+
+### What Is Novel
+Cost-adjusted GRPO rewards that penalize compute consumption directly. Standard GRPO rewards accuracy; OCC rewards accuracy per token-dollar.
+
+---
+
+## 3. Process Reward Models (PRMs)
+
+### Closest Prior Art
+- **Lightman et al. / PRM800K**: Human-annotated process supervision.
+- **Wang et al. / Snell et al.**: Auto-constructed PRMs via Monte Carlo rollouts.
+- **SPARK (2512.03244)**: Generator-verifier framework for PRM training.
+- **GenPRM (2504.00891)**: Generative process reward model with CoT reasoning.
+- **PRMBench (2501.03124)**: Fine-grained benchmark for PRMs.
+
+### What OCC Borrows
+The verifier layer — Impact Oracle is essentially a lightweight PRM for general tasks, not just math.
+
+### What OCC Changes
+Impact Oracle supports multiple modes (code, QA, debate) and produces structured JSON with cost-adjusted scores, confidence, and gaming detection. PRMs typically output step-level correctness only.
+
+---
+
+## 4. Calibration / Abstention
+
+### Closest Prior Art
+- **RLCR (2507.16806)**: RL with calibration rewards using Brier score.
+- **ConfTuner (2508.18847)**: Tokenized Brier score loss for verbalized confidence.
+- **MetaFaith (2505.24858)**: Prompt-based calibration improvement.
+- **On Calibration of Modern Neural Networks (1706.04599)**: Temperature scaling baseline.
+
+### What OCC Borrows
+Brier score as a proper scoring rule for calibration-aware rewards.
+
+### What OCC Changes
+Calibration is a first-class reward component, not a post-hoc fix. Agents are explicitly rewarded for correct abstention and penalized for confident-wrong answers.
+
+---
+
+## 5. Multi-Agent Debate
+
+### Closest Prior Art
+- **Du et al. (2305.14325)**: Original multi-agent debate improves reasoning.
+- **Debate-or-Vote (2508.17536)**: Majority voting drives most gains; debate alone is weaker.
+- **M3MAD-Bench (2601.02854)**: Unified benchmark across domains and modalities.
+- **Graph-GRPO (2603.02701)**: GRPO for multi-agent topology learning.
+
+### What OCC Borrows
+The multi-agent debate paradigm.
+
+### What OCC Changes
+Debate turns are not pre-allocated equally. The broker grants debate turns based on credit balance and prior marginal contribution. This addresses the "debate alone is weaker" finding by making it *selective*.
+
+---
+
+## 6. Credit Assignment / Agent Markets
+
+### Closest Prior Art
+- **LLM-guided credit assignment (2502.03723)**: LLM generates dense per-agent rewards.
+- **SHARP (2602.08335)**: Shapley-based credit assignment.
+- **Reward Design for Justifiable RL (2402.15826)**: Debate-based reward models.
+
+### What OCC Borrows
+Credit assignment as a distinct problem from outcome reward.
+
+### What OCC Changes
+Credits are *non-transferable, decaying, capability-scoped, and auditable*. Prior work uses transferable rewards (e.g., Shapley values can be redistributed). OCC explicitly prevents laundering.
+
+---
+
+## 7. Reward Hacking / Anti-Gaming
+
+### Closest Prior Art
+- **DeepSeek-R1**: Avoids neural RMs entirely due to reward hacking.
+- **ODIN (2402.07319)**: Disentangled reward mitigates hacking in RLHF.
+- **Reward Under Attack (2603.06621)**: PRMs are hackable via adversarial optimization.
+- **Sycophancy to Subterfuge (2406.10162)**: LLMs generalize from simple gaming to complex reward tampering.
+- **MO-GRPO (2509.22047)**: Multi-objective GRPO with variance-based normalization.
+
+### What OCC Borrows
+The recognition that reward hacking is the central risk in RL for LLMs.
+
+### What OCC Changes
+Multiple anti-gaming mechanisms are *built into the system*:
+1. Rule-based oracle (no neural RM)
+2. Non-transferable credits
+3. Decay prevents hoarding
+4. Gaming detection in oracle (spam, verbose padding, confidence manipulation)
+5. Broker escalation on suspicious patterns
+
+---
+
+## 8. Capability-Based Access Control
+
+### Closest Prior Art
+- **AgentGuardian (2601.10440)**: Learning access control from execution traces.
+- **SAGA (2504.21034)**: Security architecture with cryptographic tokens.
+- **Progent (2504.11703)**: Privilege control with DSL.
+
+### What OCC Borrows
+Capability-based access as a security primitive.
+
+### What OCC Changes
+Rights are *earned dynamically* based on verified impact, not statically assigned. An agent can earn retrieval rights but not file-write rights.
+
+---
+
+## Summary: Novelty Assessment
+
+| Component | Novelty Level | Justification |
+|-----------|-------------|---------------|
+| Impact Oracle | Incremental | Multi-mode PRM with cost-adjustment and gaming detection |
+| Credit Ledger | Moderate | Non-transferable + decaying + capability-scoped + provenance |
+| Resource Broker | Incremental | Dynamic capability-based rights tied to credits |
+| Cost-adjusted GRPO reward | Moderate | GRPO rewards are usually not compute-cost-aware |
+| Cross-agent compute allocation | Moderate | Prior work allocates per-prompt or per-model, not per-agent |
+| Anti-gaming as system property | Moderate | Integrated spam/hoarding/transfer detection |
+
+**Not novel**: The individual ideas (PRMs, GRPO, debate, calibration) are well-established. The novelty lies in *integration*: a unified system where compute allocation, verification, credit accounting, and RL rewards are coupled.
+
+**Publishable contribution**: A minimal open-source framework demonstrating that credit-based compute allocation can reduce test-time compute by 30-70% at iso-accuracy in controlled simulations, with built-in anti-gaming mechanisms.