Exp 164: upload guided-decoding-adapter (Exp 137 artifact)

README.md

@@ -0,0 +1,197 @@
+---
+tags:
+  - energy-based-model
+  - guided-decoding
+  - constraint-satisfaction
+  - jax
+  - carnot
+license: apache-2.0
+---
+
+> **Research Artifact — Not Production-Ready**
+>
+> Real-model validation is pending (Exp-111). Exp-110 results use a mock LLM
+> with deterministic error injection. The constraint checker works correctly
+> (0.006 ms/check on CPU); the guidance logic is unvalidated on live models.
+
+# guided-decoding-adapter
+
+Energy-guided decoding adapter for any HuggingFace causal LM.
+
+Attaches Carnot's constraint energy pipeline to the token generation loop.
+Each token step runs a constraint violation check on the text generated so far;
+violating tokens are penalised by subtracting `alpha × violation_count` from
+all logits before sampling.
+
+## How It Works
+
+```
+prompt → encode → [forward pass → check constraints → penalise logits → sample] × N → text
+```
+
+The constraint checker (`AutoExtractor`) detects violations across four domains:
+
+| Domain | Constraint types |
+|--------|-----------------|
+| Arithmetic | addition, multiplication, bounds |
+| Code | type checks, return types, initialisation |
+| Logic | implication, exclusion, disjunction, negation, universal |
+| Natural language | NL consistency |
+
+Energy is a plain violation count (not a calibrated probability).  The penalty
+is applied uniformly across the vocabulary — token ranking is preserved while
+overall entropy increases, discouraging the model from continuing down a
+constraint-violating path.
+
+## Latency Profile
+
+From Exp-102 (CPU, JAX_PLATFORMS=cpu, 1000-iteration benchmark):
+
+| Measurement | Value |
+|---|---|
+| Constraint check p50 | 0.006 ms |
+| Constraint check p99 | 0.034 ms |
+| Extraction p50 | 0.276 ms |
+| Per-token budget fraction | 0.04% of 20 ms/token |
+| Verdict | **Fits in real-time generation budget** |
+
+## Usage
+
+```python
+from carnot.inference.guided_decoding import GuidedDecoder
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load model (any HF causal LM)
+model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B")
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")
+model.eval()
+
+# Load adapter from local directory or HuggingFace Hub
+decoder = GuidedDecoder.from_pretrained("Carnot-EBM/guided-decoding-adapter")
+
+# Generate with constraint guidance
+result = decoder.generate(model, tokenizer, "What is 47 + 28?")
+print(result.text)
+print(f"Energy checks: {result.energy_checks}, final energy: {result.final_energy}")
+```
+
+### Override defaults
+
+```python
+decoder = GuidedDecoder.from_pretrained(
+    "Carnot-EBM/guided-decoding-adapter",
+    alpha=1.0,           # stronger guidance
+    check_every_k=5,     # check every 5 tokens (faster, less precise)
+    energy_threshold=0.5 # only penalise when violations > 0.5
+)
+```
+
+### Load from a local export directory
+
+```python
+decoder = GuidedDecoder.from_pretrained("./exports/guided-decoding-adapter")
+```
+
+## Return Value
+
+`generate()` returns a `GuidedDecodingResult`:
+
+| Field | Type | Description |
+|---|---|---|
+| `text` | `str` | Generated text (prompt excluded) |
+| `tokens_generated` | `int` | Number of tokens produced |
+| `energy_checks` | `int` | Times constraint check ran |
+| `mean_penalty` | `float` | Average logit penalty applied |
+| `latency_seconds` | `float` | Wall-clock time |
+| `final_energy` | `float` | Violation count after last check |
+
+## Constraint Weights
+
+Default weights are stored in `constraint_weights.safetensors`.  Load and inspect:
+
+```python
+from safetensors.numpy import load_file
+weights = load_file("constraint_weights.safetensors")
+print(weights["all_weights"])   # shape (12,) float32
+print(weights["default_alpha"]) # [0.5]
+```
+
+## Compatible Models
+
+Tested target models (Exp-110):
+- `Qwen/Qwen3.5-0.8B`
+- `google/gemma-4-E4B-it`
+
+Any HuggingFace `AutoModelForCausalLM` with `.logits` output should work.
+The adapter does not modify model weights.
+
+
+## Benchmark Results (Exp-138 & Exp-140)
+
+> **Note — Simulated Inference**: All benchmark numbers below were produced
+> with a *simulated* (mock) LLM, not a real transformer model.  The constraint
+> checker and logit-penalty logic are real; the generation loop uses a
+> deterministic stand-in.  Live-model E2E validation is pending (Exp-111).
+
+### Accuracy (Exp-138, n=200/50/100, simulated inference)
+
+| Dataset | Baseline | Guided | Guided+Verify-Repair | Delta (guided) |
+|---------|----------|--------|----------------------|----------------|
+| GSM8K (math) | 55.5% | 62.5% | 65.0% | **+7.0%** |
+| HumanEval (code) | 100.0% | 100.0% | — | **+0.0%** |
+| TruthfulQA | 55.0% | 56.0% | 61.0% | **+1.0%** |
+
+### Latency (Exp-138, n=485 samples, CPU)
+
+| Metric | Value |
+|--------|-------|
+| Constraint-check p50 | 0.0719 ms |
+| Constraint-check p99 | 0.1275 ms |
+
+### Latency — KAN Projection Mode (Exp-140, batch=1, CPU)
+
+| Operation | p50 | p99 |
+|-----------|-----|-----|
+| Logit projection (energy gradient) | 0.077 ms | 0.271 ms |
+| Total per-token (grad + projection) | 0.405 ms | 0.924 ms |
+
+Exp-140 pass criterion: total p50 < 5 ms — **PASSED**
+(actual 0.4054 ms vs 5.0 ms threshold).
+
+## Installation
+
+```bash
+pip install carnot
+```
+
+Requires Python 3.11+.  See [pypi.org/project/carnot](https://pypi.org/project/carnot)
+for the full package including the verify-repair pipeline.
+
+## Limitations
+
+1. **Simulated inference benchmark**: Exp-138 and Exp-140 used a mock LLM.
+   Numbers show constraint-checker and logit-penalty overhead, not end-to-end
+   accuracy on real models.  Treat accuracy deltas as directional, not final.
+2. **No KV-cache**: Full forward pass every token.  Keep `max_tokens < 256`.
+3. **Uniform penalty**: Adjusts entropy across the whole vocabulary; does not
+   steer towards specific correct tokens.
+4. **Energy is a violation count**: Not a calibrated probability.  High `alpha`
+   + many violations → very flat distribution (model may repeat or stall).
+5. **Min-text guard**: `AutoExtractor` skips texts < 5 chars (early tokens).
+6. **Live-model E2E pending**: Exp-111 validation against Qwen/Gemma not done yet.
+
+## Spec
+
+- REQ-VERIFY-001: Constraint energy computed from partial text at each step.
+- SCENARIO-VERIFY-004: Energy penalises logits before sampling.
+
+## Citation
+
+```bibtex
+@misc{carnot2026guided,
+  title  = {Carnot Guided Decoding Adapter},
+  author = {Carnot-EBM},
+  year   = {2026},
+  url    = {https://github.com/Carnot-EBM/carnot-ebm}
+}
+```

config.json

@@ -0,0 +1,66 @@
+{
+  "adapter_type": "guided_decoding",
+  "carnot_version": "0.1.0",
+  "spec": ["REQ-VERIFY-001", "SCENARIO-VERIFY-004"],
+
+  "constraint_types": [
+    "arithmetic",
+    "type_check",
+    "return_type",
+    "return_value_type",
+    "bound",
+    "initialization",
+    "implication",
+    "exclusion",
+    "disjunction",
+    "negation",
+    "universal",
+    "nl_consistency"
+  ],
+
+  "default_weights": {
+    "arithmetic":       1.0,
+    "type_check":       1.0,
+    "return_type":      1.0,
+    "return_value_type":1.0,
+    "bound":            1.0,
+    "initialization":   0.8,
+    "implication":      0.9,
+    "exclusion":        0.9,
+    "disjunction":      0.7,
+    "negation":         0.8,
+    "universal":        0.7,
+    "nl_consistency":   0.5
+  },
+
+  "default_alpha": 0.5,
+  "default_check_every_k": 1,
+  "default_energy_threshold": 0.0,
+  "default_max_tokens": 256,
+  "default_temperature": 1.0,
+
+  "compatible_model_families": [
+    "Qwen/Qwen3-0.6B",
+    "Qwen/Qwen3.5-0.8B",
+    "Qwen/Qwen2.5-*",
+    "google/gemma-4-*",
+    "meta-llama/Llama-*",
+    "mistralai/Mistral-*"
+  ],
+
+  "latency_profile": {
+    "constraint_check_p50_ms": 0.006,
+    "constraint_check_p99_ms": 0.034,
+    "extraction_p50_ms": 0.275,
+    "per_token_overhead_budget_fraction": 0.0004,
+    "source_experiments": ["Exp-102", "Exp-110"]
+  },
+
+  "known_limitations": [
+    "AutoExtractor min-text guard (< 5 chars) skips early tokens",
+    "No KV-cache: full forward pass every token — use max_tokens < 256 for speed",
+    "Uniform logit penalty preserves token ranking but does not steer vocabulary selection",
+    "Energy is a violation count, not a calibrated probability",
+    "Real-model validation pending (Exp-111); Exp-110 used MockArithmeticLLM"
+  ]
+}

constraint_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:070451f8e88383b12f8957c8f703b5eec2f99673f8e302795d14f0fb56d4222e
+size 1192

example.py

@@ -0,0 +1,44 @@
+"""Minimal usage example for the guided-decoding-adapter.
+
+Run from the carnot repo root:
+    JAX_PLATFORMS=cpu python exports/guided-decoding-adapter/example.py
+"""
+
+import os
+os.environ.setdefault("JAX_PLATFORMS", "cpu")
+
+from unittest.mock import MagicMock
+import torch
+
+from carnot.inference.guided_decoding import GuidedDecoder
+
+# Load adapter from this directory (local usage)
+# To load from HuggingFace Hub swap the path for the repo ID:
+#   decoder = GuidedDecoder.from_pretrained("Carnot-EBM/guided-decoding-adapter")
+decoder = GuidedDecoder.from_pretrained("exports/guided-decoding-adapter")
+
+# --- Minimal mock model and tokenizer (no GPU / model download needed) ---
+# Replace these two blocks with real HF model/tokenizer for production use:
+#   model     = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B")
+#   tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")
+step = [0]
+
+def _forward(input_ids):
+    logits = torch.zeros(1, input_ids.shape[1], 10)
+    logits[0, -1, 1 if step[0] >= 3 else 0] = 10.0  # EOS after 3 tokens
+    step[0] += 1
+    out = MagicMock(); out.logits = logits; return out
+
+model = MagicMock()
+model.side_effect = _forward
+model.parameters = MagicMock(return_value=iter([torch.zeros(1)]))
+
+tokenizer = MagicMock()
+tokenizer.eos_token_id = 1
+tokenizer.encode = MagicMock(return_value=torch.tensor([[2, 3, 4]]))
+tokenizer.decode = MagicMock(side_effect=lambda ids, **kw: "" if ids.item() == 1 else "A")
+
+result = decoder.generate(model, tokenizer, "What is 47 + 28?")
+print("Generated:", result.text)
+print(f"Tokens: {result.tokens_generated}  Checks: {result.energy_checks}  "
+      f"Mean penalty: {result.mean_penalty:.3f}  Latency: {result.latency_seconds*1000:.1f}ms")