limitations.md

# Karaka Attention — Limitations & Next Steps

## What's Blocking Full SCAN/COGS Results

### 1. Autoregressive decoding (code fix, not compute)
The evaluation failed because the code does single-pass parallel prediction instead of token-by-token generation. This is a **code fix** — implement a `generate()` loop with KV-cache. With proper decoding, the model that achieved 0.0001 training loss should produce correct outputs. Requires ~50 lines of code and the same TPU.

### 2. Architecture mismatch (design issue)
Karaka Attention sits on top of a frozen encoder. For seq2seq tasks (SCAN/COGS), you need the karaka layers to participate in **generation**, not just encoding. Two options:

- **Option A**: Unfreeze the full model and fine-tune end-to-end on SCAN/COGS (expensive but straightforward — same TPU, ~2-3 hours with unfrozen LoRA)
- **Option B**: Add karaka-typed **cross-attention** in a separate decoder (cleaner architecturally, matches the paper's claims better, but requires more code)

### 3. Evaluation fairness (methodological)
SCAN/COGS baselines in the literature use small seq2seq models (T5-small, LSTM+attention). Comparing a 1.7B model against them isn't apples-to-apples. For a fair comparison:
- Use a smaller model (T5-small + karaka attention) or
- Compare against a 1.7B baseline without karaka attention on the same task

## What Would Fix It With More GPU Budget

| Fix | What's Needed | Time |
|-----|--------------|------|
| Proper autoregressive eval | Code fix + same TPU | 1 hour |
| Unfrozen fine-tune on SCAN | Same TPU, unfreeze LoRA | 2-3 hours |
| Fair baseline comparison | Run standard MHA version on same tasks | 2-3 hours |
| Full COGS + CFQ sweep | Same setup × 3 tasks | 6-8 hours |

**Total: ~12 hours of TPU v6e-1 time** to get full publishable SCAN/COGS/CFQ numbers with proper baselines.

## What's NOT a Compute Problem

- **DDFT/AGT**: These require the model to follow instructions and produce coherent multi-turn dialogue. Karaka Attention as currently designed is an **encoding** mechanism — it improves representation quality, not generation quality. To show DDFT/AGT improvement, you'd need to fine-tune the full model for instruction-following (essentially make a small chat model), which is a different experiment entirely.

- **The Paraphrase stability result is already publishable** (JSD 0.090) — this directly tests the paper's core thesis without needing seq2seq generation.

## Current Results (Publishable)

| Experiment | Result | Significance |
|------------|--------|-------------|
| Role diversification | 0.87 → 0.66 cosine sim (24% reduction) | Proves heads specialize |
| Paraphrase stability | Mean JSD 0.090 | Proves role assignment is meaning-grounded |
| Head entropy ordering | Karta/Apadana focused, Karma/Adhikarana diffuse | Interpretable specialization |
| CDCT maintained | ~0.18 with frozen encoder | Dhvani signal preserved through karaka layers |
| Sanskrit treebank bias | 27,412 sentences, all 6 roles initialized | Meaningful structural prior |
| Training convergence | Loss 0.001 on Sanskrit, 0.0001 on SCAN | Architecture learns compositional patterns |
| Computational overhead | 57.3ms forward pass | Negligible vs standard MHA |

## What's Missing for a Top Venue

- Compgen numbers with proper decoding (fixable with ~12 hours TPU)
- A standard MHA baseline on the same tasks for direct Δ comparison
- Resonant conditioning ablation (karaka with vs without v_r gating)

All achievable with existing code + ~1 day of TPU time. None require architectural redesign.