FULL_PIPELINE_DETAILED.md

# 🚀 FULL PIPELINE: 3-Week Detailed Implementation Plan

**Goal:** 42-44% → 60-70%+ (SOTA-competitive)

**Status:** Approved for full implementation with unlimited LLM API

---

## 📅 **WEEK 1: Language & Data (Day 1-7)**

### **Day 1: Language Embeddings Setup**

**Morning (4h):**
```bash
# Install dependencies
pip install sentence-transformers openai anthropic

# Test embedding generation
python -c "
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-mpnet-base-v2')
emb = model.encode(['pick the cube'])
print(f'Embedding shape: {emb.shape}')  # Should be (1, 768)
"
```

**Afternoon (4h):**
- Create instruction embedding script
- Generate embeddings for all 3.5K groups
- Save to disk (cache for fast loading)

**Files to create:**
- `dovla_cil/utils/language_embeddings.py`
- `scripts/generate_instruction_embeddings.py`

---

### **Day 2: Modify Architecture for Language**

**Morning (4h):**
- Update `DoVLATransformer` to accept lang_dim=768
- Modify cross-attention to fuse obs+lang
- Test forward/backward with language

**Afternoon (4h):**
- Update training dataset to load embeddings
- Modify collate_fn for language batching
- Test full training loop

**Files to modify:**
- `dovla_cil/models/dovla_transformer.py`
- `scripts/train_dovla_transformer.py`

---

### **Day 3-4: Retrain with Language (48h)**

**Submit 3 jobs:**
```bash
sbatch scripts/slurm/train_transformer_lang.sbatch  # 3 seeds
```

**Monitor training:**
- Val top-1 should be 65-70% (vs 63% without lang)
- Losses should decrease smoothly
- Expected final: 50-55% selected success

**While training runs:**
- Prepare LLM data augmentation code
- Setup OpenClaude API integration

---

### **Day 5: LLM Data Augmentation**

**Morning (4h):**
- OpenClaude API integration
- Synthetic instruction generation

```python
def generate_synthetic_instructions(state_desc, num=5):
    prompt = f"""
    Given robot state: {state_desc}
    Generate {num} diverse instructions that could be goals.
    Format: one per line, natural language.
    
    Examples:
    - Pick up the red cube
    - Move the cube to the left
    - Stack the blue block on top
    """
    
    response = openai.ChatCompletion.create(
        model="gpt-4",  # Or claude-3-opus
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content.split('\n')
```

**Afternoon (4h):**
- Generate synthetic data for 10K additional samples
- Create augmented dataset
- Validate quality manually (sample 100)

---

### **Day 6: Counterfactual Explanations**

**LLM-generated failure explanations:**
```python
def explain_failure(state, action, outcome):
    prompt = f"""
    State: {state}
    Action: {action}
    Outcome: {outcome['success']} (reward: {outcome['reward']})
    
    In 1 sentence, explain why this action 
    {'succeeded' if outcome['success'] else 'failed'}.
    
    Focus on physical reasoning and constraints.
    """
    
    explanation = claude_api.generate(prompt)
    return explanation
```

**Generate explanations for:**
- All 56K actions in dataset
- Focus on failures (more informative)
- Cache to disk

---

### **Day 7: Retrain with Augmented Data**

**Submit training with:**
- Original 3.5K groups
- +10K synthetic instruction variations
- +56K action explanations (auxiliary loss)

**Expected improvement:** +2-5% → **52-57% total**

---

## 📅 **WEEK 2: Architecture & Training (Day 8-14)**

### **Day 8-9: Multi-Scale Transformer**

**Architecture:**
```python
class MultiScaleTransformer(nn.Module):
    def __init__(self):
        self.small = DoVLATransformer(d_model=128, n_layers=2)
        self.medium = DoVLATransformer(d_model=256, n_layers=3)
        self.large = DoVLATransformer(d_model=512, n_layers=4)
        
        # Learned ensemble weights
        self.ensemble_weights = nn.Parameter(torch.ones(3))
    
    def forward(self, obs, actions, lang):
        s1 = self.small(obs, actions, lang)
        s2 = self.medium(obs, actions, lang)
        s3 = self.large(obs, actions, lang)
        
        weights = F.softmax(self.ensemble_weights, dim=0)
        return weights[0]*s1 + weights[1]*s2 + weights[2]*s3
```

**Train 3 scales separately, then ensemble**

---

### **Day 10: Action-Conditioned Attention**

**Add action-specific attention:**
```python
class ActionConditionedAttention(nn.Module):
    def __init__(self):
        # Learn to attend to relevant state parts per action
        self.action_encoder = nn.Linear(action_dim, d_model)
        self.state_attention = nn.MultiheadAttention(d_model, n_heads)
    
    def forward(self, state, action):
        # Action vector guides what to look at in state
        action_query = self.action_encoder(action)
        attended_state, _ = self.state_attention(
            action_query, state, state
        )
        return attended_state
```

---

### **Day 11-12: Hard Negative Mining**

**Mine confusing pairs:**
```python
def mine_hard_negatives(model, dataset, k=5):
    """Find pairs where model is most confused."""
    hard_pairs = []
    
    for group in dataset:
        scores = model.predict(group)
        
        # Find pairs where:
        # 1. Model predicts A > B
        # 2. Ground truth is B > A
        # 3. Margin is small (confusing)
        
        for i, j in all_pairs:
            pred_margin = scores[i] - scores[j]
            true_margin = rewards[i] - rewards[j]
            
            if sign(pred_margin) != sign(true_margin):
                confusion = abs(pred_margin)
                if confusion < threshold:  # Close call
                    hard_pairs.append((group, i, j, confusion))
    
    # Return top-k% hardest
    return sorted(hard_pairs, key=lambda x: x[-1])[:int(len(hard_pairs)*k/100)]
```

**Retrain focusing 70% on hard pairs, 30% on all pairs**

---

### **Day 13: Curriculum Learning**

**Task difficulty ranking:**
```python
task_difficulty = {
    'PickCube-v1': 1,      # Easy
    'PushCube-v1': 2,      # Medium
    'PullCube-v1': 2,      # Medium
    'LiftPegUpright-v1': 3,  # Hard
    'StackCube-v1': 4,     # Very hard
    'PegInsertionSide-v1': 5  # Hardest
}

# Training schedule
def get_tasks_for_epoch(epoch, total_epochs=50):
    progress = epoch / total_epochs
    max_difficulty = 1 + progress * 4  # 1 → 5 over training
    
    return [t for t, d in task_difficulty.items() if d <= max_difficulty]
```

---

### **Day 14: Self-Training with LLM Feedback**

**LLM provides corrective feedback:**
```python
def get_llm_feedback(state, action_a, action_b, model_pred, ground_truth):
    if model_pred == ground_truth:
        return None  # Model correct
    
    prompt = f"""
    The model incorrectly predicted action A is better than B.
    Actually, B is better.
    
    State: {state}
    Action A: {action_a}
    Action B: {action_b}
    
    What physical reasoning explains why B > A?
    What should the model learn to avoid this mistake?
    
    Response format:
    - Key insight: [1 sentence]
    - Focus on: [state feature to attend to]
    """
    
    feedback = claude_api.generate(prompt)
    return feedback
```

**Use feedback as auxiliary training signal**

**Week 2 expected result:** 57-62%

---

## 📅 **WEEK 3: Ensemble & Advanced (Day 15-21)**

### **Day 15-16: Multi-Model Ensemble**

**Train 5 diverse architectures:**
```python
models = {
    'transformer_small': DoVLATransformer(d_model=256, n_layers=2),
    'transformer_large': DoVLATransformer(d_model=512, n_layers=4),
    'mlp_deep': DeepMLP(hidden=[512, 512, 256]),
    'multiscale': MultiScaleTransformer(),
    'action_conditioned': ActionConditionedTransformer()
}

# Train each independently
for name, model in models.items():
    train(model, dataset)
    save(model, f'checkpoints/{name}_best.pt')
```

**Ensemble strategies:**
- Voting (majority vote)
- Averaging (mean scores)
- Stacking (meta-learner on top)

---

### **Day 17-18: LLM as Final Judge**

**Most powerful improvement (+10-15%):**

```python
def llm_action_ranking(state, instruction, candidate_actions, model_scores):
    """Use LLM to re-rank top-k actions from model."""
    
    # Get top-5 from model ensemble
    top_k = 5
    top_actions = get_top_k(candidate_actions, model_scores, k=top_k)
    
    # Format for LLM
    action_descriptions = [
        f"{i+1}. {describe_action(a)}" 
        for i, a in enumerate(top_actions)
    ]
    
    prompt = f"""
    You are a robot action selection expert.
    
    State:
    {describe_state(state)}
    
    Goal:
    {instruction}
    
    Candidate actions:
    {chr(10).join(action_descriptions)}
    
    Rank these actions from 1 (best) to {top_k} (worst).
    Consider:
    - Physics (will it work?)
    - Safety (any collisions?)
    - Efficiency (direct path?)
    - Goal achievement
    
    Output ONLY the ranking numbers: [best_idx, 2nd_best, ...]
    Example: [3, 1, 5, 2, 4]
    """
    
    response = claude_api.generate(prompt, max_tokens=50)
    llm_ranking = parse_ranking(response)
    
    # Return best action according to LLM
    return top_actions[llm_ranking[0]]
```

**This is the BIGGEST single improvement!**

---

### **Day 19: Retrieval-Augmented Generation**

**RAG for similar examples:**
```python
def retrieve_similar_states(current_state, dataset, k=10):
    """Find k most similar states with successful actions."""
    
    # Embed all states
    state_embeddings = embed_all_states(dataset)
    current_emb = embed_state(current_state)
    
    # Cosine similarity
    similarities = cosine_similarity(current_emb, state_embeddings)
    top_k_idx = torch.topk(similarities, k).indices
    
    # Return successful examples
    examples = [
        dataset[i] for i in top_k_idx 
        if dataset[i].reward.terminal_success
    ]
    
    return examples

# Use in LLM prompt
similar = retrieve_similar_states(state, dataset, k=5)
prompt = f"""
Current state: {state}
Similar successful examples:
{format_examples(similar)}

Based on these, rank the candidate actions.
"""
```

---

### **Day 20: Chain-of-Thought Reasoning**

**Make LLM explain step-by-step:**
```python
prompt = f"""
State: {state}
Goal: {instruction}
Actions: {actions}

For each action, reason step-by-step:

Action 1: {action_1}
Step 1: What will happen physically?
Step 2: Will it achieve the goal?
Step 3: Any risks or failures?
Step 4: Overall rating (1-10):

[Repeat for all actions]

Final ranking: [best to worst]
"""
```

**More expensive but more accurate**

---

### **Day 21: Full System Evaluation**

**Test complete pipeline:**
```python
def evaluate_full_pipeline(dataset):
    results = 
    
    # 1. Baseline Transformer (no improvements)
    results['baseline'] = evaluate(transformer_basic)
    
    # 2. + Language
    results['language'] = evaluate(transformer_lang)
    
    # 3. + Data augmentation
    results['data_aug'] = evaluate(transformer_lang_aug)
    
    # 4. + Architecture improvements
    results['architecture'] = evaluate(multiscale_model)
    
    # 5. + Training improvements
    results['training'] = evaluate(trained_with_curriculum)
    
    # 6. + Ensemble
    results['ensemble'] = evaluate(ensemble_model)
    
    # 7. + LLM judge (FINAL)
    results['final'] = evaluate(system_with_llm_judge)
    
    return results
```

**Expected final result: 60-70%+**

---

## 📊 **EXPECTED PROGRESS TRACKING**

| Checkpoint | Selected Success | Improvement | Cumulative |
|---|---|---|---|
| Current Transformer | 42-44% | - | Baseline |
| +Language (Day 4) | 50-55% | +8-11% | +8-11% |
| +Data Aug (Day 7) | 52-57% | +2-5% | +10-15% |
| +Architecture (Day 10) | 54-59% | +2-4% | +12-17% |
| +Training (Day 14) | 57-62% | +3-5% | +15-20% |
| +Ensemble (Day 16) | 60-65% | +3-5% | +18-23% |
| +LLM Judge (Day 18) | **65-75%** | +10-15% | **+23-33%** |
| +RAG+CoT (Day 20) | **67-78%** | +2-5% | **+25-36%** |

**Final target: 65-75% selected success**

---

## 💰 **API Cost Estimation**

**With unlimited API:**
- Embeddings: sentence-transformers (free, local)
- Synthetic data: ~10K LLM calls
- Explanations: ~56K LLM calls
- LLM judge: ~3.5K calls/eval × 10 evals = 35K calls
- RAG: ~3.5K calls
- CoT: ~3.5K calls (expensive, 500 tokens/call)

**Total: ~110K LLM API calls over 3 weeks**

**With Claude API:** ~$550-1,100 (at $5-10 per 1M tokens)
**Your case: Unlimited → FREE!** 🎉

---

## 🎯 **SUCCESS CRITERIA**

**Minimum success (Week 2):**
- 55%+ selected success
- Better than baseline (+12%)
- Publishable improvement

**Target (Week 3):**
- 60%+ selected success
- Strong CVPR paper
- Clear ablation study

**Stretch (if LLM judge works well):**
- 70%+ selected success
- SOTA-competitive at small scale
- Major contribution

---

## 📋 **NEXT IMMEDIATE ACTIONS**

**Now (while current Transformer trains):**
1. ✅ Setup environment (pip install dependencies)
2. ✅ Test language embedding generation
3. ✅ Create implementation skeleton

**When current training finishes (2h):**
1. Evaluate baseline (42-44%)
2. Start Week 1 Day 1 (language integration)
3. Launch parallel experiments

---

**Ready to start implementation?** 🚀

Let me know when to begin Day 1, or I can start preparing now!