make reward dynamic based oon action sequence

Dockerfile

@@ -11,7 +11,8 @@ RUN pip install --no-cache-dir \
     pandas==2.1.3 \
     numpy==1.26.2 \
     openai==1.3.7 \
-    pyyaml==6.0.1
+    pyyaml==6.0.1 \
+    python-dotenv==1.0.0
 
 # Copy all project files
 COPY . .

README.md

@@ -108,7 +108,10 @@ run a complete cleaning pipeline in the right sequence.
 
 ## 🏆 Reward Function
 
-Rewards are computed after every step providing dense signal:
+Rewards are computed after every step providing dense signal for each data quality component.
+**Importantly, the reward function penalizes out-of-order operations**, encouraging agents to follow the optimal data cleaning sequence.
+
+### Reward Components
 
 | Component | Description |
 |-----------|-------------|
@@ -117,9 +120,47 @@ Rewards are computed after every step providing dense signal:
 | dtype_score | Proportion of columns with correct data types |
 | outlier_score | How close numeric distributions are to gold |
 | schema_score | Proportion of column names matching gold |
-| penalty | Small penalty for using too many steps |
+| penalty | Penalty for out-of-order operations or too many steps |
+
+### Sequence Penalties
+
+The reward function enforces optimal operation order:
+
+```
+Optimal Sequence:
+1. Remove Duplicates    (clean redundant data first)
+2. Fix Data Types       (understand structure)
+3. Fill Missing Values  (based on correct types)
+4. Remove Outliers      (after understanding distribution)
+5. Validate Schema      (final verification)
+```
+
+**Penalty Rules:**
+- **Out-of-order operation:** -0.08 (e.g., filling missing before fixing types)
+- **Repeated operation:** -0.02 (e.g., filling missing twice in a row)
+- **Exceeded step limit:** -0.05 (using >80% of max_steps)
+
+**Total reward = weighted sum of components - penalties (0.0 to 1.0)**
+
+For example:
+- Good sequence → Rewards improve with each step toward 1.0 ✅
+- Bad sequence → Rewards degrade as penalties accumulate ❌
+
+---
 
-**Total reward = weighted sum of components (0.0 to 1.0)**
+## 📊 Baseline Scores
+
+Baseline scores with **gpt-4o-mini** (sequence-aware reward function):
+
+| Task | Difficulty | Score | Notes |
+|------|-----------|-------|-------|
+| easy_dedup_rename | Easy | 0.9900 | Follows optimal sequence |
+| medium_missing_dtype | Medium | 0.7000 | Some out-of-order operations |
+| hard_full_pipeline | Hard | 0.6636 | Oscillates between operations |
+| **Average** | - | **0.7845** | Room for improvement |
+
+**Note:** Scores are lower than original because agents sometimes violate the sequence.
+This is intentional—it teaches agents the correct workflow!
 
 ---
 
@@ -162,29 +203,13 @@ state = requests.get(
 ### Run Baseline Inference
 ```bash
 export HF_TOKEN=your_token_here
-export MODEL_NAME=meta-llama/Llama-3.3-70B-Instruct
-export API_BASE_URL=https://router.huggingface.co/v1
+export MODEL_NAME=gpt-4o-mini
+export API_BASE_URL=https://api.openai.com/v1
 
 python inference.py
 ```
 
 ---
-
-## 📊 Baseline Scores
-
-Scores produced by `meta-llama/Llama-3.3-70B-Instruct`:
-
-| Task | Score |
-|------|-------|
-| easy_dedup_rename | 1.0000 |
-| medium_missing_dtype | 1.0000 |
-| hard_full_pipeline | 1.0000 |
-| expert_sales_pipeline | 1.0000 |
-| **Average** | **1.0000** |
-
----
-
-## 📁 Project Structure
 ```
 data-cleaning-openenv/
 ├── main.py              # FastAPI server

environment.py

@@ -32,6 +32,7 @@ class DataCleaningEnv:
         self.done: bool = False
         self.max_steps: int = 10
         self.reward_history = []
+        self.actions_taken: List[str] = []  # Track sequence of actions
         self._load_task_metadata()
 
     # ─────────────────────────────────────────
@@ -113,6 +114,62 @@ class DataCleaningEnv:
             message=message
         )
 
+    def _compute_sequence_penalty(self) -> float:
+        """
+        Penalize illogical action sequences.
+
+        Optimal sequence:
+        1. remove_duplicates (clean up redundant data)
+        2. fix_dtype (understand structure)
+        3. fill_missing (based on correct types)
+        4. remove_outliers (after understanding distribution)
+        5. validate_schema (final check)
+
+        Penalties:
+        - Doing operations out of order: -0.05 per violation
+        - Repeating same operation: -0.02 per repeat
+        """
+        OPTIMAL_ORDER = [
+            "remove_duplicates",
+            "fix_dtype",
+            "fill_missing_mean",
+            "fill_missing_mode",
+            "fill_missing_median",
+            "remove_outliers",
+            "validate_schema",
+        ]
+
+        penalty = 0.0
+        last_order_idx = -1
+        prev_action = None
+
+        # Track positions of operations in optimal order
+        for action in self.actions_taken:
+            if action == "finish":
+                continue
+
+            # Penalize repeated same actions (doing same thing twice in a row)
+            if action == prev_action and action not in ["fill_missing_mean", "fill_missing_mode", "fill_missing_median", "remove_outliers"]:
+                penalty += 0.02
+
+            # Find this action in optimal order
+            if action in OPTIMAL_ORDER:
+                current_idx = OPTIMAL_ORDER.index(action)
+
+                # Penalize out-of-order operations
+                if current_idx < last_order_idx:
+                    # Going backward in sequence (e.g., doing remove_duplicates after fill_missing)
+                    penalty += 0.08
+
+                last_order_idx = current_idx
+            else:
+                # Unknown operation
+                penalty += 0.01
+
+            prev_action = action
+
+        return min(0.25, penalty)  # Cap penalty at 0.25
+
     def _compute_reward(self) -> Reward:
         df   = self.current_df
         gold = self.gold_df
@@ -194,10 +251,15 @@ class DataCleaningEnv:
             matched   = sum(1 for c in gold_cols if c in curr_cols)
             schema_score = matched / len(gold_cols) if gold_cols else 0.0
 
-        # ── Penalty for too many steps ───────────────────────────────
+        # ── Penalty for too many steps + sequence violations ─────────────
         step_ratio = self.step_count / self.max_steps
+        base_penalty = 0.0
         if step_ratio > 0.8:
-            penalty = 0.05
+            base_penalty = 0.05
+
+        # Add sequence-based penalty
+        sequence_penalty = self._compute_sequence_penalty()
+        penalty = base_penalty + sequence_penalty
 
         # ── Weighted total ───────────────────────────────────────────
         weights = {
@@ -388,6 +450,7 @@ class DataCleaningEnv:
         self.step_count   = 0
         self.done         = False
         self.reward_history = []
+        self.actions_taken = []  # Reset action history
 
         obs    = self._get_observation("Environment reset. Start cleaning!")
         reward = Reward(total=0.0)
@@ -442,6 +505,9 @@ class DataCleaningEnv:
         except Exception as e:
             message = f"Operation failed: {str(e)}"
 
+        # ── Track action for sequence penalties ───────────────────────────
+        self.actions_taken.append(action.operation)
+
         # ── Check max steps ──────────────────────────────────────────
         if self.step_count >= self.max_steps:
             self.done = True

inference.py

@@ -15,9 +15,13 @@ import sys
 import json
 import time
 from typing import List, Dict, Any
+from dotenv import load_dotenv
 
 from openai import OpenAI
 
+# Load environment variables from .env file
+load_dotenv()
+
 # ─────────────────────────────────────────
 # CONFIG
 # ─────────────────────────────────────────

openenv.yaml

@@ -97,10 +97,11 @@ reward_space:
 
 baseline_scores:
   model: gpt-4o-mini
-  easy_dedup_rename: 1.0000
-  medium_missing_dtype: 0.6643
-  hard_full_pipeline: 0.8386
-  average: 0.8343
+  description: Sequence-aware reward function (rewards optimal order of operations)
+  easy_dedup_rename: 0.9900
+  medium_missing_dtype: 0.7000
+  hard_full_pipeline: 0.6636
+  average: 0.7845
 
 runtime:
   language: python