🎯 What Is This Model?

This is a LoRA adapter on top of unsloth/Qwen2.5-0.5B-Instruct, trained with GRPO (Group Relative Policy Optimization) to solve long-horizon ESG corporate decision-making.

The agent acts as a corporate sustainability strategist, choosing monthly interventions to hit ESG targets within a fixed budget. The challenge: actions have delayed compound effects (e.g. solar panels take months to pay off), requiring multi-step reasoning.

🔄 ESG RL ENVIRONMENT LOOP
🏢 Company State	🤖 LLM Agent
• Carbon: 1200 tons • Renewable: 12% • Budget: $500K • Month: 1/6	{"action": 0, "reasoning": "Installing solar now gives 6 months of renewable boost."}

🏆 Benchmark Performance

Agent	🟢 Easy	🟡 Medium	🔴 Hard	📊 Overall
🎲 Random Baseline	0.740	0.643	0.678	0.687
🧠 Heuristic Baseline	1.000	0.847	0.852	0.900
🚀 This Model (V2 GRPO)	1.000	0.880 ✨	0.726	0.869

✨ Key Achievement: This model outperformed the hand-coded heuristic on the Medium task (0.880 vs 0.847), demonstrating genuine zero-shot RL generalization — not just memorization.

📈 Gains Over Random

Label       Performance Increase      Pct     Range
----------- ------------------------- ------- --------------------
Easy        ████████████████████████░  +35.1% (0.740 → 1.000)
Medium      █████████████████████████  +36.8% (0.643 → 0.880)
Hard        █████░░░░░░░░░░░░░░░░░░░░  + 7.1% (0.678 → 0.726)
Overall     ██████████████████░░░░░░░  +26.5% (0.687 → 0.869)

⚙️ Training Configuration

Parameter	Value
Base Model	`unsloth/Qwen2.5-0.5B-Instruct`
Method	GRPO (Group Relative Policy Optimization)
Framework	TRL + Unsloth
Hardware	NVIDIA A100 80GB
Precision	bf16
LoRA Rank	16
LoRA Alpha	32
Learning Rate	8e-6
Batch Size	4 per device
GRPO Generations	6 per prompt
Max New Tokens	192
Total Steps	150
Curriculum	50 easy → 60 medium → 40 hard
Dataset Size	350 samples (heuristic + random + adversarial)
Training Time	~40 minutes
Training Cost	~$4.50

📅 3-Stage Curriculum

Step 0 --------- Step 50 --------- Step 110 --------- Step 150
  |                  |                  |                  |
  |   [EASY]         |   [MEDIUM]       |   [HARD]         |
  |   50 steps       |   60 steps       |   40 steps       |
  +------------------+------------------+------------------+

🧪 Reward System (4 Independent Signals — No LLM Judge)

Signal	Weight	Type	Description
⚙️ Environment Outcome	45%	Shaped	Reward from stepping the ESG simulator
📋 Format Compliance	25%	Binary	Valid `{"action": int, "reasoning": str}` JSON
🛡️ Anti-Cheat	15%	Penalty	Penalizes NO_ACTION spam & action repetition
📈 Task Progress	15%	Terminal	Final deterministic grader score

All reward functions are verifiable, rule-based, and LLM-free. No GPT-as-judge.

⚡ Quick Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# ── 1. Load model ─────────────────────────────────────────────────
base = AutoModelForCausalLM.from_pretrained("unsloth/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(
    base, "tharun5054/esg-rl-agent-grpo", subfolder="lora_adapter"
)
tokenizer = AutoTokenizer.from_pretrained(
    "tharun5054/esg-rl-agent-grpo", subfolder="lora_adapter"
)
model.eval()

# ── 2. Build ESG scenario ─────────────────────────────────────────
prompt = """You are an ESG sustainability strategist. Current state:
- Carbon Emissions: 1200 tons/month (target: -15%)
- Renewable Energy: 12% (target: 30%)
- Available Budget: $500,000
- Month: 1 of 6

Actions: 0=Solar($150K,6mo), 1=HVAC($80K,12mo), 2=Recycling($25K),
         3=Water($60K,12mo), 4=Carbon Offset($40K), 5=Diversity($50K),
         6=Wellness($30K), 7=Audit($15K), 8=NoAction

Output JSON only:"""

# ── 3. Generate ───────────────────────────────────────────────────
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=128,
        temperature=0.7, do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
# → {"action": 0, "reasoning": "Solar panels provide 6 months of renewable energy boost..."}

📂 Repository Structure

Click to see all files

tharun5054/esg-rl-agent-grpo/
│
├── 🔧 lora_adapter/
│   ├── adapter_config.json          ← LoRA config (rank=16, alpha=32)
│   ├── adapter_model.safetensors    ← Trained weights (~35MB)
│   ├── tokenizer.json               ← Qwen2.5 tokenizer
│   └── tokenizer_config.json
│
├── 📊 results/
│   ├── trained_v2.json              ← Full benchmark (5 seeds × 3 tasks)
│   ├── baseline_heuristic.json      ← Heuristic baseline
│   ├── baseline_random.json         ← Random baseline
│   ├── score_comparison.png         ← Bar chart comparison
│   ├── reward_history.png           ← Reward curves per step
│   └── esg_metrics.png              ← Final ESG metric breakdown
│
├── 📋 EVALUATION_REPORT.md          ← Detailed V2 analysis
└── 📋 SUBMISSION.md                 ← Hackathon submission brief

🌐 Related Links

	Resource	Link
📦	Source Code	TharunBabu-05/OPEN-ENV
🚀	Training Space	tharun5054/esg-rl-train
🌍	Live Env API	tharun5054/esg-compliance-env
📊	Evaluation Report	`EVALUATION_REPORT.md`