README.md

---
title: FinePrint-Env
emoji: "\U0001F4DC"
colorFrom: yellow
colorTo: red
sdk: docker
app_port: 7860
tags:
  - openenv
  - reinforcement-learning
  - policy-compliance
  - drift-detection
  - customer-service
pinned: false
---

# FinePrint-Env: Consumer Policy Drift Detection Environment

> **[Live Demo & API](https://huggingface.co/spaces/PraneshkumarR/fineprint-env)** | **[Training Notebook (Colab)](https://colab.research.google.com/drive/1s6bUXezbqFNDYnQvmMs5pnQY232QXvJk?usp=sharing)**

## Overview

FinePrint-Env is a reinforcement learning environment where AI agents learn to detect policy changes and maintain compliance in customer service workflows. Built for the **Meta PyTorch OpenEnv Hackathon x Scaler School of Technology**, it provides a realistic simulation of policy drift scenarios ranging from simple quoting to adversarial multi-version silent drift.

## Motivation

- **Policies change constantly** — pricing, return windows, subscription terms shift weekly. An agent quoting a return policy updated 10 minutes ago creates **legal and financial liability**.
- **No existing RL environment tests drift detection** — FinePrint-Env fills this gap with 8 policy versions, 5 customer workflows, and deterministic compliance grading.
- **70% of drifts are silent** — no system notification is sent. The agent must learn to detect drift from user-level signals and staleness alone.

## The Problem

Production LLMs assume static knowledge. In reality, policies, pricing, and rules change constantly. An agent quoting a return policy that was updated 10 minutes ago creates **legal and financial liability**. No existing benchmark tests or trains this capability.

## The Solution

FinePrint teaches models a single critical meta-skill: **when to call `request_verification()`** — the binary decision that separates safe agents from dangerous ones. Rather than memorizing policies, the model learns to recognize *drift signals* (user contradictions, staleness, system notifications) and re-ground itself before responding.

## Why Not Just RAG? Why Not Agentic Workflows?

This is the question everyone asks. Here's why neither solves the actual problem:

### RAG (Retrieval-Augmented Generation)

RAG retrieves fresh documents at query time. Sounds perfect — until you realize:

- **RAG doesn't know *when* to retrieve.** It either retrieves every time (wasteful, slow, expensive) or relies on a fixed schedule (misses urgent changes). There's no learned judgment about *staleness*.
- **RAG has no concept of drift severity.** A return window changing from 30→14 days is catastrophic. A FAQ typo fix is irrelevant. RAG treats both the same — it just fetches.
- **RAG doesn't penalize stale answers.** If the retriever returns a cached/stale chunk, the model quotes it confidently. There's no feedback loop teaching it that "this information might be outdated."
- **RAG is reactive, not proactive.** It responds to queries. It never says *"wait, I should double-check this before answering"* — that's a learned meta-skill, not a retrieval pattern.

### Agentic Workflows (Tool-Using LLMs)

Agents with tools can call APIs, search databases, and verify information. But:

- **Tool availability ≠ tool wisdom.** Giving a model a `verify_policy()` tool doesn't mean it knows *when* to call it. Without training, agents either never verify (dangerous) or verify every step (unusable in production).
- **No reward signal for drift detection.** Agentic frameworks like LangChain/CrewAI provide tools but no RL reward for using them at the right moment. The agent has no incentive to develop timing intuition.
- **Hardcoded verification rules are brittle.** You could write `if steps_since_verify > 5: verify()` — but that's a heuristic, not intelligence. It doesn't adapt to context (high-stakes question vs casual chat).
- **No benchmark exists to measure this.** How do you evaluate whether your agent verifies at the right time? There's no leaderboard, no graded task, no compliance score. You just hope it works.

### What FinePrint Actually Does Differently

FinePrint doesn't retrieve documents or provide tools — it **trains the judgment layer** that sits above both:

| Capability | RAG | Agentic | FinePrint |
|---|---|---|---|
| Access to fresh data | ✅ retrieves | ✅ tools | ✅ `request_verification()` |
| Knows *when* to refresh | ❌ always/never | ❌ hardcoded | ✅ **learned via RL** |
| Drift severity awareness | ❌ | ❌ | ✅ reward-weighted |
| Penalizes stale answers | ❌ | ❌ | ✅ -8.0 per stale quote |
| Trains verification timing | ❌ | ❌ | ✅ +3.0 timely, +1.0 late |
| Graded compliance tasks | ❌ | ❌ | ✅ 3 difficulty levels |
| Works *with* RAG/agents | — | — | ✅ trains the meta-skill they lack |

**The insight:** RAG and agentic workflows solve *access* to fresh information. FinePrint solves *judgment* about when that access matters. They're complementary — FinePrint trains the decision layer that makes RAG and tool-use actually safe.

## Environment Description

The environment simulates a customer service agent handling consumer workflows (shopping, returns, subscriptions, bookings, complaints) while company policies change silently in the background. The agent must use available commands to inspect policies, quote values accurately, detect drift, and maintain compliance.

## Action Space

| Command | Arguments | Description |
|---------|-----------|-------------|
| `view_policies` | (none) | View currently cached policy values |
| `view_workflow` | (none) | View current workflow state and conversation |
| `check_compliance` | (none) | Check current compliance status |
| `request_verification` | (none) | Refresh policy cache and detect drift |
| `quote_policy` | `policy_field`, `quoted_value` | Quote a specific policy field to customer |
| `respond_to_user` | `message` | Send a general message to the customer |
| `take_action` | `message` | Process a workflow action (checkout, return, etc.) |
| `escalate` | `message` | Escalate to supervisor (only when drift detected) |
| `abort_workflow` | `message` | Abort current workflow (only when justified) |
| `clarify` | `message` | Ask customer for clarification |
| `submit` | (none) | Submit for final grading |

## Observation Space

Each step returns an observation containing:

- **output** -- Command result text (policy values, compliance status, workflow state, etc.)
- **task_description** -- Current task description and objectives
- **workflow_names** -- List of available workflows
- **available_commands** -- Available actions the agent can take
- **done** -- Whether the episode is complete
- **reward** -- Score (0.0--1.0) returned on submission

## Tasks

### Task 1: quote_accuracy (Easy)

Quote policies correctly across `shop` and `return` workflows with no drift.

- **Expected difficulty:** Easy
- **Max steps:** 20

### Task 2: drift_detection (Medium)

Handle 3 workflows while detecting policy changes. 30% drift probability with 50% silent ratio.

- **Expected difficulty:** Medium
- **Max steps:** 30

### Task 3: compliance_storm (Hard)

All 5 workflows under aggressive silent drift across 8 policy versions. 50% drift probability with 80% silent ratio.

- **Expected difficulty:** Hard
- **Max steps:** 45

## Reward Function

```
score = 0.3 * (compliance_accuracy) + 0.5 * (workflow_completion) + 0.2 * (drift_responsiveness)
```

| Component | Weight | Description |
|-----------|--------|-------------|
| Compliance accuracy | 0.3 | Proportion of policy quotes that are correct |
| Workflow completion | 0.5 | Proportion of workflows completed |
| Drift responsiveness | 0.2 | Proportion of drifts detected via verification |

### Step-Level Rewards (14 signals: 7 positive, 7 negative)

| Event | Reward |
|-------|--------|
| Correct policy quote | **+10.0** |
| Timely drift detection (≤ 2 steps) | **+3.0** |
| Late drift detection (> 2 steps) | **+1.0** |
| Freshness bonus (verified ≤ 2 steps ago) | **+1.0** |
| High user satisfaction | **+2.0** |
| Zero compliance failures (terminal) | **+20.0** |
| Stale policy citation (HIGH severity) | **−8.0** |
| Incorrect value quoted | **−4.0** |
| User satisfaction < 0.3 | **−5.0** |
| Unnecessary escalation | **−4.0** |
| Unnecessary abort | **−3.0** |
| Unnecessary verification | **−0.5** |
| Any compliance failure (terminal) | **−30.0** |

## Setup & Usage

### Local Development

```bash
pip install -r requirements.txt
uvicorn server.app:app --host 0.0.0.0 --port 7860 --reload
```

### Docker

```bash
docker build -t fineprint-env .
docker run -p 7860:7860 fineprint-env
```

### API Usage

```python
import requests

BASE = "http://localhost:7860"

# Reset with a specific task
obs = requests.post(f"{BASE}/reset", json={"options": {"task_id": "quote_accuracy"}}).json()

# View policies
obs = requests.post(f"{BASE}/step", json={"action": {"command": "view_policies", "args": {}}}).json()
print(obs["output"])

# Quote a policy
obs = requests.post(f"{BASE}/step", json={
    "action": {"command": "quote_policy", "args": {"policy_field": "return.window_days", "quoted_value": "30"}}
}).json()

# Submit for grading
obs = requests.post(f"{BASE}/step", json={"action": {"command": "submit", "args": {}}}).json()
print(f"Score: {obs['reward']}")
```

### Python Client

```python
from client import FinePrintClient

client = FinePrintClient(base_url="http://localhost:7860")
client.reset(task_id="drift_detection")
obs = client.step("view_policies")
obs = client.step("quote_policy", policy_field="return.window_days", quoted_value="30")
obs = client.step("submit")
```

### Gymnasium Interface (standalone)

```python
import gymnasium as gym

env = gym.make("FinePrint-v0")
obs, info = env.reset(seed=42)

action = {"action_type": 0}  # request_verification
obs, reward, terminated, truncated, info = env.step(action)
```

## Baseline Scores

| Task | Score | Steps |
|------|-------|-------|
| quote_accuracy | ~0.80 | 8--12 |
| drift_detection | ~0.55 | 15--20 |
| compliance_storm | ~0.25 | 25--35 |

## Running the Baseline

```bash
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o-mini"
export OPENAI_API_KEY="your-key"
export ENV_URL="http://localhost:7860"
python inference.py
```

## Policy Drift

Eight policy versions are composed via **delta merging** — each version overrides specific fields from the base while inheriting the rest:

| Version | Change | Severity |
|---------|--------|----------|
| `v1_base` | Baseline policies | — |
| `v2_return_change` | Return window 30 → 14 days, refund → store credit | HIGH |
| `v3_shipping_change` | Free shipping threshold $50 → $75 | MEDIUM |
| `v4_subscription_change` | Auto-renewal: off → mandatory | HIGH |
| `v5_cancellation_fee` | Booking cancellation fee $0 → $25 | MEDIUM |
| `v6_complaint_change` | Max compensation $200 → $50, escalation removed | HIGH |
| `v7_scope_change` | Electronics returns eliminated, price match removed | CRITICAL |
| `v8_pricing_change` | Tax included in price, bulk discount removed | MEDIUM |

## Drift Signals

The agent receives 4 types of signals that policies may have changed:

| Signal | Explicitness | Example |
|--------|-------------|---------|
| System notification | Explicit | `"POLICY UPDATE: Version v3 is now active"` |
| User contradiction | Implicit | `"But the website says 14 days, not 30..."` |
| User confusion | Implicit | `"That doesn't match what I was told"` |
| Staleness counter | Passive | Steps since last `request_verification()` |

**70% of drifts are silent** — no system notification is sent. The agent must learn to detect drift from user-level signals and staleness alone.

## Training

FinePrint uses **GRPO (Group Relative Policy Optimization)** to fine-tune a language model with LoRA adapters.

### Default Configuration

| Parameter | Value |
|-----------|-------|
| Base model | `Qwen/Qwen2.5-1.5B-Instruct` |
| LoRA rank / alpha | 16 / 32 |
| Episodes | 200 |
| Rollouts per update | 8 |
| Learning rate | 2e-5 |
| Discount (γ) | 0.99 |
| PPO clip (ε) | 0.2 |
| Entropy coefficient | 0.01 |
| Drift probability | 0.25 |
| Silent drift ratio | 0.70 |
| Max episode steps | 60 |
| Workflows per episode | 5 |

### Running Training

```bash
# Local (requires GPU + Unsloth)
python training/train_unsloth.py

# Google Colab
# Open FinePrint_Colab.ipynb

# HuggingFace Jobs
# Open FinePrint_HFJobs.ipynb
```

## Results

Training on **Qwen2.5-1.5B-Instruct** for 80 episodes (20 GRPO updates):

| Updates | Avg Reward |
|---------|-----------|
| 1–4 | −3.4 |
| 5–8 | +0.6 |
| 9–12 | +5.7 |
| 13–16 | +6.7 |
| 17–20 | +7.2 |

The model improved from **−2.4 to +7.8** reward over training, with entropy staying healthy (1.15 → 1.22, no mode collapse) and valid output samples increasing (81 → 106).

## Technical Details

- Built with **FastAPI** + **Pydantic** for typed request/response models
- Core environment logic uses **Gymnasium** interface with numpy observations
- HTTP wrapper exposes standard OpenEnv endpoints for remote agent interaction
- 8 policy versions loaded via JSON delta-merging from `policies/` directory
- Deterministic compliance grading via field-level policy comparison
- Supports concurrent sessions via `session_id` parameter
- Runs on 2 vCPU / 8 GB RAM within 20 minutes

## Project Structure

```
fineprint/
├── server/                  # HTTP API layer (FastAPI)
│   ├── app.py               #   FastAPI endpoints + landing page
│   ├── fineprint_environment.py  #   HTTP environment wrapper
│   └── tasks.py             #   3 graded task definitions
├── fineprint/               # Core package
│   ├── env.py               #   Gymnasium-compatible RL environment
│   ├── policies.py          #   Policy loading, versioning, delta merging
│   ├── drift.py             #   Drift scheduling (when/how policies change)
│   ├── state.py             #   Episode state management
│   ├── workflows.py         #   5 consumer workflow definitions
│   ├── checker.py           #   Compliance validation engine
│   ├── rewards.py           #   Reward shaping calculator (14 signals)
│   └── utils.py             #   Shared utilities
├── policies/                # 8 policy versions (JSON) + manifest
├── training/                # GRPO training & evaluation scripts
│   ├── train_unsloth.py     #   Training loop (Unsloth + LoRA)
│   └── eval.py              #   Post-training evaluation
├── tests/                   # Unit tests (pytest)
├── models.py                # Typed Pydantic models (Action, Observation, State)
├── client.py                # HTTP client for remote interaction
├── inference.py             # Baseline inference script with mandatory logging
├── openenv.yaml             # OpenEnv spec configuration
├── Dockerfile               # HuggingFace Spaces container
├── pyproject.toml           # Modern build configuration
├── config.py                # TrainingConfig dataclass
└── requirements.txt         # Dependencies
```

## OpenEnv Spec Compliance

- step(action) returns observation, reward, done
- reset() returns initial observation
- state() returns episode metadata
- openenv.yaml with spec_version 1
- Typed Pydantic models for all request/response schemas
- Containerized with Docker
- Deployed to HuggingFace Spaces
- Mandatory stdout logging: `[START]`, `[STEP]`, `[END]`
- 3 graded tasks with deterministic scoring
- Baseline inference script included

## Blog

Read the detailed writeup: [FinePrint: Teaching Language Models That Knowledge Has an Expiration Date](blog.md)

> **[Live Demo & API](https://huggingface.co/spaces/PraneshkumarR/fineprint-env)** | **[Training Notebook (Colab)](https://colab.research.google.com/drive/1s6bUXezbqFNDYnQvmMs5pnQY232QXvJk?usp=sharing)**

## License

[MIT](LICENSE)

---

<div align="center">

Built for **Meta PyTorch OpenEnv Hackathon × Scaler School of Technology** — Consumer Policy Drift Detection 

</div>