| --- |
| title: Contract Validation Environment Server |
| emoji: 📝 |
| colorFrom: blue |
| colorTo: green |
| sdk: docker |
| pinned: false |
| app_port: 8000 |
| base_path: /web |
| tags: |
| - openenv |
| --- |
| |
| # Contract Validation Environment |
|
|
| The **Contract Validation Environment** is an OpenEnv-compliant RL and LLM benchmark designed to test an agent's ability to act as a precise legal assistant. The agent must review various contract clauses, identify specific legal risks (e.g., liability, termination, payment), and correctly flag them without generating false positives on standard, safe clauses. |
|
|
| ## 🚀 Motivation |
| Legal contract review is a massive industry, but off-the-shelf LLMs often struggle with "alert fatigue"—flagging everything as a risk. This environment challenges agents to precisely isolate genuine liabilities across varying difficulty levels while explicitly rewarding speed and accuracy. |
|
|
| --- |
|
|
| ## 🎯 Tasks & Difficulty |
|
|
| The environment features 3 deterministic tasks with increasing complexity: |
|
|
| 1. **Easy:** 1 clause. Contains a single, explicit liability risk. Tests basic risk identification. |
| 2. **Medium:** 3 clauses. Requires identifying payment and termination risks while actively ignoring a safe governing-law distractor clause. |
| 3. **Hard:** 5 clauses. A complex mix of confidentiality, liability, and compliance risks interspersed with dense, safe, standard boilerplate clauses. Challenges frontier models to avoid false positives. |
|
|
| --- |
|
|
| ## 📊 Environment Details |
|
|
| ### Action Space (`ContractValidationAction`) |
| - `clause_id` (int): The ID of the clause being reviewed (Set to 0 if submitting final). |
| - `risk_type` (str): The identified risk (e.g., 'liability', 'payment', 'termination', 'confidentiality', 'compliance', or 'none'). |
| - `submit_final` (bool): Set to `True` when the agent has finished flagging risks to end the episode and receive a final score. |
| - `explanation` (str): The agent's chain-of-thought or reasoning for the decision. |
|
|
| ### Observation Space (`ContractValidationObservation`) |
| - `task_level` (str): Difficulty level of the current task ("easy", "medium", "hard"). |
| - `contract_clauses` (list): List of dictionaries containing the `id` and `text` of the contract clauses to review. |
| - `flagged_risks` (dict): A dictionary mapping clause IDs to the risks currently flagged by the agent. |
| - `step_count` (int): Number of steps taken in the current episode. |
| - `reward` (float): The reward delta granted for the most recent action. |
| - `done` (bool): Whether the episode has concluded. |
| - `info` (dict): Additional environment info, including the current internal `score` from the grader. |
|
|
| ### Reward Function & Grader |
| The environment utilizes a **trajectory-based reward system**. The grader calculates a score between `0.0` and `1.0` based on precision and recall. |
| * **Positive Reward:** Granted for newly correct flags. |
| * **Negative Penalty:** Applied for flagging safe clauses or assigning the wrong risk type. |
| * **Step Penalty:** A `-0.02` penalty is applied per step to encourage the agent to evaluate the contract efficiently. Rewards are clamped to `max(0.0)` to ensure compatibility with OpenEnv graders. |
| * **Completion Bonus:** A `+0.5` bonus is awarded if the agent submits the contract with a perfect 1.0 grader score. |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ``` |
| contract_validation/ |
| ├── .dockerignore # Docker build exclusions |
| ├── .env # Local environment variables (API keys - DO NOT COMMIT) |
| ├── .gitignore # Git tracking exclusions (ignores .env, caches, etc.) |
| ├── __init__.py # Module exports |
| ├── README.md # Project documentation (with tags: - openenv) |
| ├── openenv.yaml # OpenEnv manifest |
| ├── pyproject.toml # Project metadata and dependencies |
| ├── uv.lock # Locked dependencies (generated) |
| ├── client.py # ContractValidationEnv client |
| ├── inference.py # Evaluation script for the OpenEnv grader (JSON logging) |
| ├── models.py # Action and Observation Pydantic models |
| ├── Dockerfile # Container image definition |
| └── server/ |
| ├── __init__.py # Server module exports |
| ├── contract_validation_environment.py # Core environment logic and task data |
| └── app.py # FastAPI application (HTTP + WebSocket endpoints) |
| ``` |
|
|
