Upload folder using huggingface_hub

README.md

@@ -13,17 +13,24 @@ base_path: /web
 
 # API Contract Validator — OpenEnv Environment
 
-An OpenEnv RL environment where AI agents learn to validate API request/response payloads against OpenAPI specifications. Agents identify type mismatches, missing required fields, invalid enum values, and breaking changes between API versions.
+An OpenEnv RL environment where AI agents learn to validate API request/response
+payloads against OpenAPI specifications. Agents identify type mismatches, missing
+required fields, invalid enum values, format constraint violations, breaking API
+changes, cross-field arithmetic errors, and authentication schema violations.
 
 ## Why This Environment?
 
-API contract violations are one of the **top causes of production incidents** in microservice architectures. Every API integration requires validating payloads against specs — a tedious, error-prone task that developers perform daily. This environment trains agents to automate this critical workflow.
+API contract violations are one of the **top causes of production incidents** in
+microservice architectures. Every API integration requires validating payloads
+against specs — a tedious, error-prone task that developers perform daily. This
+environment trains agents to automate this critical workflow.
 
 **Real-world applications:**
 - CI/CD pipeline contract testing
-- API gateway validation
-- SDK compatibility checking
-- Migration safety audits between API versions
+- API gateway request/response validation
+- SDK compatibility checking between API versions
+- OAuth2 and auth schema enforcement
+- Migration safety audits
 
 ## How It Works
 
@@ -31,31 +38,36 @@ Each episode presents the agent with:
 1. An **OpenAPI specification** defining expected types, required fields, and constraints
 2. A **payload** containing planted violations
 
-The agent inspects both and reports violations **one per step**. The environment grades each report against ground-truth violations and provides immediate feedback.
+The agent inspects both and reports violations **one per step**. The environment
+grades each report against ground-truth violations and provides immediate reward feedback.
 
 ```
-reset() → Agent sees spec + payload
-step(violation_report) → Correct? +1.0 | False positive? -0.3 | Duplicate? -0.1
-step(DONE) → Episode ends with completeness bonus
+reset()  →  Agent sees spec + payload with N hidden violations
+step(violation_report)  →  Correct? +1.0 | Wrong path, right type? +0.3 | False positive? -0.3 | Duplicate? -0.1
+step(HINT)  →  Receive a location clue at -0.5 cost
+step(DONE)  →  Episode ends with completeness bonus +0.5 × (found/total)
 ```
 
-## Tasks (5 difficulty levels)
+## Tasks (6 difficulty levels)
 
 | Task | Difficulty | Violations | Max Steps | What the Agent Must Find |
 |------|-----------|------------|-----------|--------------------------|
-| `find_type_mismatches` | Easy | 4 | 10 | Type mismatches, missing required fields, invalid enums at the top level |
-| `validate_nested_objects` | Medium | 7 | 15 | Violations inside nested objects and arrays — requires traversing deep structures |
-| `detect_breaking_changes` | Hard | 9 | 20 | Breaking changes between two API spec versions — type changes, removed fields, narrowed enums, new requirements |
-| `validate_response_schema` | Expert | 10 | 25 | Subtle format errors in an API response: invalid date formats, pattern mismatches, out-of-range numerics, and bad enum values scattered across nested objects and arrays |
-| `validate_cross_field_constraints` | Expert | 7 | 18 | Cross-field arithmetic and date ordering constraints on Invoice API — due_date must be after invoice_date, line totals, subtotal sum, tax calculation, discount rules for trial accounts |
+| `find_type_mismatches` | Easy | 4 | 10 | Type mismatches, missing required fields, invalid enums at the top level. Sampled from a pool of 12 — 495 unique episode combinations |
+| `validate_nested_objects` | Medium | 7 | 15 | Violations inside nested objects and arrays — requires traversing deep structures. 2 variants: Order Service / Event Booking |
+| `detect_breaking_changes` | Hard | 9 | 20 | Breaking changes between two API spec versions — type changes, removed fields, narrowed enums, new required fields |
+| `validate_response_schema` | Expert | 10 | 25 | Subtle format errors in an API response: invalid date formats, pattern mismatches, out-of-range numerics, bad enum values. 2 variants |
+| `validate_cross_field_constraints` | Expert | 7 | 18 | Cross-field arithmetic and date ordering on Invoice API — line totals, subtotal sum, tax calculation, discount rules for trial accounts |
+| `validate_auth_request` | Expert | 6 | 14 | OAuth2 token and API key management violations — invalid grant types, bad scopes, MFA token patterns, IP format, rate limits. 2 variants |
 
 ### Randomised Episode Generation
 
-All tasks support seed-based randomisation, making the environment suitable for **training** agents, not just evaluating them:
+All tasks support seed-based randomisation, making the environment suitable for
+**training** (varied seeds) as well as **evaluation** (fixed seeds):
 
-- `find_type_mismatches` — samples 4 violations from a pool of 8 (70 unique combinations)
-- `validate_nested_objects` — two complete scenario variants (Order Service / Event Booking)
-- `validate_response_schema` — two complete scenario variants with different violation sets
+- `find_type_mismatches` — samples 4 from a pool of 12 violations (495 unique combinations)
+- `validate_nested_objects` — 2 complete scenario variants (Order Service / Event Booking)
+- `validate_response_schema` — 2 complete scenario variants with different violation sets
+- `validate_auth_request` — 2 complete scenario variants (OAuth2 / API key management)
 - Pass `seed` in the `reset()` call to select a deterministic episode
 
 ## Action Space
@@ -64,8 +76,8 @@ Each step the agent submits a `ValidatorAction`:
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `field_path` | `str` | Dot-notation path to the violated field (e.g., `customer.email`, `items[1].quantity`). Use `DONE` to end. |
-| `violation_type` | `str` | One of: `type_mismatch`, `missing_required`, `invalid_enum`, `format_error`, `extra_field`, `breaking_change` |
+| `field_path` | `str` | Dot-notation path to the violated field (e.g. `customer.email`, `items[1].quantity`). Special values: `DONE` to end episode, `HINT` for a location clue |
+| `violation_type` | `str` | One of: `type_mismatch`, `missing_required`, `invalid_enum`, `format_error`, `extra_field`, `breaking_change`, `cross_field_constraint` |
 | `description` | `str` | Human-readable explanation of the violation |
 | `suggested_fix` | `str` | Optional suggested correction |
 
@@ -76,28 +88,30 @@ After each step the agent receives a `ValidatorObservation`:
 | Field | Type | Description |
 |-------|------|-------------|
 | `task_name` | `str` | Current task identifier |
-| `task_description` | `str` | Natural-language instructions |
+| `task_description` | `str` | Natural-language instructions for the agent |
 | `api_spec` | `dict` | The OpenAPI specification (or version diff for hard task) |
-| `payload` | `dict` | The payload to validate |
+| `payload` | `dict` | The API payload to validate |
 | `violations_found` | `list[dict]` | Violations correctly identified so far |
-| `violations_remaining` | `int` | How many planted violations are still undetected |
+| `violations_remaining` | `int` | Number of planted violations still undetected |
 | `feedback` | `str` | Result of the last submitted report |
-| `max_steps` | `int` | Step budget for the episode |
+| `max_steps` | `int` | Step budget for this episode |
 | `done` | `bool` | Whether the episode has ended |
 | `reward` | `float` | Reward for the last action |
 
 ## Reward Function
 
-The reward function provides **partial progress signals** — not binary end-of-episode scoring:
+Partial progress signals — not binary end-of-episode scoring:
 
 | Event | Reward | Rationale |
 |-------|--------|-----------|
-| Correct violation found | +1.0 | Primary incentive — each discovery is rewarded |
-| False positive | -0.3 | Penalises guessing without being too harsh |
-| Duplicate report | -0.1 | Light penalty — agent should track what it already reported |
-| DONE signal | +0.5 × (found/total) | Bonus proportional to completeness |
+| Correct violation (path + type match) | **+1.0** | Primary incentive |
+| Proximity match (right path, wrong type) | **+0.3** | Encourages finding the right field first |
+| HINT requested | **−0.5** | Informative but expensive |
+| Duplicate report | **−0.1** | Light penalty — track what you already found |
+| False positive | **−0.3** | Penalises guessing |
+| DONE signal | **+0.5 × (found/total)** | Completeness bonus |
 
-**Final score** = `correct_violations / total_violations` ∈ [0.0, 1.0]
+**Final score** = `correct_violations / total_violations` clamped to `(0.01, 0.99)`
 
 ## Setup
 
@@ -138,9 +152,6 @@ python inference.py
 
 ```bash
 openenv validate
-
-# Full pre-submission check
-./validate-submission.sh https://your-space.hf.space
 ```
 
 ## Baseline Scores
@@ -152,6 +163,7 @@ openenv validate
 | `detect_breaking_changes` | Qwen2.5-72B-Instruct | ~0.44 | 12–18 |
 | `validate_response_schema` | Qwen2.5-72B-Instruct | ~0.40 | 15–22 |
 | `validate_cross_field_constraints` | Qwen2.5-72B-Instruct | ~0.43 | 10–16 |
+| `validate_auth_request` | Qwen2.5-72B-Instruct | ~0.60 | 8–12 |
 
 *Scores are approximate and may vary with temperature/sampling.*
 
@@ -160,20 +172,20 @@ openenv validate
 ```
 api_contract_validator/
 ├── openenv.yaml              # OpenEnv manifest
-├── pyproject.toml             # Python project metadata
-├── Dockerfile                 # Container definition
-├── inference.py               # Baseline inference script
-├── README.md                  # This file
-├── models.py                  # Pydantic models (Action, Observation, State)
-├── client.py                  # WebSocket client (EnvClient subclass)
-├── __init__.py                # Package exports
+├── pyproject.toml            # Python project metadata
+├── Dockerfile                # Container definition
+├── inference.py              # Baseline inference script
+├── README.md                 # This file
+├── models.py                 # Pydantic models (Action, Observation, State)
+├── client.py                 # WebSocket client (EnvClient subclass)
+├── __init__.py               # Package exports
 └── server/
     ├── __init__.py
-    ├── app.py                 # FastAPI wiring (create_app)
-    ├── environment.py         # Core environment logic (reset/step/state)
-    ├── spec_generator.py      # Task scenarios with planted violations
-    ├── rewards.py             # Reward computation
-    └── requirements.txt       # Server dependencies
+    ├── app.py                # FastAPI wiring (create_app)
+    ├── environment.py        # Core environment logic (reset/step/state)
+    ├── spec_generator.py     # Task scenarios with planted violations
+    ├── rewards.py            # Reward computation
+    └── requirements.txt      # Server dependencies
 ```
 
 ## License

server/rewards.py

@@ -2,17 +2,19 @@
 Reward computation for the API Contract Validator Environment.
 
 Provides partial-progress reward signals rather than binary end-of-episode
-scoring.  The reward function has several interesting properties:
-
-  - Correct violation found          → +1.0  (primary incentive)
-  - Path-only match (wrong type)     → +0.3  (proximity signal — learn location first)
-  - HINT requested                   → -0.5  (expensive but informative)
-  - Duplicate report                 → -0.1  (light penalty, track what you've found)
-  - False positive                   → -0.3  (penalise guessing)
-  - DONE signal                      → +0.5 × (found/total)  (completeness bonus)
-
-The proximity reward creates a richer gradient: agents learn to locate the
-right field first, then refine their violation classification.
+scoring, creating a richer gradient for RL training:
+
+  - Correct violation found      → +1.0   primary incentive, one reward per unique violation
+  - Path-only match (wrong type) → +0.3   proximity signal — agent found the right field,
+                                           now needs to classify it correctly; each path
+                                           earns proximity reward at most once
+  - HINT requested               → -0.5   informative but expensive; use sparingly
+  - Duplicate report             → -0.1   light penalty — agent should track findings
+  - False positive               → -0.3   penalises random guessing
+  - DONE signal                  → +0.5 × (found/total)   completeness bonus on exit
+
+Final episode score = correct / total, clamped to (0.01, 0.99) so it is
+always strictly between 0 and 1 as required by the OpenEnv eval pipeline.
 """
 
 from dataclasses import dataclass

server/spec_generator.py

@@ -2,15 +2,22 @@
 Spec generator for the API Contract Validator Environment.
 
 Generates OpenAPI specifications, payloads with planted violations,
-and ground-truth violation records for four difficulty levels.
+and ground-truth violation records for six difficulty levels:
+
+    find_type_mismatches        Easy   — 4 violations from pool of 12 (495 combos)
+    validate_nested_objects     Medium — 7 violations, 2 scenario variants
+    detect_breaking_changes     Hard   — 9 breaking changes between v1 and v2
+    validate_response_schema    Expert — 10 format/constraint violations, 2 variants
+    validate_cross_field_constraints Expert — 7 arithmetic + date cross-field errors
+    validate_auth_request       Expert — 6 auth/security violations, 2 variants
 
 Each generator accepts an optional *seed* for deterministic randomisation:
-- seed=None  → fixed canonical scenario (backward-compatible)
-- seed=int   → reproducible randomised variant
+    seed=None  → fixed canonical scenario (backward-compatible defaults)
+    seed=int   → reproducible randomised variant (use for RL training)
 
 This makes the environment suitable for both evaluation (fixed seed) and
-training (varied seeds), which is the key distinction between a one-shot
-evaluator and a genuine RL training environment.
+training (varied seeds) — the key distinction between a one-shot evaluator
+and a genuine RL training environment.
 """
 
 import random