Spaces:

infoclassifier
/

InfoClassifier

Sleeping

UTKARSH MISHRA

Initial commit: InfoClassifier project

e5e36db 7 months ago

11.3 kB

---
title: InformationIntegrityEnv
emoji: 🔍
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.11.0
app_file: app.py
pinned: false
license: mit
tags:
  - openenv
  - content-moderation
  - rl-environment
  - agent-benchmark
---
# InformationIntegrityEnv

A **deterministic, benchmarkable OpenEnv environment** for evaluating LLM agents on reasoning-based content moderation.

Agents must verify claims, weigh conflicting evidence, and make explainable moderation decisions (`allow` / `flag` / `remove`). Three difficulty tiers — easy, medium, hard — provide 15 fully static scenarios that run completely offline.

---

## Architecture

```
tasks/
  easy/scenarios.json     ← 5 static scenarios (clear-cut misinformation)
  medium/scenarios.json   ← 5 static scenarios (misleading / mixed evidence)
  hard/scenarios.json     ← 5 static scenarios (conflicting evidence, nuanced)
  graders.py              ← EasyGrader / MediumGrader / HardGrader
  registry.py             ← TASKS["easy"] / TASKS["medium"] / TASKS["hard"]
        │
        ▼
InformationIntegrityEnv         (environment.py)
  reset(difficulty, scenario_id) → Observation
  step(action)                   → (Observation, Reward, done, info)
  state()                        → full debug state
        │
  ┌─────┴──────┐
  │            │
inference.py   server.py
(benchmark     (FastAPI
 agent loop)    REST API)
```

No external API calls occur during evaluation. `DataCollector` is only used to build/refresh datasets.

---

## Quick Start

### 1. Install

```bash
pip install -r requirements.txt
```

### 2. Set API keys (for the LLM agent — not needed for just the server)

```bash
export HF_TOKEN=sk-...          # or OPENAI_API_KEY
export MODEL_NAME=gpt-4o
export API_BASE_URL=https://api.openai.com/v1
```

### 3. Run the full benchmark

```bash
python inference.py --benchmark --out results.json
```

This loops over all 15 scenarios (5 per difficulty) and prints:

```
[START] episode=E01 difficulty=easy claims=1 sources=5
[STEP]  step=1 action=check_source reward=+0.200 total=+0.200 done=False ...
[STEP]  step=2 action=verify_claim  reward=+0.200 total=+0.400 done=False ...
[STEP]  step=3 action=decide        reward=+1.000 total=+1.400 done=True  ...
[END]   episode=E01 steps=3 decision=remove total_reward=+1.4000 grader_score=0.8750
...
difficulty=easy   avg_score=0.7800  avg_reward=1.2000  n=5
difficulty=medium avg_score=0.6300  avg_reward=0.9000  n=5
difficulty=hard   avg_score=0.5200  avg_reward=0.6500  n=5
Difficulty tiers well separated: easy > medium > hard ✓
```

### 4. Single episode (local, no server)

```bash
# Random scenario from medium dataset
python inference.py --local --difficulty medium
# Specific scenario by index
python inference.py --local --difficulty hard --scenario-id 2
# Specific scenario by id
python inference.py --local --difficulty easy --scenario-id easy_001
# From a custom JSON file (DataCollector output)
python inference.py --local --scenario my_scenario.json
```

### 5. Start the API server

```bash
python server.py
# or
uvicorn server:app --reload
```

---

## Action Space

| Action | Required fields | Description |
|--------|----------------|-------------|
| `verify_claim` | `claim_id` | Investigate a claim; returns true / false / partially_true / unknown |
| `check_source` | `source_id` | Read a source; returns title, text, stance, reliability |
| `decide` | `decision` | Final call: `allow` / `flag` / `remove` — ends episode |
All actions accept an optional `reasoning` string (logged, not graded).
**Minimum evidence required before deciding** (or premature penalty applies):
- ≥ 1 claim verified
- ≥ 2 sources checked
---
## Observation Space
```json
{
  "content": "The post text under review",
  "claims": [
    {"id": "claim_001", "text": "...", "verification_status": "unverified", "confidence": 0.0}
  ],
  "available_sources": [
    {"id": "src_001", "title": "...", "text": "...", "url": "...",
     "stance": "contradicting", "reliability": 0.97, "source_type": "official"}
  ],
  "interaction_history": [],
  "status_flags": {
    "decision_made": false,
    "sufficient_evidence": false,
    "premature_decision": false,
    "redundant_actions": 0,
    "max_steps_reached": false
  },
  "step_count": 0,
  "task_difficulty": "medium",
  "episode_id": "uuid"
}
```
---

## Reward Function

| Component | Value | Trigger |
|-----------|-------|---------|
| Verify claim | +0.20 | Per unique claim verified |
| Check source | +0.20 | Per unique source checked |
| Multi-source bonus | +0.20 | When ≥ 3 unique sources checked |
| Correct decision | +1.00 | Exact ground truth match |
| Partial decision | +0.50 | One tier off (flag↔remove, flag↔allow) |
| Premature decision | −0.50 | Deciding before sufficient evidence |
| Redundant action | −0.10 | Re-checking same claim/source |
| Inefficiency | −0.05 | Per step beyond efficient threshold |
| Over-aggression | −0.30 | `remove` when ground truth is `allow`/`flag` |

Total clipped to `[-2.0, +2.0]`. All values configurable in `reward_config.yaml`.

---

## Grader Scoring

Each episode is scored `[0.0, 1.0]` across four dimensions:

| Dimension | Easy | Medium | Hard |
|-----------|------|--------|------|
| Decision correctness | 50% | 35% | 30% |
| Evidence breadth | 25% | 35% | 30% |
| Reasoning depth | 15% | 20% | 30% |
| Efficiency | 10% | 10% | 10% |

---

## Task Datasets

### Easy (5 scenarios)
Clear-cut false claims contradicted by multiple authoritative sources (WHO, CDC, NASA, FDA).
Ground truth: 3× remove, 1× flag, 1× allow.

| ID | Topic | Ground Truth |
|----|-------|-------------|
| easy_001 | Bleach cures COVID | remove |
| easy_002 | NASA fabricated climate data | remove |
| easy_003 | mRNA vaccines alter DNA | remove |
| easy_004 | Sydney is Australia's capital | flag |
| easy_005 | Webb telescope early universe | allow |
### Medium (5 scenarios)
Partially true or misleadingly framed content. Mixed evidence requires weighing.
Ground truth: 4× flag, 1× allow.
| ID | Topic | Ground Truth |
|----|-------|-------------|
| medium_001 | Sugar more addictive than cocaine | flag |
| medium_002 | Real unemployment is 20%+ | flag |
| medium_003 | AI will eliminate 85M jobs | flag |
| medium_004 | Pharma knew OxyContin was addictive | allow |
| medium_005 | Facebook hid teen mental health data | flag |

### Hard (5 scenarios)
Conflicting credible evidence, genuinely contested claims, contested policy debates.
Ground truth: 5× flag.

| ID | Topic | Ground Truth |
|----|-------|-------------|
| hard_001 | NATO provoked Ukraine invasion | flag |
| hard_002 | Lockdowns killed more than COVID | flag |
| hard_003 | SEC crackdown is banking conspiracy | flag |
| hard_004 | AI poses extinction-level risk | flag |
| hard_005 | British colonialism net beneficial | flag |
---
## API Reference
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/health` | Liveness probe |
| GET | `/tasks` | List all available scenario datasets |
| POST | `/reset` | Start a new episode |
| POST | `/step` | Submit one action |
| GET | `/state/{session_id}` | Full internal state (debug/audit) |
| GET | `/scenario/{session_id}` | Scenario metadata (no ground truth) |
| DELETE | `/session/{session_id}` | End session and free memory |
| POST | `/collect` | Build a live scenario via DataCollector (dataset creation only) |

### Reset — evaluation path (no DataCollector needed)

```bash
# Random scenario from easy dataset
curl -X POST http://localhost:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"difficulty": "easy"}'
# Specific scenario by index
curl -X POST http://localhost:8000/reset \
  -d '{"difficulty": "medium", "scenario_id": 2}'
# Specific scenario by id
curl -X POST http://localhost:8000/reset \
  -d '{"difficulty": "hard", "scenario_id": "hard_001"}'
```

### Full episode example

```bash
SESSION=$(curl -s -X POST http://localhost:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"difficulty": "easy"}' | jq -r .session_id)
# Check a source
curl -X POST http://localhost:8000/step \
  -d "{\"session_id\": \"$SESSION\", \"action\": {\"type\": \"check_source\", \"source_id\": \"src_001\"}}"
# Verify a claim
curl -X POST http://localhost:8000/step \
  -d "{\"session_id\": \"$SESSION\", \"action\": {\"type\": \"verify_claim\", \"claim_id\": \"claim_001\"}}"
# Decide
curl -X POST http://localhost:8000/step \
  -d "{\"session_id\": \"$SESSION\", \"action\": {\"type\": \"decide\", \"decision\": \"remove\"}}"
```

---

## Docker

### Server (default)

```bash
docker build -t info-integrity-env .
docker run --env-file .env -p 8000:8000 info-integrity-env
```

### Benchmark (offline — no external services needed)

```bash
docker run --env-file .env info-integrity-env \
  python inference.py --benchmark --out /tmp/results.json
```

### Full stack with Redis (for live data collection)

```bash
docker compose up --build
```

---

## Baseline Results

| Difficulty | Avg Grader Score | Avg Reward | Decision Accuracy |
|-----------|-----------------|------------|-----------------|
| Easy | ~0.78 | ~1.20 | ~85% |
| Medium | ~0.63 | ~0.90 | ~70% |
| Hard | ~0.52 | ~0.65 | ~55% |

*Results vary by LLM. Scores above from `gpt-4o` with temperature=0.*

---

## Extending the Environment

| File | What to change |
|------|---------------|
| `tasks/{difficulty}/scenarios.json` | Add or edit scenarios (run `openenv validate` after) |
| `reward_config.yaml` | Tune reward/penalty magnitudes |
| `tasks/graders.py` | Register custom graders: `GraderFactory.register("hard_v2", MyGrader)` |
| `data_collector.py` | Add new search backends via `SearchBackend` ABC |
| `domain_reliability.yaml` | Add trusted domains with custom reliability scores |

---

## Project Structure

```
├── models.py                    Pydantic models (Observation, Action, Reward, Scenario, ...)
├── environment.py               InformationIntegrityEnv (OpenEnv core)
├── reward.py                    Configurable dense reward function
├── explainability.py            Structured reasoning + template-based explanation
├── data_collector.py            Multi-backend data collection (dataset creation only)
├── inference.py                 Benchmark agent + single-episode runner
├── server.py                    FastAPI server (HF Space / Docker)
├── tasks/
│   ├── __init__.py
│   ├── graders.py               Easy / Medium / Hard deterministic graders
│   ├── registry.py              TASKS registry — lazy-loads scenario datasets
│   ├── easy/
│   │   └── scenarios.json       5 easy scenarios
│   ├── medium/
│   │   └── scenarios.json       5 medium scenarios
│   └── hard/
│       └── scenarios.json       5 hard scenarios
├── openenv.yaml                 OpenEnv v1.1.0 specification
├── reward_config.yaml           Reward magnitudes (runtime-configurable)
├── domain_reliability.yaml      Domain trust scores for DataCollector
├── Dockerfile                   Production container
├── docker-compose.yml           Local dev stack (server + Redis)
└── requirements.txt
```

	---
	title: InformationIntegrityEnv
	emoji: 🔍
	colorFrom: blue
	colorTo: green
	sdk: gradio
	sdk_version: 6.11.0
	app_file: app.py
	pinned: false
	license: mit
	tags:
	- openenv
	- content-moderation
	- rl-environment
	- agent-benchmark
	---

	# InformationIntegrityEnv

	A deterministic, benchmarkable OpenEnv environment for evaluating LLM agents on reasoning-based content moderation.

	Agents must verify claims, weigh conflicting evidence, and make explainable moderation decisions (`allow` / `flag` / `remove`). Three difficulty tiers — easy, medium, hard — provide 15 fully static scenarios that run completely offline.

	---

	## Architecture

	```
	tasks/
	easy/scenarios.json ← 5 static scenarios (clear-cut misinformation)
	medium/scenarios.json ← 5 static scenarios (misleading / mixed evidence)
	hard/scenarios.json ← 5 static scenarios (conflicting evidence, nuanced)
	graders.py ← EasyGrader / MediumGrader / HardGrader
	registry.py ← TASKS["easy"] / TASKS["medium"] / TASKS["hard"]
	│
	▼
	InformationIntegrityEnv (environment.py)
	reset(difficulty, scenario_id) → Observation
	step(action) → (Observation, Reward, done, info)
	state() → full debug state
	│
	┌─────┴──────┐
	│ │
	inference.py server.py
	(benchmark (FastAPI
	agent loop) REST API)
	```

	No external API calls occur during evaluation. `DataCollector` is only used to build/refresh datasets.

	---

	## Quick Start

	### 1. Install

	```bash
	pip install -r requirements.txt
	```

	### 2. Set API keys (for the LLM agent — not needed for just the server)

	```bash
	export HF_TOKEN=sk-... # or OPENAI_API_KEY
	export MODEL_NAME=gpt-4o
	export API_BASE_URL=https://api.openai.com/v1
	```

	### 3. Run the full benchmark

	```bash
	python inference.py --benchmark --out results.json
	```

	This loops over all 15 scenarios (5 per difficulty) and prints:

	```
	[START] episode=E01 difficulty=easy claims=1 sources=5
	[STEP] step=1 action=check_source reward=+0.200 total=+0.200 done=False ...
	[STEP] step=2 action=verify_claim reward=+0.200 total=+0.400 done=False ...
	[STEP] step=3 action=decide reward=+1.000 total=+1.400 done=True ...
	[END] episode=E01 steps=3 decision=remove total_reward=+1.4000 grader_score=0.8750

	...

	difficulty=easy avg_score=0.7800 avg_reward=1.2000 n=5
	difficulty=medium avg_score=0.6300 avg_reward=0.9000 n=5
	difficulty=hard avg_score=0.5200 avg_reward=0.6500 n=5
	Difficulty tiers well separated: easy > medium > hard ✓
	```

	### 4. Single episode (local, no server)

	```bash
	# Random scenario from medium dataset
	python inference.py --local --difficulty medium

	# Specific scenario by index
	python inference.py --local --difficulty hard --scenario-id 2

	# Specific scenario by id
	python inference.py --local --difficulty easy --scenario-id easy_001

	# From a custom JSON file (DataCollector output)
	python inference.py --local --scenario my_scenario.json
	```

	### 5. Start the API server

	```bash
	python server.py
	# or
	uvicorn server:app --reload
	```

	---

	## Action Space

	\| Action \| Required fields \| Description \|
	\|--------\|----------------\|-------------\|
	\| `verify_claim` \| `claim_id` \| Investigate a claim; returns true / false / partially_true / unknown \|
	\| `check_source` \| `source_id` \| Read a source; returns title, text, stance, reliability \|
	\| `decide` \| `decision` \| Final call: `allow` / `flag` / `remove` — ends episode \|

	All actions accept an optional `reasoning` string (logged, not graded).

	Minimum evidence required before deciding (or premature penalty applies):
	- ≥ 1 claim verified
	- ≥ 2 sources checked

	---

	## Observation Space

	```json
	{
	"content": "The post text under review",
	"claims": [
	{"id": "claim_001", "text": "...", "verification_status": "unverified", "confidence": 0.0}
	],
	"available_sources": [
	{"id": "src_001", "title": "...", "text": "...", "url": "...",
	"stance": "contradicting", "reliability": 0.97, "source_type": "official"}
	],
	"interaction_history": [],
	"status_flags": {
	"decision_made": false,
	"sufficient_evidence": false,
	"premature_decision": false,
	"redundant_actions": 0,
	"max_steps_reached": false
	},
	"step_count": 0,
	"task_difficulty": "medium",
	"episode_id": "uuid"
	}
	```

	---

	## Reward Function

	\| Component \| Value \| Trigger \|
	\|-----------\|-------\|---------\|
	\| Verify claim \| +0.20 \| Per unique claim verified \|
	\| Check source \| +0.20 \| Per unique source checked \|
	\| Multi-source bonus \| +0.20 \| When ≥ 3 unique sources checked \|
	\| Correct decision \| +1.00 \| Exact ground truth match \|
	\| Partial decision \| +0.50 \| One tier off (flag↔remove, flag↔allow) \|
	\| Premature decision \| −0.50 \| Deciding before sufficient evidence \|
	\| Redundant action \| −0.10 \| Re-checking same claim/source \|
	\| Inefficiency \| −0.05 \| Per step beyond efficient threshold \|
	\| Over-aggression \| −0.30 \| `remove` when ground truth is `allow`/`flag` \|

	Total clipped to `[-2.0, +2.0]`. All values configurable in `reward_config.yaml`.

	---

	## Grader Scoring

	Each episode is scored `[0.0, 1.0]` across four dimensions:

	\| Dimension \| Easy \| Medium \| Hard \|
	\|-----------\|------\|--------\|------\|
	\| Decision correctness \| 50% \| 35% \| 30% \|
	\| Evidence breadth \| 25% \| 35% \| 30% \|
	\| Reasoning depth \| 15% \| 20% \| 30% \|
	\| Efficiency \| 10% \| 10% \| 10% \|

	---

	## Task Datasets

	### Easy (5 scenarios)
	Clear-cut false claims contradicted by multiple authoritative sources (WHO, CDC, NASA, FDA).
	Ground truth: 3× remove, 1× flag, 1× allow.

	\| ID \| Topic \| Ground Truth \|
	\|----\|-------\|-------------\|
	\| easy_001 \| Bleach cures COVID \| remove \|
	\| easy_002 \| NASA fabricated climate data \| remove \|
	\| easy_003 \| mRNA vaccines alter DNA \| remove \|
	\| easy_004 \| Sydney is Australia's capital \| flag \|
	\| easy_005 \| Webb telescope early universe \| allow \|

	### Medium (5 scenarios)
	Partially true or misleadingly framed content. Mixed evidence requires weighing.
	Ground truth: 4× flag, 1× allow.

	\| ID \| Topic \| Ground Truth \|
	\|----\|-------\|-------------\|
	\| medium_001 \| Sugar more addictive than cocaine \| flag \|
	\| medium_002 \| Real unemployment is 20%+ \| flag \|
	\| medium_003 \| AI will eliminate 85M jobs \| flag \|
	\| medium_004 \| Pharma knew OxyContin was addictive \| allow \|
	\| medium_005 \| Facebook hid teen mental health data \| flag \|

	### Hard (5 scenarios)
	Conflicting credible evidence, genuinely contested claims, contested policy debates.
	Ground truth: 5× flag.

	\| ID \| Topic \| Ground Truth \|
	\|----\|-------\|-------------\|
	\| hard_001 \| NATO provoked Ukraine invasion \| flag \|
	\| hard_002 \| Lockdowns killed more than COVID \| flag \|
	\| hard_003 \| SEC crackdown is banking conspiracy \| flag \|
	\| hard_004 \| AI poses extinction-level risk \| flag \|
	\| hard_005 \| British colonialism net beneficial \| flag \|

	---

	## API Reference

	\| Method \| Endpoint \| Description \|
	\|--------\|----------\|-------------\|
	\| GET \| `/health` \| Liveness probe \|
	\| GET \| `/tasks` \| List all available scenario datasets \|
	\| POST \| `/reset` \| Start a new episode \|
	\| POST \| `/step` \| Submit one action \|
	\| GET \| `/state/{session_id}` \| Full internal state (debug/audit) \|
	\| GET \| `/scenario/{session_id}` \| Scenario metadata (no ground truth) \|
	\| DELETE \| `/session/{session_id}` \| End session and free memory \|
	\| POST \| `/collect` \| Build a live scenario via DataCollector (dataset creation only) \|

	### Reset — evaluation path (no DataCollector needed)

	```bash
	# Random scenario from easy dataset
	curl -X POST http://localhost:8000/reset \
	-H "Content-Type: application/json" \
	-d '{"difficulty": "easy"}'

	# Specific scenario by index
	curl -X POST http://localhost:8000/reset \
	-d '{"difficulty": "medium", "scenario_id": 2}'

	# Specific scenario by id
	curl -X POST http://localhost:8000/reset \
	-d '{"difficulty": "hard", "scenario_id": "hard_001"}'
	```

	### Full episode example

	```bash
	SESSION=$(curl -s -X POST http://localhost:8000/reset \
	-H "Content-Type: application/json" \
	-d '{"difficulty": "easy"}' \| jq -r .session_id)

	# Check a source
	curl -X POST http://localhost:8000/step \
	-d "{\"session_id\": \"$SESSION\", \"action\": {\"type\": \"check_source\", \"source_id\": \"src_001\"}}"

	# Verify a claim
	curl -X POST http://localhost:8000/step \
	-d "{\"session_id\": \"$SESSION\", \"action\": {\"type\": \"verify_claim\", \"claim_id\": \"claim_001\"}}"

	# Decide
	curl -X POST http://localhost:8000/step \
	-d "{\"session_id\": \"$SESSION\", \"action\": {\"type\": \"decide\", \"decision\": \"remove\"}}"
	```

	---

	## Docker

	### Server (default)

	```bash
	docker build -t info-integrity-env .
	docker run --env-file .env -p 8000:8000 info-integrity-env
	```

	### Benchmark (offline — no external services needed)

	```bash
	docker run --env-file .env info-integrity-env \
	python inference.py --benchmark --out /tmp/results.json
	```

	### Full stack with Redis (for live data collection)

	```bash
	docker compose up --build
	```

	---

	## Baseline Results

	\| Difficulty \| Avg Grader Score \| Avg Reward \| Decision Accuracy \|
	\|-----------\|-----------------\|------------\|-----------------\|
	\| Easy \| ~0.78 \| ~1.20 \| ~85% \|
	\| Medium \| ~0.63 \| ~0.90 \| ~70% \|
	\| Hard \| ~0.52 \| ~0.65 \| ~55% \|

	Results vary by LLM. Scores above from `gpt-4o` with temperature=0.

	---

	## Extending the Environment

	\| File \| What to change \|
	\|------\|---------------\|
	\| `tasks/{difficulty}/scenarios.json` \| Add or edit scenarios (run `openenv validate` after) \|
	\| `reward_config.yaml` \| Tune reward/penalty magnitudes \|
	\| `tasks/graders.py` \| Register custom graders: `GraderFactory.register("hard_v2", MyGrader)` \|
	\| `data_collector.py` \| Add new search backends via `SearchBackend` ABC \|
	\| `domain_reliability.yaml` \| Add trusted domains with custom reliability scores \|

	---

	## Project Structure

	```
	├── models.py Pydantic models (Observation, Action, Reward, Scenario, ...)
	├── environment.py InformationIntegrityEnv (OpenEnv core)
	├── reward.py Configurable dense reward function
	├── explainability.py Structured reasoning + template-based explanation
	├── data_collector.py Multi-backend data collection (dataset creation only)
	├── inference.py Benchmark agent + single-episode runner
	├── server.py FastAPI server (HF Space / Docker)
	├── tasks/
	│ ├── __init__.py
	│ ├── graders.py Easy / Medium / Hard deterministic graders
	│ ├── registry.py TASKS registry — lazy-loads scenario datasets
	│ ├── easy/
	│ │ └── scenarios.json 5 easy scenarios
	│ ├── medium/
	│ │ └── scenarios.json 5 medium scenarios
	│ └── hard/
	│ └── scenarios.json 5 hard scenarios
	├── openenv.yaml OpenEnv v1.1.0 specification
	├── reward_config.yaml Reward magnitudes (runtime-configurable)
	├── domain_reliability.yaml Domain trust scores for DataCollector
	├── Dockerfile Production container
	├── docker-compose.yml Local dev stack (server + Redis)
	└── requirements.txt
	```