README.md

FraudShield 🛡️

Production-grade OpenEnv environment for e-commerce fraud detection

FraudShield simulates real marketplace fraud review workflows. Agents inspect transactions and predict fraud/legitimate status, receiving dense rewards shaped by business impact, confidence calibration, and classification accuracy.

Key Features:

• ✅ Real-world task (marketplace fraud detection)
• ✅ Deterministic graders with 3 difficulty levels (easy → medium → hard)
• ✅ Dense reward function (business-cost sensitive)
• ✅ Frozen snapshot (reproducible, 108 cases)
• ✅ Production-ready (Docker + FastAPI)
• ✅ Baseline scores verified (0.8660 final score)

The environment is grounded in real public fraud data, but it does not fetch live records during reset() or step(). Instead, it uses a frozen, versioned snapshot stored in data/fraudshield_cases.json. That gives you real-world grounding with deterministic grading, fast Docker startup, and reproducible evaluation on Hugging Face Spaces.

Competition fit

FraudShield is designed around the Round 1 requirements:

• Real-world task: marketplace fraud review, not a toy environment
• OpenEnv interface: typed action, observation, reward, plus reset(), step(), and state()
• Three graded tasks: easy, medium, hard
• Dense reward shaping: correctness, business impact, confidence calibration, and bad-action penalties
• Baseline inference: root inference.py, OpenAI-client path for competition mode
• Docker/HF Space target: FastAPI app on port 7860
• Reproducibility: frozen snapshot data and fixed seed

Current readiness status

What has been verified locally in this repo:

• python inference.py passes
• API smoke checks for /health and /reset pass
• The snapshot bundle loads correctly
• Python import/compile sanity passes

What still must be verified on a machine with the right tooling installed:

• openenv validate openenv.yaml
• docker build and docker run
• Hugging Face router path with a valid MODEL_NAME and HF_TOKEN
• Final Hugging Face Space deployment ping

Note:

• uv.lock is checked in so the OpenEnv validator accepts the project structure on this machine
• If you have uv installed, regenerate it with uv lock before final submission

Why this design

For an OpenEnv submission, the safest pattern is:

• Fetch or refresh public source data offline
• Build a deterministic FraudShield snapshot
• Commit the snapshot used for evaluation
• Keep the environment runtime fully offline

That avoids runtime API failures, privacy issues, and non-reproducible scores.

Real-world data strategy

FraudShield currently builds its snapshot from the public Kaggle / ULB credit card fraud dataset:

• Source ID: kaggle_creditcardfraud
• Dataset: mlg-ulb/creditcardfraud
• URL: https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud

The loader is now source-agnostic in code:

• data_loader.py exposes a public-source snapshot pipeline
• download_kaggle_data.py refreshes the local source CSV and rebuilds the frozen snapshot
• fraudshield_env.py reads the snapshot only at runtime

The checked-in snapshot currently reports:

• Snapshot ID: fraudshield-realworld-v2
• Schema version: 2.0
• Seed: 42
• Task sizes: easy 24, medium 36, hard 48

Tasks

Task	Cases	Goal	What makes it hard
Easy	24	Catch obvious fraud while avoiding basic false positives	Single-transaction red flags are strong and low-noise
Medium	36	Balance fraud capture with calibration	No single signal is decisive; tradeoffs matter
Hard	48	Handle coordinated abuse and edge-case legitimate traffic	Fraud rings and flash-sale behavior intentionally overlap

Action space

Agents emit a single FraudCheckAction:

FraudCheckAction(
    transaction_id: str,
    decision: Literal["fraud", "legitimate"],
    confidence: float,  # 0.0 to 1.0
    reasoning: str,
)

Observation space

Each step returns a FraudCheckObservation with:

• Structured transaction facts such as amount, seller age, buyer age, geo mismatch, rating, prior flags, chargeback rate, shared-device counts, and address velocity
• Historical context such as seller velocity, linked cards, refund counts, cluster alert score, and source snapshot metadata
• Task metadata including difficulty and episode step

Reward design

Rewards in fraudshield_env.py are dense and cost-sensitive:

• Correct fraud catches receive the strongest positive reward
• Correct legitimate approvals still earn positive reward, but less than catching fraud
• False negatives are punished more than false positives
• Confidence is rewarded when it matches hidden case difficulty and punished when it is overconfident
• Submitting the wrong transaction_id adds an extra penalty

Graders

The three task graders in graders.py are deterministic and return scores from 0.0 to 1.0.

• Easy: accuracy, F1, recall, and specificity
• Medium: F1, ROC-AUC, precision, and confidence calibration
• Hard: recall, precision, F1, ROC-AUC, and calibration

Baseline inference

The required root script is inference.py.

• Competition mode: if API_BASE_URL, MODEL_NAME, and HF_TOKEN are set, it uses the OpenAI client against that endpoint
• Local smoke-test mode: if those variables are missing, it falls back to a deterministic heuristic agent
• If those variables are set but invalid, the script now fails loudly instead of silently switching agents

Required environment variables for the competition path:

API_BASE_URL=https://router.huggingface.co/v1
MODEL_NAME=<your-model-id>
HF_TOKEN=<your-token>

If your Hugging Face Space rejects underscores in variable names, FraudShield also accepts these aliases:

APIBASEURL=https://router.huggingface.co/v1
MODELNAME=<your-model-id>
HFTOKEN=<your-token>

Run it with:

python inference.py

The script writes fraudshield_baseline_results.json to the project root.

Tested local baseline

I reran the baseline after the snapshot-loader changes. With the deterministic heuristic fallback and seed 42, the tested local scores are:

Task	Score
Easy	1.0000
Medium	0.8773
Hard	0.7206
Final	0.8660

Project layout

fraudshield/
|-- data/
|   |-- fraudshield_cases.json
|-- server/
|   |-- __init__.py
|   `-- app.py
|-- data_loader.py
|-- download_kaggle_data.py
|-- Dockerfile
|-- fraudshield_env.py
|-- graders.py
|-- inference.py
|-- inference_llm.py
|-- llm_agent.py
|-- models.py
|-- openenv.yaml
`-- pyproject.toml

Quick Start

1. Installation

# Install dependencies
pip install -e .

# (Optional) For local data refresh
pip install -e ".[dev]"

2. Run Baseline Locally

# Heuristic agent (no API call)
python inference.py

# Expected output: fraudshield_baseline_results.json with score ≈ 0.8660

3. Deploy with Docker

# Build
docker build . -t fraudshield:v0.2.0

# Run
docker run -p 7860:7860 fraudshield:v0.2.0

# Test
curl http://localhost:7860/health

4. Hugging Face Space Deployment

1. Create Space on huggingface.co/spaces
2. Select "Docker" runtime
3. Connect your GitHub repository
4. HF automatically detects Dockerfile and deploys
5. Set environment variables (optional for LLM mode):

   API_BASE_URL=https://router.huggingface.co/v1
   MODEL_NAME=<your-model>
   HF_TOKEN=<your-token>
   

API Examples

Reset Environment

curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task":"easy"}'

Submit Action

curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{
    "transaction_id": "txn_001",
    "decision": "fraud",
    "confidence": 0.85,
    "reasoning": "High risk indicators: new seller, price anomaly"
  }'

Get Episode State

curl http://localhost:7860/state | jq .

Rebuilding Data Snapshot (Optional)

To refresh the frozen snapshot from the public Kaggle dataset:

pip install -e ".[data]"
python download_kaggle_data.py

Note: If data/creditcard.csv exists, the script rebuilds without re-downloading.

Setup

Install the project:

python -m pip install -e .

Running locally

Python API

from fraudshield_env import FraudShieldEnvironment
from models import DecisionEnum, FraudCheckAction

env = FraudShieldEnvironment(data_path="data", seed=42)
env.load_data()
reset_result = env.reset("medium")

action = FraudCheckAction(
    transaction_id=reset_result.observation.transaction_id,
    decision=DecisionEnum.LEGITIMATE,
    confidence=0.62,
    reasoning="Signals are mixed but seller history is reasonably stable.",
)

step_result = env.step(action)
print(step_result.reward.value, step_result.done)

FastAPI server

uvicorn server.app:app --host 0.0.0.0 --port 7860

Endpoints:

• GET /health
• POST /reset?task=easy|medium|hard
• POST /step
• GET /state
• GET /info
• GET /tasks

Docker

Build and run:

docker build -t fraudshield .
docker run -p 7860:7860 fraudshield

The container listens on port 7860, which matches Hugging Face Docker Spaces expectations.

Validation checklist

Before submission:

python inference.py
openenv validate openenv.yaml
docker build -t fraudshield .
docker run -p 7860:7860 fraudshield

Then verify:

• http://localhost:7860/health
• POST http://localhost:7860/reset?task=easy

What must stay private

Do not commit or publish:

• HF_TOKEN
• HFTOKEN
• OPENAI_API_KEY
• API_KEY
• kaggle.json
• .env, .env.local, or any file containing real tokens
• raw shell history or logs that include auth headers or tokens

Safe to keep public:

• API_BASE_URL
• APIBASEURL
• MODEL_NAME
• MODELNAME
• openenv.yaml
• fraudshield_baseline_results.json
• data/fraudshield_cases.json

Notes

• Runtime uses the committed snapshot only
• Public source refresh is optional and intended for offline rebuilds
• inference_llm.py remains as a thin wrapper to inference.py