REWARD_LOGIC.md

TalkingHeadBench Reward Structure

Overview

TalkingHeadBench evaluates diagnostic reasoning across 3 coupled sub-environments. The final reward is a weighted composite:

final_reward = 0.25 * subenv1_score + 0.35 * subenv2_score + 0.40 * subenv3_score

This 0.25/0.35/0.40 composite is the full pipeline scoring formula used by src/pipeline.py when all three sub-environments are evaluated together.

OpenEnv Mode Rewards

The OpenEnv server (server/talking_head_environment.py) is mode-based and returns mode-local rewards:

• image mode: reward = subenv1_score
• clips mode: reward = subenv2_score
• weights mode: reward = subenv3_score
• clips_and_weights mode: reward = 0.50 * subenv2_score + 0.50 * subenv3_score

The combined clips_and_weights mode intentionally excludes Sub-env 1 because that episode path starts at Sub-env 2 and does not execute image audit steps.

Sub-env 1: Reference Image + Prompt Audit (weight: 0.25)

Dimension	Weight	Scoring
Regime Classification	0.35	Exact match = 1.0, borderline = 0.7, wrong = 0.0
Risk Factor Recall	0.35	Set intersection recall
Prompt Modification Validity	0.30	Precision against curated valid set

Sub-env 2: Dataset Clip Audit (weight: 0.35)

Dimension	Weight	Scoring
Disposition Match	0.40	Exact + confidence calibration
Fix Instruction Quality	0.20	Precision >= 0.8 = full, >= 0.5 = half
Dataset Impact Reasoning	0.20	Keyword element matching
Override Misuse Penalty	-0.10	Unjustified override = penalty

Sub-env 3: LoRA Weight Behavioral Audit (weight: 0.40)

Dimension	Weight	Scoring
Phoneme Risk Ranking	0.25	NDCG against reference ranking
Behavior Trigger Prediction	0.20	Set F1 on (phoneme, behavior) pairs
Cluster Identification	0.20	Overlap with reference clusters
Safety Calibration	0.15	Ordinal distance
Mitigation Quality	0.20	(target, action) pair matching

Design Properties

• Deterministic: all graders are rule-based; no LLM judge is required.
• Partial credit: borderline answers receive scaled scores, not binary pass/fail.
• Cascading difficulty: Sub-env 1 risk profile influences Sub-env 2 context.
• Non-trivial scoring: multiple dimensions are evaluated per decision point.