docs/architecture.md

ConflictBench — System Architecture

Overview

ConflictBench is composed of four independent, composable components:

1. ScenarioGenerator — produces synthetic training episodes
2. ConflictBenchEnv — OpenEnv-compatible environment wrapper
3. Verifier — deterministic reward computation
4. GRPO Training Stack — Unsloth + TRL training loop

Each component has a single, well-defined interface with the next. The boundary between generation and evaluation is strict: the verifier never sees the generation logic, and the generator never sees the reward function.

---

Component 1: ScenarioGenerator (generator.py)

Purpose

Produces an unlimited supply of novel, realistic business instruction documents with ground truth embedded at generation time.

Design Decisions

Programmatic generation over LLM generation. Each scenario is built by filling templates with randomly sampled parameters. This eliminates the risk of a generative model introducing inconsistencies or biases into the training data. Every scenario is verifiably correct by construction.

Ground truth injection at generation time. When two instructions conflict, the winner (higher authority source) is determined at the moment of conflict creation and stored with the scenario object. The verifier checks completions against this stored ground truth — not against a post-hoc judgement.

Prompt length guard. Scenarios exceeding 4000 characters are discarded and regenerated. This prevents VRAM-exhausting prompts and ensures consistent context length distributions across the training batch.

Parameters

Parameter	Value	Notes
Action key groups	10	hiring, budget, data access, policy, vendor, etc.
Template variants per group	3	low/medium/high complexity phrasings
Authority sources	16	spanning 6 hierarchy levels
Instructions per episode	8–28	sampled uniformly
Conflict pairs per episode	2–6	embedded proportionally
Difficulty 1	2 conflicts, obvious source labels	80% of training mix
Difficulty 2	4 conflicts, ambiguous source labels	20% of training mix
Difficulty 3	6 conflicts, multi-hop reasoning required	eval only

Authority Hierarchy

Level 1  Legal & Compliance           (always wins)
Level 2  C-Suite / Board              (executive directives)
Level 3  VP / Senior Director         (functional leadership)
Level 4  Director / Manager           (departmental management)
Level 5  Team Lead                    (team-level direction)
Level 6  Individual Contributor       (lowest authority)

When instructions from Level 3 and Level 5 conflict, Level 3 wins. This is deterministic — no judgement required.

---

Component 2: ConflictBenchEnv (conflict_bench.py)

Purpose

Wraps the generator and verifier into an OpenEnv-compatible environment class.

Interface

env = ConflictBenchEnv()

obs = env.reset()          # → str: formatted instruction document
reward, done, info = env.step(action)  # action: str (model completion)
state = env.state          # → dict: current episode metadata

Episode Structure

Each episode is a single turn. The model receives the full instruction document as the observation and must produce a complete JSON resolution plan as the action. There is no multi-turn interaction within an episode.

This design choice is intentional: business instruction resolution is typically a single-pass task. An employee reads the full set of instructions and produces a resolution plan. They do not interactively negotiate with the environment.

---

Component 3: Verifier (verifier.py)

Purpose

Computes the composite reward from a model completion and the corresponding scenario ground truth.

Rubric Implementation

def score(completion: str, scenario: Scenario) -> ScoreBreakdown:
    r1 = score_final_state(completion, scenario)      # F1 vs ground truth
    r2 = score_no_contradictions(completion)          # binary contradiction check
    r3 = score_conflict_identification(completion, scenario)  # pair F1 × direction
    r4 = score_efficiency(completion, scenario)       # unnecessary instruction penalty
    r5 = score_format_compliance(completion)          # JSON schema validation
    composite = 0.35*r1 + 0.25*r2 + 0.20*r3 + 0.10*r4 + 0.10*r5
    return ScoreBreakdown(r1, r2, r3, r4, r5, composite)

Why Each Rubric Weight Was Chosen

• 35% Final State: The primary task is producing a correct execution plan. This is the dominant signal.
• 25% No Contradictions: A plan that contradicts itself is worse than useless — it is actively harmful. High weight ensures the model never learns to produce contradictory plans.
• 20% Conflict Identification: Explicit conflict naming builds structured reasoning, not just lucky resolution. Without this, a model could get high Final State scores by memorising common resolution patterns rather than actually identifying conflicts.
• 10% Efficiency: Prevents padding/inclusion of irrelevant instructions. Lower weight because inefficiency is less harmful than incorrect resolution.
• 10% Format: Minimum bar. Valid JSON is required for downstream use but not the primary measure of capability.

---

Component 4: GRPO Training Stack

Why GRPO

GRPO (Group Relative Policy Optimisation) works by:

1. Generating N completions per prompt
2. Scoring each with the reward function
3. Computing advantages relative to the group mean
4. Updating the policy to increase probability of above-average completions

This is ideal for ConflictBench because:

• No reference completions are needed — the model learns from its own distribution
• The reward function provides a dense, informative signal at every step
• GRPO is stable under the composite reward structure (no single rubric dominates during training instability)

Model Choice

Qwen2.5-3B-Instruct was selected for:

• Strong instruction following baseline (reduces cold-start problem)
• Sufficient capacity for structured JSON output
• 3B parameters fits comfortably in 4-bit on T4/L4/A10G for broad accessibility

LoRA Configuration

Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Rank (r): 32
Alpha: 32  (alpha = r is standard; effective scale = 1.0)
Dropout: 0.0
Bias: none
Gradient checkpointing: unsloth (memory-efficient)

Rank 32 was chosen to balance expressiveness (the model needs to learn a genuinely new capability) with memory efficiency (higher ranks risk OOM on L4/A10G).

Key Hyperparameters and Their Rationale

Hyperparameter	Value	Rationale
Learning rate	3e-6	Conservative; avoids catastrophic forgetting of instruction following
Warmup ratio	0.05	5% warmup stabilises early GRPO steps
β (KL penalty)	0.04	Prevents excessive drift from base policy; 0.02 was too low (KL diverged)
Temperature	1.0	Full diversity in GRPO generation; lower values reduce exploration
num_generations	4–6	4 is minimum for meaningful group comparison; 6 better on large VRAM
Max completion tokens	768	Average completion ~300 tokens; headroom without waste
Max prompt tokens	3200	Accommodates 28-instruction documents; prompt guard at generation enforces this

---

Data Flow Diagram

Episode Start
     │
     ▼
ScenarioGenerator.generate(difficulty)
     │  Returns: Scenario(instructions, conflicts, ground_truth, prompt)
     ▼
ConflictBenchEnv.reset()
     │  Returns: formatted_prompt (str)
     ▼
GRPOTrainer generates N completions
     │  N = num_generations (4–6)
     ▼
[For each completion]
Verifier.score(completion, scenario)
     │  Returns: ScoreBreakdown(r1..r5, composite)
     ▼
GRPOTrainer computes group advantages
     │  advantage_i = reward_i - mean(rewards)
     ▼
Policy gradient update (PPO-style clipped objective + KL penalty)
     │
     ▼
Next Episode

---

File Reference

File	Lines	Purpose
conflict_bench.py	~150	OpenEnv environment class
generator.py	~500	Scenario generation with all templates
verifier.py	~300	Five rubric functions + composite scorer
train_grpo.py	~350	Root training script (local/Kaggle)
hf_space_a100/train.py	~400	HF Spaces/Colab training with Gradio integration
hf_space_a100/app.py	~130	Gradio dashboard
diagnose_tokens.py	~80	Token budget analysis utility
app.py	~100	Local demo (base vs fine-tuned comparison)