PRD.md

ConflictBench — Product Requirements Document

Version: 1.0
Author: Harsh Jain
Status: Submitted — OpenEnv Hackathon India 2026
Last Updated: April 2026

---

1. Executive Summary

ConflictBench is a reinforcement learning training environment that teaches language models to resolve contradictory instructions using an implicit authority hierarchy. It is the first dedicated RL environment for this capability class. The product consists of a scenario generator, a deterministic verifier, a GRPO training stack, and a HuggingFace Spaces deployment with a live training dashboard.

The primary deliverable is a fine-tuned Qwen2.5-3B-Instruct LoRA adapter that demonstrably outperforms the base model on authority-aware conflict resolution tasks, trained via a reward signal that cannot be gamed without actually solving the task.

---

2. Problem Statement

2.1 The Core Problem

Language models deployed in enterprise environments routinely receive instructions from multiple stakeholders with different authority levels. When these instructions conflict — as they frequently do in real organisations — current models exhibit one of three failure modes:

1. Recency bias: follows the most recently stated instruction regardless of authority
2. Salience bias: follows the most emphatic or detailed instruction regardless of authority
3. Contradiction: produces a plan that attempts to satisfy both sides of the conflict simultaneously, resulting in a logically inconsistent output

None of these failure modes are acceptable for autonomous deployment. All three stem from the same root cause: no existing training signal specifically rewards authority-aware conflict resolution.

2.2 Why Existing Solutions Do Not Work

Prompt engineering can partially address this by explicitly stating the authority hierarchy in the system prompt. This approach does not scale: it requires the hierarchy to be known in advance, stated explicitly, and maintained by the operator. It also fails when the conflict involves instruction sources not anticipated at prompt-writing time.

Supervised fine-tuning on labelled conflict resolution examples requires human annotation of ground truth resolutions — expensive, slow to scale, and subject to annotator disagreement on complex cases.

RLHF with a preference model introduces a learned judge that may itself be miscalibrated and is vulnerable to reward model exploitation.

ConflictBench addresses all three limitations: no explicit hierarchy in the prompt, no human annotation required, and a fully deterministic reward function that cannot be exploited without solving the underlying task.

---

3. Goals and Non-Goals

Goals

• Provide a complete, runnable RL training environment for authority-aware instruction conflict resolution
• Demonstrate measurable improvement over the base model via a non-gameable reward signal
• Make the environment accessible across training platforms (HF Spaces, Colab, Kaggle, local)
• Produce a trained model artifact that can be evaluated independently
• Document the approach thoroughly enough for reproduction and extension

Non-Goals

• Production-ready enterprise deployment (this is a research prototype)
• Support for non-English instruction documents (English only in current version)
• Real-time instruction conflict detection (inference-time application is a future extension)
• Training models larger than 7B parameters (compute scope is limited to hackathon budget)

---

4. User Stories

Primary User: ML Researcher

"I want to train a language model on instruction conflict resolution without needing human-labelled examples, so that I can study how RL reward design affects authority-hierarchy learning."

Acceptance criteria:

• Environment can be instantiated and run locally with python conflict_bench.py
• Training script produces a checkpoint with measurably higher reward than baseline
• Reward function components are individually inspectable via ScoreBreakdown

Secondary User: Hackathon Judge

"I want to evaluate this submission quickly to understand what was built, verify that training works, and assess the quality of the approach."

Acceptance criteria:

• A 30–60 minute evaluation run can be completed on a free Colab T4
• The README explains the approach clearly without requiring prior familiarity
• Results tables show honest numbers with appropriate caveats about run duration

Tertiary User: Enterprise AI Team

"I want to understand whether this approach could be applied to reduce instruction conflict errors in our AI assistant deployment."

Acceptance criteria:

• The blog and documentation explain the business motivation and technical approach
• The trained model is publicly accessible on HF Hub for evaluation
• The authority hierarchy is configurable for different organisational structures

---

5. Functional Requirements

5.1 Scenario Generator

ID	Requirement	Priority
SG-01	Generate scenarios with 8–28 instructions per episode	Must Have
SG-02	Support 3 difficulty levels (2, 4, 6 conflict pairs)	Must Have
SG-03	Embed ground truth resolution at generation time	Must Have
SG-04	Enforce prompt length limit (4000 chars max)	Must Have
SG-05	Produce reproducible scenarios given a fixed seed	Should Have
SG-06	Support 10 action key groups with 3 template variants each	Must Have
SG-07	Sample from 16 authority sources across 6 hierarchy levels	Must Have

5.2 Verifier

ID	Requirement	Priority
VR-01	Implement all 5 rubric functions deterministically	Must Have
VR-02	Return ScoreBreakdown with per-rubric and composite scores	Must Have
VR-03	Handle malformed JSON completions gracefully (score = 0.0)	Must Have
VR-04	Compute F1 correctly for partial matches	Must Have
VR-05	Return composite score in range [0.0, 1.0]	Must Have

5.3 Training Stack

ID	Requirement	Priority
TR-01	Support GRPO training via TRL GRPOTrainer	Must Have
TR-02	Auto-detect GPU and set optimal configuration	Must Have
TR-03	Always use 4-bit quantisation (prevents Unsloth dtype kernel bug)	Must Have
TR-04	Save checkpoints every 50 steps	Must Have
TR-05	Auto-upload best checkpoint to HF Hub after training	Should Have
TR-06	Stream training logs to Gradio dashboard in real time	Should Have
TR-07	Support checkpoint resume after session interruption	Should Have

5.4 HF Spaces Deployment

ID	Requirement	Priority
SP-01	One-click training start via Gradio button	Must Have
SP-02	Live log streaming with auto-refresh	Must Have
SP-03	Training plots displayed after completion	Should Have
SP-04	Thread-safe log buffer	Must Have

---

6. Non-Functional Requirements

Category	Requirement
Reproducibility	Same seed → same scenarios → same training trajectory
Accessibility	Runnable on free Colab T4 (with reduced config)
Correctness	Verifier outputs identical scores for identical inputs
Transparency	All reward components inspectable individually
Maintainability	Generator, verifier, and trainer are independently testable

---

7. Architecture Summary

See doc/ARCHITECTURE.md for full detail.

ScenarioGenerator → ConflictBenchEnv → Verifier → GRPO Trainer
     (generator.py)   (conflict_bench.py)  (verifier.py)  (train_grpo.py)

---

8. Training Configuration (Production)

Parameter	Value	Rationale
Base model	Qwen2.5-3B-Instruct (4-bit)	Balance of capability and accessibility
LoRA rank	32	Sufficient expressiveness; stable on L4/A10G
Training scenarios	400	Enough for GRPO convergence at difficulty 1–2
Eval scenarios	60	Reliable eval metric with low variance
Epochs	2	Reward peaks around epoch 2; more risks KL drift
Learning rate	3e-6	Conservative; preserves instruction following
β (KL penalty)	0.04	Prevents excessive drift; 0.02 was insufficient
num_generations	4–6	4 minimum; 6 preferred on A100

---

9. Success Metrics

Metric	Baseline	Target	Achieved (Run 2)
Composite reward (peak)	0.14	0.40	0.50
Improvement over baseline	—	+100%	+257%
No contradictions rate	~31%	>60%	~74%
Conflict identification F1	~8%	>25%	~39%
Format compliance	~65%	>85%	~88%

All targets exceeded.

---

10. Known Limitations and Future Work

Current Limitations

• Training is limited to difficulty 1–2 scenarios; difficulty 3 is eval-only
• The authority hierarchy is fixed (6 tiers); domain-specific hierarchies require code changes
• Reward peak at ~step 250 suggests reward saturation under current configuration; longer runs require β adjustment and curriculum progression
• KL divergence rises monotonically; future runs should monitor this and halt early if drift exceeds threshold

Planned Extensions

1. Difficulty curriculum progression: start with difficulty 1, transition to difficulty 2–3 as reward improves
2. Domain-configurable hierarchies: parameterise the authority levels for healthcare, legal, government contexts
3. Multi-hop conflict reasoning: conflicts where resolution of pair A depends on resolution of pair B
4. Adversarial instruction injection: scenarios designed to trick the model into following low-authority instructions
5. 7B and 13B model training: with appropriate quantisation and VRAM scaling
6. Multilingual extension: Spanish, French, German authority resolution scenarios

---

11. Risks

Risk	Likelihood	Impact	Mitigation
Reward hacking via length	Low	Medium	Efficiency rubric penalises over-inclusion
JSON format memorisation without semantic understanding	Medium	High	Conflict ID F1 requires correct instruction IDs
KL divergence runaway	Medium	Medium	β=0.04 provides sufficient penalty
Unsloth kernel dtype bug on full-precision path	High (A100)	Critical	Always use 4-bit quantisation
Kaggle session timeout mid-training	High	Medium	Checkpoint-every-50-steps + resume support

---

ConflictBench PRD v1.0 — OpenEnv Hackathon India 2026