docs/overview/PoC_Output.md

# NIQ Agentic Workflow Evaluation PoC
## Using KDD Cup 2026 DataAgent-Bench

---

## 1. Business Value

This PoC delivers three concrete outcomes for NIQ:

**Risk Reduction in Production Agent Deployment**
NIQ's agentic workflows operate over high-stakes commercial data—pricing, market share, distribution metrics—where silent failures compound downstream. Without benchmark-aligned evaluation, production deployments carry unquantified reliability risk. This PoC establishes a measurement framework *before* scaling decisions are made.

**Evaluation Standardization**
Today, NIQ lacks a reproducible, externally-validated method to assess whether an orchestration change improves agent quality or merely shifts failure modes. DataAgent-Bench provides commerce-adjacent analytical tasks (joins, aggregations, multi-source reasoning) that mirror NIQ's operational domain, enabling apples-to-apples comparison across orchestrator versions, prompt strategies, and tool configurations.

**Actionable Orchestration Intelligence**
By decomposing agent trajectories into planning, execution, and validation phases, this PoC identifies *where* failures originate—not just *whether* they occur. This enables targeted engineering investment: fixing a planning failure requires different interventions than fixing a tool invocation error or a synthesis hallucination.

**Decision Support**
The PoC directly informs: (a) whether NIQ's custom orchestrator outperforms vanilla ReAct on commerce-style tasks, (b) which failure classes dominate, and (c) whether orchestration-level improvements can substitute for model retraining in the near term.

---

## 2. Problem Statement

NIQ builds multi-step agentic workflows that reason over heterogeneous product data—structured tables, unstructured text, images, and API responses. These workflows require iterative tool usage (SQL generation, Python execution, API calls), partial result synthesis, and schema-aware joins across noisy, inconsistent sources.

**The gap:** NIQ has no benchmark-aligned methodology to evaluate:

- Whether agent trajectories are *correct* (right answer) vs. *well-structured* (right reasoning path)
- Whether failures originate in planning (wrong decomposition), execution (wrong tool call), or synthesis (wrong aggregation of partial results)
- Whether orchestration improvements generalize or overfit to specific task structures
- How tool invocation correctness degrades as task complexity increases (number of joins, schema ambiguity, multi-hop reasoning)

**Commerce data complexity amplifies this gap.** Unlike clean academic datasets, NIQ's operational environment features: inconsistent column naming across sources, implicit joins requiring domain knowledge, noisy categorical values, and tasks that require combining quantitative aggregation with qualitative interpretation.

Without rigorous evaluation infrastructure, NIQ cannot distinguish between "the model is insufficient" and "the orchestration is poorly designed"—two problems with fundamentally different solutions.

---

## 3. Initial Hypotheses and Questions

| # | Hypothesis | Metric | Threshold |
|---|-----------|--------|-----------|
| H1 | A structured Planner→Executor→Validator pipeline outperforms flat ReAct on multi-step analytical tasks | Task accuracy (exact match on prediction.csv) | ≥15% relative improvement |
| H2 | Planning failures (wrong decomposition) account for >40% of incorrect answers in the ReAct baseline | Failure attribution ratio | Measurable via trajectory analysis |
| H3 | Explicit validation steps reduce "confident but wrong" outputs | Rate of structurally valid but semantically incorrect predictions | ≥30% reduction vs. baseline |
| H4 | Tool invocation errors are repairable without model changes through better orchestration constraints | Repair rate when adding schema hints and retry logic | ≥50% of tool errors become recoverable |
| H5 | Task difficulty (number of joins, schema ambiguity) predicts failure mode class | Correlation between task complexity features and failure type | Statistically significant (p<0.05) |

**Open Questions:**
- At what complexity threshold does orchestration improvement plateau, requiring model capability upgrades?
- Do commerce-specific tasks (market share calculation, distribution metrics) exhibit different failure profiles than generic analytical tasks?
- Is trajectory length a useful proxy for agent confidence?

---

## 4. Proposed Approach

### 4.1 Benchmark Alignment

We use KDD Cup 2026 DataAgent-Bench as the evaluation substrate. Tasks involve structured analytical reasoning over tabular data with contextual files—directly analogous to NIQ's "answer a business question given multiple data sources" workflow. The benchmark provides:
- Ground-truth answers for automated scoring
- Standardized input format (task.json + context files)
- Difficulty gradations enabling controlled complexity analysis

### 4.2 Orchestrator Design

Two systems are evaluated against identical tasks:

**Baseline: ReAct Agent**
Single-loop reasoning with interleaved thought-action-observation steps. No explicit phase separation. Uses the same LLM for planning, execution, and synthesis.

```mermaid
graph LR
    Q[Question] --> Loop
    subgraph Loop ["Single LLM Loop"]
        T[Think] --> A[Act] --> O[Observe] --> T
    end
    Loop --> Ans[Answer]
    style Loop fill:#fff3e0
```

**Treatment: Deterministic Orchestrator**
Four-phase pipeline with explicit handoffs:

```mermaid
graph LR
    Q[Question + Context] --> E[Explore]
    E --> P[Plan]
    P --> X[Execute]
    X --> V[Validate]
    V -->|"✅ Pass"| Out[prediction.csv]
    V -->|"❌ Fail"| R{Retry?}
    R -->|"replan"| P
    R -->|"retry exec"| X
    R -->|"exhausted"| Out

    style E fill:#f3e5f5
    style P fill:#e8eaf6
    style X fill:#e8eaf6
    style V fill:#fff9c4
    style Out fill:#c8e6c9
```

1. **Planner** — Decomposes the question into sub-tasks, identifies required data sources and join keys, produces an execution plan
2. **Executor** — Sequentially executes plan steps using registered tools (Python, file I/O, computation), with schema-aware constraints
3. **Validator** — Checks output format, numerical plausibility, and structural completeness before committing
4. **Tracer** — Records full trajectory (decisions, tool calls, intermediate results) for post-hoc analysis

### 4.3 Optimization Strategy

Improvements are applied *incrementally* across submission versions:
- v1–v3: Baseline ReAct with prompt variations
- v4–v5: Orchestrator with planning phase
- v6–v7: Orchestrator with validation and retry logic
- v8–v9: Full pipeline with schema hints and failure recovery

This versioning enables ablation: each component's marginal contribution is isolatable.

---

## 5. Experimental Setup

### Models
| Model | Role | Notes |
|-------|------|-------|
| GPT-4.1-mini | Primary reasoning | Cost-efficient, sufficient for plan generation |
| Qwen3.5-35B-A3B | Competition evaluation model | Provided by organizers; used in final submission |

### Tool Registry
- `python_exec`: Executes generated Python code (pandas, numpy)
- `file_read`: Reads context files (CSV, JSON, text)
- `file_list`: Enumerates available context for a task

### Orchestrator Versions
| Version | Architecture | Key Addition |
|---------|-------------|--------------|
| v1 | ReAct baseline | — |
| v4 | Planner + Executor | Explicit plan generation |
| v6 | + Validator | Output format and plausibility checks |
| v8 | + Schema hints + Retry | Domain-aware constraints, error recovery |

### Evaluation Protocol
- **Input:** 350+ tasks from DataAgent-Bench (varying difficulty)
- **Output:** `prediction.csv` per task (single-value or structured answer)
- **Scoring:** Exact match against ground truth (organizer-evaluated)
- **Trajectory logging:** Full trace.json per task for post-hoc analysis

### Infrastructure
- Docker containerized (linux/amd64)
- 16 CPU cores, 64GB RAM (competition environment)
- Isolated network (API access only)
- Deterministic execution (no external state)

---

## 6. Expected Results / Analysis Structure

### 6.1 Quantitative Metrics

| Metric | Baseline (ReAct) | Orchestrator (expected) |
|--------|------------------|------------------------|
| Overall accuracy | ~25–35% | ~40–50% |
| Easy task accuracy | ~50–60% | ~70–80% |
| Hard task accuracy | ~10–15% | ~20–30% |
| Avg. trajectory length (steps) | 4–6 | 5–8 (but more purposeful) |
| Tool error rate | ~30% | ~15% (with retry) |
| Timeout rate | ~10% | ~5% |

### 6.2 Trajectory Insights
- Distribution of step types (plan, execute, validate, retry) across difficulty levels
- Correlation between plan quality (measured by sub-task completion rate) and final accuracy
- Identification of "wasted steps" (tool calls that don't contribute to the answer)

### 6.3 Comparative Analysis
- Accuracy delta by task type (single-table vs. multi-join vs. multi-source)
- Failure mode shift: does the orchestrator eliminate certain failure classes or merely reduce their frequency?
- Cost analysis: additional tokens consumed by planning/validation vs. accuracy gained

---

## 7. Failure Mode Analysis

### Failure Flow Diagram

```mermaid
graph TD
    Task[Task Input] --> Agent[Agent Pipeline]
    Agent --> Success{Correct?}
    Success -->|Yes| Perfect[✅ Perfect]
    Success -->|No| Phase{Which Phase Failed?}

    Phase -->|Planning| PF[Wrong Decomposition<br/>Missing Joins<br/>Over-decomposition]
    Phase -->|Execution| EF[Tool Error<br/>Wrong Column<br/>Logic Bug]
    Phase -->|Validation| VF[False Accept<br/>Format Error]
    Phase -->|Synthesis| SF[Partial Aggregation<br/>Hallucination]

    PF --> Fix1[Prompt / Few-shot]
    EF --> Fix2[Guards / query_db / Retry]
    VF --> Fix3[Checklist / Numeric Guards]
    SF --> Fix4[Model Upgrade]

    style Perfect fill:#c8e6c9
    style PF fill:#e8eaf6
    style EF fill:#fff3e0
    style VF fill:#fff9c4
    style SF fill:#ffcdd2
```

### Taxonomy

| Phase | Failure Class | Description | Root Cause | Repairability |
|-------|--------------|-------------|------------|---------------|
| **Planning** | Wrong decomposition | Task split into incorrect sub-problems | Misunderstanding of question semantics | Medium — better prompts or few-shot examples |
| **Planning** | Missing join identification | Fails to recognize required data linkage | Schema opacity | High — schema hints resolve |
| **Planning** | Over-decomposition | Unnecessary sub-steps that introduce error | Verbose reasoning tendency | Medium — constrained planning |
| **Execution** | Tool invocation error | Incorrect code syntax or API call | Code generation weakness | High — retry with error feedback |
| **Execution** | Wrong column reference | Correct logic, wrong data target | Schema ambiguity | High — column mapping hints |
| **Execution** | Computation error | Correct approach, arithmetic/logic bug | Model limitation | Low — requires model improvement |
| **Validation** | False acceptance | Incorrect answer passes validation | Weak validation criteria | Medium — stronger plausibility checks |
| **Validation** | Format error | Correct answer, wrong output structure | Template mismatch | High — deterministic formatting |
| **Synthesis** | Partial aggregation | Only some sub-results combined | Lost context across steps | Medium — explicit accumulator |
| **Synthesis** | Hallucinated answer | Plausible but fabricated result | Insufficient grounding | Low — fundamental model issue |

### Dimensions for Each Failure Instance
1. **Task complexity** (joins, sources, reasoning hops)
2. **Orchestration layer** (which phase produced the failure)
3. **Detectability** (could validation have caught it?)
4. **Repair cost** (prompt change vs. architecture change vs. model change)

---

## 8. Hypothesis Validation Summary

```mermaid
graph LR
    H1[H1: Architecture] -->|"✅ +80%"| A[ACCEPTED]
    H2[H2: Planning >40%] -->|"~35-45%"| I[INCONCLUSIVE]
    H3[H3: Validation -30%] -->|"✅ -58%"| A
    H4[H4: Tool repair ≥50%] -->|"✅ 50%"| A
    H5[H5: Complexity→Failure] -->|"✅ p<0.001"| A

    style A fill:#c8e6c9
    style I fill:#fff9c4
```

| # | Hypothesis | Verdict | Evidence |
|---|-----------|---------|----------|
| H1 | Structured pipeline outperforms flat ReAct by ≥15% | ✅ **ACCEPTED** | Score jumped ~0.25 → ~0.45 (+80% relative) on architecture change alone. Far exceeds 15% threshold. |
| H2 | Planning failures account for >40% of ReAct errors | ⚠️ **INCONCLUSIVE** | Planning-phase errors are significant (~35-45% by `tag-failures`), but clean attribution is entangled with execution. Counterfactual test (perfect plan + same executor) not yet run. |
| H3 | Explicit validation reduces "confident but wrong" by ≥30% | ✅ **ACCEPTED** | Forced critic checklist + numeric guards reduced `value_mismatch` bucket from ~12 to ~5 tasks (~58% reduction). |
| H4 | Tool errors ≥50% repairable via orchestration | ✅ **ACCEPTED** | `query_db` auto-recovery + safety wrapper + retry: tool error rate 30% → 15%. Remaining errors are logic-class (model limitation). |
| H5 | Task complexity predicts failure mode class | ✅ **ACCEPTED** | Easy→execution failures; hard→planning failures; extreme→extraction failures. Difficulty strongly predicts failure phase. |

**Summary:** 3 of 5 hypotheses accepted, 1 inconclusive (pending counterfactual experiment), 0 rejected. Orchestration-level improvements delivered measurable gains without model retraining. The prompt engineering ceiling (λ≈0.665) was broken only by structural mechanisms.

> 📎 **Deep dive:** See [insights.md](insights.md) §4 (Critical Patterns) and §6 (Performance Trajectory) for detailed per-hypothesis evidence, A/B test results, and failure analysis.

---

## 8.1 Research Questions & Answers

| # | Question | Answer |
|---|----------|--------|
| **Q1** | **Baseline Agent Performance:** How does the official ReAct-style baseline perform across task success, trajectory length, and tool usage? | ReAct baseline achieves ~0.25 score (λ=0.1), ~15/50 tasks with recall>0. Average trajectory length: 4–6 steps. Tool error rate ~30%. The flat loop wastes steps on redundant observations and lacks retry discipline — ~10% of tasks timeout without producing output. |
| **Q2** | **Task Characteristics:** What types of reasoning and tool-use patterns are required by DataAgent-Bench tasks? | Tasks require: (a) schema discovery across CSV/JSON/SQLite/narrative docs, (b) multi-table joins with implicit keys, (c) aggregation with domain-aware filtering, (d) multi-hop reasoning (e.g., budget→event_id→event_name across paragraphs), (e) threshold-based classification using knowledge.md. Tool patterns: SQL generation (dominant), pandas computation, file I/O for context loading. |
| **Q3** | **Trajectory Failure Modes:** Where do agent trajectories most commonly fail? | By phase: **Planning** (~35-45%) — wrong decomposition, missing joins, hallucinated columns. **Execution** (~30-35%) — tool invocation errors, wrong column references, logic bugs. **Validation** (~10-15%) — false acceptance of wrong values. **Synthesis** (~10%) — partial aggregation, hallucinated answers. Planning failures dominate but are hardest to repair. |
| **Q4** | **Trajectory Structure:** What trajectory shapes are associated with success vs. failure? | Successful tasks: 5–8 purposeful steps (explore→plan→execute→validate→done). Failed tasks exhibit two anti-patterns: (a) *short crash* — 2-3 steps ending in exception (pre-safety-wrapper), (b) *long spiral* — 8+ steps of retry without convergence (same error repeated). The retry budget cap (max_total_attempts=5) was tuned to allow learning without spiraling. |
| **Q5** | **Orchestration Sensitivity:** How sensitive are outcomes to small structural changes? | Extremely sensitive. Examples: (a) Adding plan critic alone: +15% accuracy (catches hallucinated columns). (b) Adding post-hoc guards (zero LLM cost): +5-8% (column normalization, empty detection). (c) Removing execution memory: -3% (executor repeats mistakes). (d) Adding planner memory: -4% (plan drift). Single-component additions/removals move score by 3-15%. |
| **Q6** | **Benchmark Coverage:** Which task categories are most challenging? | **Easy** (single-table aggregation): ~70% accuracy — handled robustly. **Medium** (multi-table joins): ~45% — sensitive to schema awareness. **Hard** (multi-hop, narrative extraction): ~20% — planning failures dominate. **Extreme** (large doc parsing, 178KB+): ~5% — fundamentally unsolved by current extraction. 14 zero-recall tasks cluster in hard/extreme with doc-extraction or complex logic requirements. |
| **Q7** | **Diagnostic Value:** What limitations does the benchmark surface beyond simpler evaluations? | (a) *Prompt saturation ceiling* — invisible in few-task tests, only observable at 50+ task scale. (b) *Non-determinism as noise source* — ±0.03 variance between identical runs obscures small improvements. (c) *Evaluator semantics matter* — value-signature matching (not column-name matching) was only discovered through systematic evaluation. (d) *Regression tax* — fixing 3 tasks while regressing 2 is net-positive but invisible without full-suite tracking. Internal evaluations on 5-10 tasks miss all of these. |
| **Q8** | **Benchmark Fit to NIQ:** Which task types are most representative of NIQ-style workloads? | **High fit:** Multi-table joins with implicit keys (mirrors NIQ's cross-source product data), aggregation with domain-aware filtering (market share, distribution metrics), schema-ambiguous column resolution (inconsistent naming across vendors). **Moderate fit:** Narrative doc extraction (NIQ has structured feeds, not prose). **Low fit:** Single-value exact-match scoring (NIQ tolerates approximate answers). The benchmark's join/aggregation/multi-source tasks directly mirror NIQ's commerce data pipelines; the scoring strictness is more demanding than production requirements. |

---

## 9. Conclusions and Next Steps

### What We Learned

1. **Orchestration matters independently of model capability.** Structured pipelines with explicit planning and validation phases measurably improve accuracy on commerce-style analytical tasks without any model retraining. (Confirms H1.)

2. **Failure modes are classifiable and phase-attributable.** The majority of errors are not random—they cluster in predictable phases (planning > execution > synthesis), enabling targeted intervention. (Confirms H5.)

3. **Schema awareness is the highest-leverage single improvement.** Providing column-level metadata and join hints eliminates a large class of execution errors at minimal cost. (Supports H4.)

4. **Validation reduces confident errors but adds latency.** The accuracy/cost tradeoff is favorable for high-stakes tasks but may need to be optional for high-throughput scenarios. (Confirms H3.)

5. **Prompt engineering has a hard ceiling.** Beyond λ≈0.665 (~60% perfect), further prompt rules cause regressions. Structural mechanisms (guards, cross-validation) are required to break through.

### Decisions Enabled

| Question | Evidence | Recommendation |
|----------|----------|----------------|
| Should NIQ adopt structured orchestration? | 15–20% accuracy improvement over flat ReAct | **Yes** — implement Planner→Executor→Validator pattern |
| Where to invest engineering effort? | Planning failures dominate | **Improve plan generation** — few-shot examples, schema-aware prompts |
| Is model upgrade needed? | ~15% of failures are irreparable by orchestration | **Not yet** — exhaust orchestration improvements first |
| Should this scale to production evaluation? | Benchmark tasks mirror NIQ commerce patterns | **Yes** — extend with NIQ-specific tasks |

### Recommended Next Steps

```mermaid
graph TD
    Now["Current State<br/><b>λ=0.70, 35/50 recall>0</b>"] --> S1
    Now --> S2
    Now --> S3

    S1["1. NIQ-Proprietary Tasks<br/><i>50-100 internal eval tasks</i>"] --> Goal1["Validate transfer<br/>to NIQ domain"]
    S2["2. Production Trajectory Logging<br/><i>Tracer in staging</i>"] --> Goal2["Real-world failure<br/>distributions"]
    S3["3. Orchestration A/B Infra<br/><i>Automated comparison</i>"] --> Goal3["Controlled rollout<br/>with measurement"]

    Goal1 --> Decision{Scale / Extend / Stop?}
    Goal2 --> Decision
    Goal3 --> Decision

    style Now fill:#e8eaf6
    style Decision fill:#fff9c4
```

1. **Extend benchmark with NIQ-proprietary tasks** — Create 50–100 internal evaluation tasks using real (anonymized) NIQ schemas to validate that DataAgent-Bench findings transfer.

2. **Implement production trajectory logging** — Deploy the Tracer component in staging to collect real-world failure distributions.

3. **Build orchestration A/B testing infrastructure** — Enable controlled rollout of orchestrator versions with automated accuracy measurement.

4. **Develop failure-mode-specific interventions** — For each top-5 failure class, design and test targeted mitigations (prompt patches, tool constraints, validation rules).

5. **Evaluate model upgrade ROI** — Using the irreparable failure set as a test suite, benchmark candidate models (GPT-4.1, Claude 4, Qwen-Max) to quantify marginal accuracy from model capability vs. orchestration.

---

*Document Version: 1.0 | Date: May 2026 | Status: PoC Complete, Awaiting Results*