Spaces:
Running
NIQ Agentic Workflow Evaluation PoC
Using KDD Cup 2026 DataAgent-Bench
1. Business Value
This PoC delivers three concrete outcomes for NIQ:
Risk Reduction in Production Agent Deployment NIQ's agentic workflows operate over high-stakes commercial data—pricing, market share, distribution metrics—where silent failures compound downstream. Without benchmark-aligned evaluation, production deployments carry unquantified reliability risk. This PoC establishes a measurement framework before scaling decisions are made.
Evaluation Standardization Today, NIQ lacks a reproducible, externally-validated method to assess whether an orchestration change improves agent quality or merely shifts failure modes. DataAgent-Bench provides commerce-adjacent analytical tasks (joins, aggregations, multi-source reasoning) that mirror NIQ's operational domain, enabling apples-to-apples comparison across orchestrator versions, prompt strategies, and tool configurations.
Actionable Orchestration Intelligence By decomposing agent trajectories into planning, execution, and validation phases, this PoC identifies where failures originate—not just whether they occur. This enables targeted engineering investment: fixing a planning failure requires different interventions than fixing a tool invocation error or a synthesis hallucination.
Decision Support The PoC directly informs: (a) whether NIQ's custom orchestrator outperforms vanilla ReAct on commerce-style tasks, (b) which failure classes dominate, and (c) whether orchestration-level improvements can substitute for model retraining in the near term.
2. Problem Statement
NIQ builds multi-step agentic workflows that reason over heterogeneous product data—structured tables, unstructured text, images, and API responses. These workflows require iterative tool usage (SQL generation, Python execution, API calls), partial result synthesis, and schema-aware joins across noisy, inconsistent sources.
The gap: NIQ has no benchmark-aligned methodology to evaluate:
- Whether agent trajectories are correct (right answer) vs. well-structured (right reasoning path)
- Whether failures originate in planning (wrong decomposition), execution (wrong tool call), or synthesis (wrong aggregation of partial results)
- Whether orchestration improvements generalize or overfit to specific task structures
- How tool invocation correctness degrades as task complexity increases (number of joins, schema ambiguity, multi-hop reasoning)
Commerce data complexity amplifies this gap. Unlike clean academic datasets, NIQ's operational environment features: inconsistent column naming across sources, implicit joins requiring domain knowledge, noisy categorical values, and tasks that require combining quantitative aggregation with qualitative interpretation.
Without rigorous evaluation infrastructure, NIQ cannot distinguish between "the model is insufficient" and "the orchestration is poorly designed"—two problems with fundamentally different solutions.
3. Initial Hypotheses and Questions
| # | Hypothesis | Metric | Threshold |
|---|---|---|---|
| H1 | A structured Planner→Executor→Validator pipeline outperforms flat ReAct on multi-step analytical tasks | Task accuracy (exact match on prediction.csv) | ≥15% relative improvement |
| H2 | Planning failures (wrong decomposition) account for >40% of incorrect answers in the ReAct baseline | Failure attribution ratio | Measurable via trajectory analysis |
| H3 | Explicit validation steps reduce "confident but wrong" outputs | Rate of structurally valid but semantically incorrect predictions | ≥30% reduction vs. baseline |
| H4 | Tool invocation errors are repairable without model changes through better orchestration constraints | Repair rate when adding schema hints and retry logic | ≥50% of tool errors become recoverable |
| H5 | Task difficulty (number of joins, schema ambiguity) predicts failure mode class | Correlation between task complexity features and failure type | Statistically significant (p<0.05) |
Open Questions:
- At what complexity threshold does orchestration improvement plateau, requiring model capability upgrades?
- Do commerce-specific tasks (market share calculation, distribution metrics) exhibit different failure profiles than generic analytical tasks?
- Is trajectory length a useful proxy for agent confidence?
4. Proposed Approach
4.1 Benchmark Alignment
We use KDD Cup 2026 DataAgent-Bench as the evaluation substrate. Tasks involve structured analytical reasoning over tabular data with contextual files—directly analogous to NIQ's "answer a business question given multiple data sources" workflow. The benchmark provides:
- Ground-truth answers for automated scoring
- Standardized input format (task.json + context files)
- Difficulty gradations enabling controlled complexity analysis
4.2 Orchestrator Design
Two systems are evaluated against identical tasks:
Baseline: ReAct Agent Single-loop reasoning with interleaved thought-action-observation steps. No explicit phase separation. Uses the same LLM for planning, execution, and synthesis.
graph LR
Q[Question] --> Loop
subgraph Loop ["Single LLM Loop"]
T[Think] --> A[Act] --> O[Observe] --> T
end
Loop --> Ans[Answer]
style Loop fill:#fff3e0
Treatment: Deterministic Orchestrator Four-phase pipeline with explicit handoffs:
graph LR
Q[Question + Context] --> E[Explore]
E --> P[Plan]
P --> X[Execute]
X --> V[Validate]
V -->|"✅ Pass"| Out[prediction.csv]
V -->|"❌ Fail"| R{Retry?}
R -->|"replan"| P
R -->|"retry exec"| X
R -->|"exhausted"| Out
style E fill:#f3e5f5
style P fill:#e8eaf6
style X fill:#e8eaf6
style V fill:#fff9c4
style Out fill:#c8e6c9
- Planner — Decomposes the question into sub-tasks, identifies required data sources and join keys, produces an execution plan
- Executor — Sequentially executes plan steps using registered tools (Python, file I/O, computation), with schema-aware constraints
- Validator — Checks output format, numerical plausibility, and structural completeness before committing
- Tracer — Records full trajectory (decisions, tool calls, intermediate results) for post-hoc analysis
4.3 Optimization Strategy
Improvements are applied incrementally across submission versions:
- v1–v3: Baseline ReAct with prompt variations
- v4–v5: Orchestrator with planning phase
- v6–v7: Orchestrator with validation and retry logic
- v8–v9: Full pipeline with schema hints and failure recovery
This versioning enables ablation: each component's marginal contribution is isolatable.
5. Experimental Setup
Models
| Model | Role | Notes |
|---|---|---|
| GPT-4.1-mini | Primary reasoning | Cost-efficient, sufficient for plan generation |
| Qwen3.5-35B-A3B | Competition evaluation model | Provided by organizers; used in final submission |
Tool Registry
python_exec: Executes generated Python code (pandas, numpy)file_read: Reads context files (CSV, JSON, text)file_list: Enumerates available context for a task
Orchestrator Versions
| Version | Architecture | Key Addition |
|---|---|---|
| v1 | ReAct baseline | — |
| v4 | Planner + Executor | Explicit plan generation |
| v6 | + Validator | Output format and plausibility checks |
| v8 | + Schema hints + Retry | Domain-aware constraints, error recovery |
Evaluation Protocol
- Input: 350+ tasks from DataAgent-Bench (varying difficulty)
- Output:
prediction.csvper task (single-value or structured answer) - Scoring: Exact match against ground truth (organizer-evaluated)
- Trajectory logging: Full trace.json per task for post-hoc analysis
Infrastructure
- Docker containerized (linux/amd64)
- 16 CPU cores, 64GB RAM (competition environment)
- Isolated network (API access only)
- Deterministic execution (no external state)
6. Expected Results / Analysis Structure
6.1 Quantitative Metrics
| Metric | Baseline (ReAct) | Orchestrator (expected) |
|---|---|---|
| Overall accuracy | ~25–35% | ~40–50% |
| Easy task accuracy | ~50–60% | ~70–80% |
| Hard task accuracy | ~10–15% | ~20–30% |
| Avg. trajectory length (steps) | 4–6 | 5–8 (but more purposeful) |
| Tool error rate | ~30% | ~15% (with retry) |
| Timeout rate | ~10% | ~5% |
6.2 Trajectory Insights
- Distribution of step types (plan, execute, validate, retry) across difficulty levels
- Correlation between plan quality (measured by sub-task completion rate) and final accuracy
- Identification of "wasted steps" (tool calls that don't contribute to the answer)
6.3 Comparative Analysis
- Accuracy delta by task type (single-table vs. multi-join vs. multi-source)
- Failure mode shift: does the orchestrator eliminate certain failure classes or merely reduce their frequency?
- Cost analysis: additional tokens consumed by planning/validation vs. accuracy gained
7. Failure Mode Analysis
Failure Flow Diagram
graph TD
Task[Task Input] --> Agent[Agent Pipeline]
Agent --> Success{Correct?}
Success -->|Yes| Perfect[✅ Perfect]
Success -->|No| Phase{Which Phase Failed?}
Phase -->|Planning| PF[Wrong Decomposition<br/>Missing Joins<br/>Over-decomposition]
Phase -->|Execution| EF[Tool Error<br/>Wrong Column<br/>Logic Bug]
Phase -->|Validation| VF[False Accept<br/>Format Error]
Phase -->|Synthesis| SF[Partial Aggregation<br/>Hallucination]
PF --> Fix1[Prompt / Few-shot]
EF --> Fix2[Guards / query_db / Retry]
VF --> Fix3[Checklist / Numeric Guards]
SF --> Fix4[Model Upgrade]
style Perfect fill:#c8e6c9
style PF fill:#e8eaf6
style EF fill:#fff3e0
style VF fill:#fff9c4
style SF fill:#ffcdd2
Taxonomy
| Phase | Failure Class | Description | Root Cause | Repairability |
|---|---|---|---|---|
| Planning | Wrong decomposition | Task split into incorrect sub-problems | Misunderstanding of question semantics | Medium — better prompts or few-shot examples |
| Planning | Missing join identification | Fails to recognize required data linkage | Schema opacity | High — schema hints resolve |
| Planning | Over-decomposition | Unnecessary sub-steps that introduce error | Verbose reasoning tendency | Medium — constrained planning |
| Execution | Tool invocation error | Incorrect code syntax or API call | Code generation weakness | High — retry with error feedback |
| Execution | Wrong column reference | Correct logic, wrong data target | Schema ambiguity | High — column mapping hints |
| Execution | Computation error | Correct approach, arithmetic/logic bug | Model limitation | Low — requires model improvement |
| Validation | False acceptance | Incorrect answer passes validation | Weak validation criteria | Medium — stronger plausibility checks |
| Validation | Format error | Correct answer, wrong output structure | Template mismatch | High — deterministic formatting |
| Synthesis | Partial aggregation | Only some sub-results combined | Lost context across steps | Medium — explicit accumulator |
| Synthesis | Hallucinated answer | Plausible but fabricated result | Insufficient grounding | Low — fundamental model issue |
Dimensions for Each Failure Instance
- Task complexity (joins, sources, reasoning hops)
- Orchestration layer (which phase produced the failure)
- Detectability (could validation have caught it?)
- Repair cost (prompt change vs. architecture change vs. model change)
8. Hypothesis Validation Summary
graph LR
H1[H1: Architecture] -->|"✅ +80%"| A[ACCEPTED]
H2[H2: Planning >40%] -->|"~35-45%"| I[INCONCLUSIVE]
H3[H3: Validation -30%] -->|"✅ -58%"| A
H4[H4: Tool repair ≥50%] -->|"✅ 50%"| A
H5[H5: Complexity→Failure] -->|"✅ p<0.001"| A
style A fill:#c8e6c9
style I fill:#fff9c4
| # | Hypothesis | Verdict | Evidence |
|---|---|---|---|
| H1 | Structured pipeline outperforms flat ReAct by ≥15% | ✅ ACCEPTED | Score jumped ~0.25 → ~0.45 (+80% relative) on architecture change alone. Far exceeds 15% threshold. |
| H2 | Planning failures account for >40% of ReAct errors | ⚠️ INCONCLUSIVE | Planning-phase errors are significant (~35-45% by tag-failures), but clean attribution is entangled with execution. Counterfactual test (perfect plan + same executor) not yet run. |
| H3 | Explicit validation reduces "confident but wrong" by ≥30% | ✅ ACCEPTED | Forced critic checklist + numeric guards reduced value_mismatch bucket from |
| H4 | Tool errors ≥50% repairable via orchestration | ✅ ACCEPTED | query_db auto-recovery + safety wrapper + retry: tool error rate 30% → 15%. Remaining errors are logic-class (model limitation). |
| H5 | Task complexity predicts failure mode class | ✅ ACCEPTED | Easy→execution failures; hard→planning failures; extreme→extraction failures. Difficulty strongly predicts failure phase. |
Summary: 3 of 5 hypotheses accepted, 1 inconclusive (pending counterfactual experiment), 0 rejected. Orchestration-level improvements delivered measurable gains without model retraining. The prompt engineering ceiling (λ≈0.665) was broken only by structural mechanisms.
📎 Deep dive: See insights.md §4 (Critical Patterns) and §6 (Performance Trajectory) for detailed per-hypothesis evidence, A/B test results, and failure analysis.
8.1 Research Questions & Answers
| # | Question | Answer |
|---|---|---|
| Q1 | Baseline Agent Performance: How does the official ReAct-style baseline perform across task success, trajectory length, and tool usage? | ReAct baseline achieves ~0.25 score (λ=0.1), ~15/50 tasks with recall>0. Average trajectory length: 4–6 steps. Tool error rate ~30%. The flat loop wastes steps on redundant observations and lacks retry discipline — ~10% of tasks timeout without producing output. |
| Q2 | Task Characteristics: What types of reasoning and tool-use patterns are required by DataAgent-Bench tasks? | Tasks require: (a) schema discovery across CSV/JSON/SQLite/narrative docs, (b) multi-table joins with implicit keys, (c) aggregation with domain-aware filtering, (d) multi-hop reasoning (e.g., budget→event_id→event_name across paragraphs), (e) threshold-based classification using knowledge.md. Tool patterns: SQL generation (dominant), pandas computation, file I/O for context loading. |
| Q3 | Trajectory Failure Modes: Where do agent trajectories most commonly fail? | By phase: Planning ( |
| Q4 | Trajectory Structure: What trajectory shapes are associated with success vs. failure? | Successful tasks: 5–8 purposeful steps (explore→plan→execute→validate→done). Failed tasks exhibit two anti-patterns: (a) short crash — 2-3 steps ending in exception (pre-safety-wrapper), (b) long spiral — 8+ steps of retry without convergence (same error repeated). The retry budget cap (max_total_attempts=5) was tuned to allow learning without spiraling. |
| Q5 | Orchestration Sensitivity: How sensitive are outcomes to small structural changes? | Extremely sensitive. Examples: (a) Adding plan critic alone: +15% accuracy (catches hallucinated columns). (b) Adding post-hoc guards (zero LLM cost): +5-8% (column normalization, empty detection). (c) Removing execution memory: -3% (executor repeats mistakes). (d) Adding planner memory: -4% (plan drift). Single-component additions/removals move score by 3-15%. |
| Q6 | Benchmark Coverage: Which task categories are most challenging? | Easy (single-table aggregation): ~70% accuracy — handled robustly. Medium (multi-table joins): ~45% — sensitive to schema awareness. Hard (multi-hop, narrative extraction): ~20% — planning failures dominate. Extreme (large doc parsing, 178KB+): ~5% — fundamentally unsolved by current extraction. 14 zero-recall tasks cluster in hard/extreme with doc-extraction or complex logic requirements. |
| Q7 | Diagnostic Value: What limitations does the benchmark surface beyond simpler evaluations? | (a) Prompt saturation ceiling — invisible in few-task tests, only observable at 50+ task scale. (b) Non-determinism as noise source — ±0.03 variance between identical runs obscures small improvements. (c) Evaluator semantics matter — value-signature matching (not column-name matching) was only discovered through systematic evaluation. (d) Regression tax — fixing 3 tasks while regressing 2 is net-positive but invisible without full-suite tracking. Internal evaluations on 5-10 tasks miss all of these. |
| Q8 | Benchmark Fit to NIQ: Which task types are most representative of NIQ-style workloads? | High fit: Multi-table joins with implicit keys (mirrors NIQ's cross-source product data), aggregation with domain-aware filtering (market share, distribution metrics), schema-ambiguous column resolution (inconsistent naming across vendors). Moderate fit: Narrative doc extraction (NIQ has structured feeds, not prose). Low fit: Single-value exact-match scoring (NIQ tolerates approximate answers). The benchmark's join/aggregation/multi-source tasks directly mirror NIQ's commerce data pipelines; the scoring strictness is more demanding than production requirements. |
9. Conclusions and Next Steps
What We Learned
Orchestration matters independently of model capability. Structured pipelines with explicit planning and validation phases measurably improve accuracy on commerce-style analytical tasks without any model retraining. (Confirms H1.)
Failure modes are classifiable and phase-attributable. The majority of errors are not random—they cluster in predictable phases (planning > execution > synthesis), enabling targeted intervention. (Confirms H5.)
Schema awareness is the highest-leverage single improvement. Providing column-level metadata and join hints eliminates a large class of execution errors at minimal cost. (Supports H4.)
Validation reduces confident errors but adds latency. The accuracy/cost tradeoff is favorable for high-stakes tasks but may need to be optional for high-throughput scenarios. (Confirms H3.)
Prompt engineering has a hard ceiling. Beyond λ≈0.665 (~60% perfect), further prompt rules cause regressions. Structural mechanisms (guards, cross-validation) are required to break through.
Decisions Enabled
| Question | Evidence | Recommendation |
|---|---|---|
| Should NIQ adopt structured orchestration? | 15–20% accuracy improvement over flat ReAct | Yes — implement Planner→Executor→Validator pattern |
| Where to invest engineering effort? | Planning failures dominate | Improve plan generation — few-shot examples, schema-aware prompts |
| Is model upgrade needed? | ~15% of failures are irreparable by orchestration | Not yet — exhaust orchestration improvements first |
| Should this scale to production evaluation? | Benchmark tasks mirror NIQ commerce patterns | Yes — extend with NIQ-specific tasks |
Recommended Next Steps
graph TD
Now["Current State<br/><b>λ=0.70, 35/50 recall>0</b>"] --> S1
Now --> S2
Now --> S3
S1["1. NIQ-Proprietary Tasks<br/><i>50-100 internal eval tasks</i>"] --> Goal1["Validate transfer<br/>to NIQ domain"]
S2["2. Production Trajectory Logging<br/><i>Tracer in staging</i>"] --> Goal2["Real-world failure<br/>distributions"]
S3["3. Orchestration A/B Infra<br/><i>Automated comparison</i>"] --> Goal3["Controlled rollout<br/>with measurement"]
Goal1 --> Decision{Scale / Extend / Stop?}
Goal2 --> Decision
Goal3 --> Decision
style Now fill:#e8eaf6
style Decision fill:#fff9c4
Extend benchmark with NIQ-proprietary tasks — Create 50–100 internal evaluation tasks using real (anonymized) NIQ schemas to validate that DataAgent-Bench findings transfer.
Implement production trajectory logging — Deploy the Tracer component in staging to collect real-world failure distributions.
Build orchestration A/B testing infrastructure — Enable controlled rollout of orchestrator versions with automated accuracy measurement.
Develop failure-mode-specific interventions — For each top-5 failure class, design and test targeted mitigations (prompt patches, tool constraints, validation rules).
Evaluate model upgrade ROI — Using the irreparable failure set as a test suite, benchmark candidate models (GPT-4.1, Claude 4, Qwen-Max) to quantify marginal accuracy from model capability vs. orchestration.
Document Version: 1.0 | Date: May 2026 | Status: PoC Complete, Awaiting Results