RESEARCH PREVIEW · CDA

Concord Data Agents

Two agents that scan an undocumented, decades-old legacy table, infer its schema with zero manual mapping, and auto-generate the cleaning rules — turning the 80% of a data scientist's time spent on cleanup into a few seconds.

Launch live demo Read the code

HEADLINE RESULT

Agent 1 correctly mapped all 10 undocumented columns to canonical fields with zero manual schema work, and Agent 2's cleaning pipeline removed 8 duplicate rows from 258 raw records while standardizing 4 mixed date formats and 8 missing-value conventions.

10 / 10

legacy columns correctly mapped to canonical fields

258 → 250

rows after deduplication

legacy date formats parsed into one ISO standard

different missing-value sentinels normalized to one

METHOD

Schema first, cleaning rules second

Agent 2 never touches a column Agent 1 hasn't already classified — the same boundary a LangGraph/CrewAI handoff would enforce in production.

Scan

Agent 1 profiles every column: sample values, distinct-value count, and pattern matching infer a canonical name, type, and specific issues.

Map

Cryptic mainframe names (PT_NM, CLM_AMT_STR) are resolved to canonical fields (patient_name, claim_amount) using keyword + pattern heuristics.

Clean

Agent 2 dispatches a cleaning rule per inferred type: date parsing across 4 formats (with a proper 2-digit-year pivot), currency coercion, categorical normalization, name standardization.

Dedupe

Exact duplicate rows — another classic legacy-data artifact — are dropped after cleaning.

Score

A before/after quality dashboard measures completeness and validity per column, so the improvement is measured, not asserted.

FINDINGS

What the synthetic test runs showed

A 2-digit-year date format (10-NOV-60) initially parsed to the year 2060 under Python's default pivot — a real Y2K-style bug caught during testing and fixed with an explicit >30 → 19xx pivot rule.

Two columns (PT_NM patient name, PROV_NM provider name) initially collided into a single canonical field because of an overly broad keyword hint — fixed by making the schema-inference hints column-specific.

Categorical columns arrived with 8-9 raw spellings each for what should be a 2-4 value controlled vocabulary (gender, policy status, region) — all normalized in a single pass.