Technical_README.md

Agentic Data Engineer - Qwen3-Coder-30B-A3B-Instruct

Note 1 : Do not upload confidential, regulated, proprietary, customer, or personally identifiable production data into a public Hugging Face Space

Note 2: Data Engineering Agent demo/snippet video will be placed within the file directory if the Hugging Face Spaces runtime has any issues

Note 3: model synthesis = last step in llm-model system where individual predictions are combined into a single, final output

Note 4: Deterministic Workflow Agent: Deterministic Workflow Agent follows predefined rules and steps to complete tasks consistently and predictably over autonomous decision-making.

(this) Deterministic Workflow Agent vs Autonomous Decision-Making Agent

Deterministic Workflow Agent (my agent)

“Follow the rules and steps I was given.”

Autonomous Agent

“Figure out the best way to achieve the goal.”

	Deterministic Workflow	Autonomous Agent
Consistency	Very high	Can vary
Traceability	Easy to track	Can be harder to track
Repeatability	Same input → predictable output	May choose different actions
Decision-making	Predefined rules	Makes decisions dynamically
Flexibility	Lower	Higher

Example:

• Workflow: If an invoice is over $10,000 → send it to a manager.
• Autonomous agent: Examine the invoice, assess the situation, decide whether manager approval is needed, and determine what to do next.

Deterministic agent executes a known plan, while an autonomous agent figures out the plan as it goes.

Agent Goals:

• agentic data engineering app for profiling datasets, validating schemas, detecting data-quality failures, executing safe read-only SQL checks, and generating SQL + PySpark pipelines

• agent model loads a user dataset, creates a structured DataContext, gives a Qwen3-Coder agent access to  engineering tools, and lets the model decide which tools to call before producing a final engineering assessment and pipeline code

Data Cleaning / Audit Tasks

Data Engineering Agent can analyze and clean datasets by first understanding their structure and quality

Dataset inspection

The inspect_dataset tool can analyze:

• Shape: number of rows and columns
• Types: logical and pandas data types
• Nulls: missing-value counts and percentages
• Cardinality: number of unique values per column
• Examples: representative values from each column
• Duplicates: duplicate row counts
• Memory usage: approximate DataFrame memory footprint

Data-quality checks

run_quality_checks tool can detect:

• duplicate rows
• missing values
• blank or whitespace-only strings
• constant columns
• mixed Python value types
• likely numeric conversion failures
• extreme numeric values
• schema inconsistencies

findings can then be converted into:

Recommended fix
    ↓
SQL cleanup pipeline
    ↓
PySpark cleanup pipeline

What, Why, and General Objectives

What

Tool-using Data Engineering AI agent that profiles datasets, identifies quality problems, validates schema expectations, verifies SQL, and generates SQL + PySpark pipelines

Why

Data engineers repeatedly perform the same early-stage tasks when debugging datasets:

   analyze data
         ↓
identify schema problems
          ↓
find quality failures
          ↓
test transformation logic
          ↓
write remediation pipelines

Automating this workflow reduces repetitive manual data cleaning + analysis

LLM / Agent Stack

• Agent LLM: Qwen/Qwen3-Coder-30B-A3B-Instruct

• Inference: huggingface_hub.InferenceClient

• Agent pattern: bounded single-agent function-calling / tool-use loop

• Tool routing: tool_choice="auto"

• Maximum agent steps: 4 by default (hard safety limit on the number of loop iterations or tool calls an AI agent can perform during a single run before it is forced to stop)

  • Prevent Infinite Loops: param limit helps the agent stops from getting stuck repeating the same failed tool call or logic pattern indefinitely
  • Control Costs: limits excessive (overkill) API requests and token consumption that rack up expenses during a normal run (saves $$$$$$)
  • Manage Context Windows: prevents long chains of tool outputs and intermediate thoughts from overflowing the model's memory limits
  • Avoid Latency and Timeouts: makes sure loops don't wait for a complex or broken automation to finish

• SQL execution: DuckDB

• SQL parsing / validation: SQLGlot

• Dataset processing: pandas

• Parquet support: PyArrow

• Excel support: OpenPyXL

• Generated Code: PySpark

Goal:

data engineering is based around finding/ingesting source data, checking schema assumptions, finding quality failures, validating transformation logic, and transforming those findings into production-ready pipeline code

doing this manually across diffrerent datasets requires a large amount of repetitive analyzing and debugging

this Data Engineering Agent is designed to:

• reduce time spent manually profiling new datasets

• identify schema and data-quality risks before downstream processing

• validate expected schema contracts against real uploaded data

• allow the LLM to call data transformation functions

• validate generated SQL

• generate SQL pipelines

• generate PySpark pipelines for Spark / Databricks-style workloads

• expose the agent's tool calls through a trace for auditability

Agentic Workflow:

Uploaded dataset
      ↓
   app.py
      ↓
Dataset profiling + session state
      ↓
User engineering task
      ↓
DataEngineeringAgent
      ↓
Qwen3-Coder-30B-A3B
      ↓
HF InferenceClient
      ↓
tool_choice="auto"
      ↓
Agent decides which engineering tool(s) to call
      ↓
┌─────────────────────────────────────────────┐
│ analyze                             │
│ run_quality_checks                          │
│ validate_expected_schema                    │
│ validate_sql                                │
│ execute_readonly_sql                        │
└─────────────────────────────────────────────┘
      ↓
Tool observations returned to Qwen
      ↓
Agent evaluates whether another tool is needed
      ↓
Bounded tool-use loop
      ↓
Engineering assessment
      ↓
Data-quality findings
      ↓
Recommended fixes
      ↓
SQL Pipeline
      ↓
PySpark Pipeline
      ↓
Tool Trace + Diagnostics

Agent Loop Type

Bounded Single-Agent Function-Calling / Tool-Use Loop

• agentic pattern where AI agent autonomously calls external tools, processes the results, and repeats the process within a strictly defined limit

• unlike open-ended agents that run indefinitely, bounded loop agents implement hard-fixed constraints to prevent runaway api costs (saves $$$$$$), infinite loops, and resource exhaustion

Architecture:

observe task → choose tool → execute task using tool → return output → repeat if needed → synthesize final answer

note: single-agent architecture (not a multi-agent architecture)

LLM does not perform every data engineering operation itself, the DataEngineeringAgent exposes  Python tools to Qwen through Hugging Face function calling

llm model receives:

• System instructions

• User engineering task

• Target output

• SQL dialect

• Expected schema state

• Available tool definitions

model can then request one or more tool calls.

Example (input) Workflow:

Qwen
  ↓
inspect_dataset()
  ↓
dataset profile returned to Qwen
  ↓
run_quality_checks()
  ↓
quality report returned to Qwen
  ↓
validate_expected_schema()
  ↓
schema result returned to Qwen
  ↓
validate_sql(candidate_sql)
  ↓
SQL validation result returned to Qwen
  ↓
execute_readonly_sql(candidate_sql)
  ↓
query result returned to Qwen
  ↓
Final engineering response

loop is bounded by:

MAX_AGENT_STEPS = 4

if the model hasn't given a final answer after the configured loop, the app asks it to synthesize the final response without calling additional tools

this makes the workflow/architecture closer to a controlled tool-calling agent loop

Agent Architecture:

                         USER
                          │
                          ▼
                 ┌─────────────────┐
                 │     app.py      │
                 │ Gradio UI       │
                 │ Session state   │
                 │ Dataset upload  │
                 └────────┬────────┘
                          │
                          │ task + dataset + schema
                          ▼
              ┌────────────────────────┐
              │ DataEngineeringAgent   │
              │       agent.py         │
              └────────────┬───────────┘
                           │
                           ▼
              ┌────────────────────────┐
              │ Qwen3-Coder 30B A3B    │
              │ HF InferenceClient     │
              │ tool_choice="auto"     │
              └────────────┬───────────┘
                           │
                  decides which tool
                      to execute
                           │
       ┌───────────────────┼────────────────────┐
       │                   │                    │
       ▼                   ▼                    ▼
inspect_dataset    run_quality_checks   validate_expected_schema
       │                   │                    │
       └───────────────────┼────────────────────┘
                           │
                           ▼
                      validate_sql
                           │
                           ▼
                  execute_readonly_sql
                           │
                           ▼
                    Tool observation
                           │
                           ▼
              ┌────────────────────────┐
              │   Observation added    │
              │   back to LLM context  │
              └────────────┬───────────┘
                           │
                           ▼
                  More tools needed?
                     /           \\
                   yes            no
                    │              │
                    └──────┐       │
                           │       │
                           ▼       ▼
                      Tool loop   Final synthesis (last step in llm-model system where individual predictions are combined into a single, final output)
                                      │
                                      ▼
                           Engineering assessment

                           Data-quality findings

                             Recommended fixes

                               SQL Pipeline

                              PySpark Pipeline

                                Tool Trace

Core Architectural Patterns:

app.py
   ↓
agent.py
   ↓
DataEngineeringAgent
   ↓
Qwen3-Coder + HF tool calling
   ↓
data_engine.py (transformation functions)
   ↓
tool observations
   ↓
agent.py
   ↓
final SQL + PySpark + engineering assessment

Agent Tools

1. inspect_dataset

analyzes the uploaded dataset and returns:

• source
• row count
• column count
• memory usage
• duplicate row count
• column names
• logical types
• pandas dtypes
• null percentages
• unique counts
• example values

used when the agent needs to understand the structure of the dataset before making recommendations

---

2. run_quality_checks

runs data quality checks for:

• duplicate rows
• missing values
• constant columns
• blank / whitespace-only strings
• mixed Python value types
• likely numeric cast failures
• extreme numeric values

each issue can include:

• severity
• check name
• column
• evidence
• recommended fix

severity levels:

• high
• medium
• low

---

3. validate_expected_schema

compares the uploaded dataset against the user-editable expected JSON schema

Supported checks include:

• missing columns
• unexpected columns
• logical type mismatches
• nullable=false violations
• unique=true violations

Example schema:

{

  "columns": {

    "order_id": {
      "type": "integer",
      "nullable": false,
      "unique": true
    },

    "customer_id": {
      "type": "integer",
      "nullable": false
    },

    "amount": {
      "type": "float",
      "nullable": false
    },

    "status": {
      "type": "string",
      "nullable": true
    }
  }
}

---

4. validate_sql

Uses SQLGlot to parse and normalize candidate SQL

validation result can return:

• valid
• dialect
• normalized SQL
• parse error

Supported UI dialect selections include:

• DuckDB
• Spark
• PostgreSQL
• MySQL

---

5. execute_readonly_sql

executes safe SQL against an in-memory DuckDB table named: dataset

tool parses the SQL before execution and blocks write-oriented operations such as:

• INSERT
• UPDATE
• DELETE
• CREATE
• DROP
• ALTER
• COPY

returned metrics include:

• execution status
• rows returned
• column names
• result preview
• error information

this allows the agent to verify SQL outputs against the actual uploaded dataset

Deterministic Fallback

app also includes a non-agent fallback path

HF_TOKEN configured?

        │
   ┌────┴────┐
   │         │
  yes        no
   │         │
   ▼         ▼
Qwen agent   deterministic fallback
tool loop            │
   │                 ├── inspect_dataset
   │                 ├── run_quality_checks
   │                 ├── validate_expected_schema
   │                 ├── baseline SQL
   │                 └── baseline PySpark
   │
   └────────────┬────────────
                ▼
            UI output

fallback is also used if the Hugging Face model request fails

this means the application can still:

• analyze the uploaded dataset
• run built-in quality checks
• validate the schema
• generate a SQL pipeline
• generate a PySpark pipeline

Dataset Support

Uploaded dataset files can include:

1. CSV
2. Parquet
3. JSON
4. JSONL
5. XLSX
6. XLS

current defaults are:

• MAX_UPLOAD_MB = 50

• MAX_PROFILE_ROWS = 100000

app profiles up to the configured row limit before exposing the dataset to the engineering workflow

Agent Outputs

final agent response has to contain:

1. Engineering assessment
2. Data-quality findings
3. Recommended fixes
4. SQL Pipeline
5. PySpark Pipeline

app separately extracts the generated code blocks into dedicated UI outputs

returned app values:

• answer
• sql
• pyspark
• trace

Agent Metrics / UI

1. Dataset Workspace

dataset workspace shows:

Data preview

• preview of the uploaded dataset

Inferred schema

• column
• logical type
• pandas dtype
• null percentage
• unique count
• example values

2. Data Health

quality workflow shows:

Quality audit

• issue count
• high severity count
• medium severity count
• low severity count

Schema validation

• PASS / FAIL
• errors
• warnings
• expected columns
• actual columns

3. Agent Studio

Agent controls include:

SQL dialect

• duckdb
• spark
• postgres
• mysql

Target output

• SQL + PySpark
• SQL only
• PySpark only

Temperature

controls model generation randomness:

• 0.00  Highly deterministic

• 0.15  Default

• 0.80  More variable

• lower temperature is generally preferred for engineering related tasks

• makes the output more predictable, deterministic, and focused by consistently selecting the tokens with the highest probabilities (answers tailored to factualness rather not creativity)

4. Tool Trace

Every transformaton function invocation is appended to the agent trace

Example conceptual trace:

[
  {
    "tool": "inspect_dataset",
    "arguments": {},
    "result": {
      "rows": 25000,
      "columns": 14
    }
  },

  {
    "tool": "run_quality_checks",
    "arguments": {},
    "result": {
      "issue_count": 8,
      "high": 1,
      "medium": 5,
      "low": 2
    }
  }
]

 trace allows the user to analyze/check out:

• which tools were called
• arguments supplied to each tool
• tool observations
• number of tool calls

5. Diagnostics

runtime diagnostics include:

• model
• provider
• HF token configured
• tool-call count
• tools used

Example:

{
  "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct",
  "provider": "HF automatic routing",
  "hf_token_configured": true,
  "tool_calls": 4,
  "tools_used": [
    "inspect_dataset",
    "run_quality_checks",
    "validate_expected_schema",
    "validate_sql"
  ]
}

SQL + PySpark Execution Model

DuckDB is used for lightweight SQL validation inside the Hugging Face Space

Generated PySpark is not executed inside the application

Workflow:

Uploaded dataset
      ↓
pandas DataFrame
      ↓
DuckDB in-memory registration
      ↓
read-only SQL validation / execution

PySpark:

Dataset + agent findings
      ↓
Qwen3-Coder
      ↓
Generated PySpark
      ↓
Production target code

full Spark runtime would require a JVM and additional memory / startup overhead, so the Space generates Spark-ready PySpark rather than running Apache Spark directly

File Descriptions

agent.py

file that holds agent implementation

Contains:

• SYSTEM_PROMPT
• TOOL_SCHEMAS
• DataEngineeringAgent
• _tool()
• _fallback()
• run()
• agent tool loop
• HF InferenceClient calls
• tool_choice="auto"
• tool observations
• final response extraction

main file controlling agentic behavior

data_engine.py

transformation function calling / data-execution layer.

made up of:

• DataContext

• dataset loading

• dataset profiling

• logical-type inference

• quality checks

• schema validation

• SQLGlot validation

• DuckDB read-only SQL execution

• baseline SQL generation

• baseline PySpark generation

Architecturally:

agent.py
   ↓
decides WHAT data engineering action is needed

data_engine.py
   ↓
executes the data operation

app.py

Gradio UI and orchestration layer

config.py

agent and application runtime configuration

Agent System Prompt

agent is told to behave/act like as a senior Data Engineering Agent

Core behaviors:

• use tools to verify dataset claims before generating pipelines

• identify concrete schema and data-quality risks

• propose production-oriented fixes

• generate SQL and PySpark grounded in the uploaded dataset

• never claim generated PySpark was executed inside the application

final generated response is required to use:

• Engineering assessment

• Data-quality findings

• Recommended fixes

• SQL Pipeline

• PySpark Pipeline

Security Behavior

application restricts SQL execution to read-only analytical queries

Before DuckDB execution:

SQL
 ↓
SQLGlot parse
 ↓
AST inspection
 ↓
write operations blocked
 ↓
DuckDB execution

Blocked SQL operation classes include:

• INSERT
• UPDATE
• DELETE
• CREATE
• DROP
• ALTER
• COMMAND
• COPY

SQL is executed against an in-memory table named: dataset

Agent Pattern Workflow:

Single Agent
    +
Function Calling
    +
Deterministic Engineering Tools
    +
Bounded Tool Loop
    +
Tool Observations
    +
Final LLM Synthesis

full execution:

USER
 ↓
Gradio / app.py
 ↓
DataEngineeringAgent / agent.py
 ↓
Qwen3-Coder
 ↓
tool_choice="auto"
 ↓
Engineering Tool
 ↓
data_engine.py
 ↓
Observation
 ↓
Qwen3-Coder
 ↓
repeat up to MAX_AGENT_STEPS
 ↓
Final Engineering Assessment
 ↓
SQL
 ↓
PySpark
 ↓
Tool Trace

Core Runtime Components

DuckDB

What it is: DuckDB is an in-process analytical SQL database designed for fast local analytics on structured data such as pandas DataFrames, CSV files, and Parquet files.

Why it is used in this project: The Hugging Face Space needs a lightweight way to execute and verify SQL without running a full database server.

How it is used here:

Uploaded dataset
       ↓
pandas DataFrame
       ↓
Registered in DuckDB as: dataset
       ↓
Agent generates / validates SQL
       ↓
Safe read-only SQL executes in DuckDB
       ↓
Results returned to the agent

DuckDB is used to:

• execute read-only validation queries
• test generated SQL against the actual uploaded dataset
• analyze transformed results before final agent synthesis
• provide SQL execution without requiring PostgreSQL, MySQL, Snowflake, or another external database
• keep the Hugging Face Space lightweight

---

DuckDB Sandbox

What it is: The DuckDB sandbox is the project's controlled SQL execution environment. It allows the agent to run analytical SQL against the in-memory dataset table while blocking write-oriented operations.

Why it is used: Generated SQL should be verified against real data, but an AI agent should not have unrestricted database write access.

How it is used here:

Candidate SQL
    ↓
SQLGlot parsing
    ↓
AST / operation validation
    ↓
Write operations blocked
    ↓
DuckDB sandbox
    ↓
Read-only query result

sandbox blocks operations such as:

• INSERT
• UPDATE
• DELETE
• CREATE
• DROP
• ALTER
• COPY
• COMMAND

gives the agent a safe execution layer for testing SQL logic before returning the final pipeline

---

Spark JVM Runtime

What it is: Apache Spark normally runs on the Java Virtual Machine (JVM). PySpark is the Python interface to Spark, but executing PySpark still requires a Spark runtime and JVM underneath it

Why it is not executed inside this project: Running Spark directly inside a lightweight Hugging Face Space would add significant startup time, memory usage, JVM dependencies, and deployment complexity

How Spark is used here:

 Dataset
    ↓
Agent findings
    ↓
Qwen3-Coder
    ↓
Generated PySpark
    ↓
Spark / Databricks-ready production code

app:

• generates PySpark
• does not execute PySpark inside the Space
• uses DuckDB for lightweight local SQL verification
• treats Spark / Databricks as the downstream production execution environment

DuckDB

What: lightweight analytical SQL database that runs directly inside the application without requiring a separate database server

How it is used in this project: uploaded datasets are loaded into a pandas DataFrame and registered in DuckDB as an in-memory table named dataset.

Uploaded dataset
    ↓
pandas DataFrame
    ↓
DuckDB in-memory table: dataset
    ↓
Read-only SQL validation / execution
    ↓
Results returned to the agent

DuckDB is used to:

• execute safe read-only SQL
• test generated SQL against the uploaded dataset
• validate transformation logic before final output
• avoid running a full external database server inside the Hugging Face Space

HF Automatic Routing

LLM orchestration service that dynamically direct user prompts to the most optimal or cost-effective model based on context, complexity, or task type What it is: Hugging Face automatic routing lets InferenceClient choose an available Hugging Face inference provider for the configured model when a specific provider is not explicitly set.

Why it is used: It avoids hard-coding the application to one inference backend and keeps model deployment simpler.

How it is used here:

DataEngineeringAgent
         ↓
huggingface_hub.InferenceClient
         ↓
HF automatic routing
         ↓
Available provider for Qwen3-Coder
         ↓
Model response / tool call

Within this project:

• HF_MODEL_ID identifies the model
• HF_PROVIDER can optionally force a provider
• when HF_PROVIDER is empty, Hugging Face automatic routing is used
• the routed model performs tool selection and final engineering tasks