Can Skills Improve Codex’s Data Analysis Capabilities?

Authors: Zhisong Qiu, Chen Yan, and Jiafu Wei
DOI: 10.5281/zenodo.21292469

In our previous evaluation, we found that Codex already has strong data analysis capabilities. It can read files, write scripts, run statistical analyses, generate visualizations, and organize the results into reports.

However, in tasks that more closely resemble real-world scenarios, failures often arise from analytical conventions that are not explicitly stated in the prompt but must still be followed to satisfy the underlying business logic.

For example, in payment data analysis:

• Should all applicable fee rules be matched and accumulated, or should only the single best-matching rule be used?
• When calculating monthly transaction volume, should the calculation use the entire query period or be performed separately for each calendar month?

When these conventions are not clearly specified, Codex may produce an answer that appears reasonable and is supported by executable code. However, its calculation method may still be inconsistent with the business rules assumed by the task or with the user’s expectations.

This leads to the central question of this study:

If we turn task rules, field definitions, calculation procedures, and lessons from common failures into reusable Skills, can they improve Codex’s accuracy on similar unseen tasks?

To investigate this question, we conducted a set of experiments using the open-source DataCOPE framework. DataCOPE extracts reusable experience from an agent’s problem-solving trajectories and automatically organizes it into Skills that can support subsequent data analysis tasks.

Our experiments cover two types of scenarios.

The first covers scientific data analysis and includes one small-scale experiment:

• DSBio Spatial Neighbor: a spatial cell-neighborhood analysis task. It evaluates whether Codex can correctly filter the data, construct spatial neighborhood relationships, compare pairs of cell types, and return the result in the required format.

The second covers data analysis tasks based on realistic payment workflows. These tasks come from the DABstep dataset and include two task types:

• Highest Cost Scenario Identification: given a transaction amount, card scheme, and credit or debit condition, identify the ACI, MCC, or other transaction attribute associated with the highest fee. Here, ACI stands for Authorization Characteristics Indicator, which describes characteristics of the transaction authorization process, while MCC stands for Merchant Category Code.

• Routing and Cost Optimization: for a specified merchant and time range, identify the card scheme with the lowest or highest cost, or choose a lower-cost ACI for fraudulent transactions. Card schemes include payment networks such as Visa and Mastercard.

We focus on the following questions:

• How do we generate Skills?
• How do Skills improve Codex’s data analysis capabilities?
• What are the limitations of Skills?
• How can Skills be used more effectively for data analysis?

Our main conclusion is:

Skills can capture implicit rules, standardized procedures, and recurring failure patterns in data analysis, thereby improving Codex’s accuracy on similar tasks. However, they still depend on humans to provide clear problem definitions, data conventions, and validation of the final results.

---

How Do We Generate Skills?

1. The Open-Source DataCOPE Framework

We use the open-source DataCOPE framework to generate and iteratively improve Skills.

DataCOPE is a data analysis Skill-generation framework released as part of our DataMind project. It supports extracting experience from agent exploration trajectories, verifying the generated results, and producing reusable Skills consisting of a SKILL.md file and optional supporting scripts.

The framework contains three main components:

• Data-Analytic Agent: performs the actual data analysis task, including reading data, writing code, running calculations, and producing an answer. DataCOPE supports agents such as Codex, Claude Code, and ReAct.
• Verifier: evaluates the agent’s trajectories using a preconfigured verification method. For example, it can compare the agent’s answer with a reference answer, determine the quality of the trajectory, and pass useful examples to the next stage.
• Skill Manager: organizes the experience selected by the Verifier into a Skill or updates an existing Skill in later iterations. It can record field definitions, standard procedures, and common mistakes so that the agent relies less on guesswork when solving similar tasks.

This process can run for multiple iterations.

In iteration 0, the agent completes the training tasks without using a Skill. The Verifier evaluates its trajectories, and the Skill Manager generates the initial Skill from the resulting successes and failures.

Starting from iteration 1, the agent repeats the tasks using the Skill generated in the previous iteration. The Verifier evaluates the new trajectories, and the Skill Manager updates the Skill based on the newly observed experience.

In other words, DataCOPE analyzes training-task trajectories and converts the experience contained in those trajectories into reusable analytical rules.

The overall framework is illustrated below:

2. Experimental Setup and Results

In our experiments, we divide the tasks into a training set and a test set.

The training set is used by DataCOPE to generate and improve the Skills. The test set is used to evaluate whether those Skills can transfer to unseen tasks of the same type. The test set remains completely hidden throughout Skill generation and refinement.

In addition to the initial Skill creation in iteration 0, we perform one additional Skill-improvement iteration. The full process therefore contains two rounds of agent execution.

We use Codex GPT-5.5 as the Data-Analytic Agent and Claude Code Sonnet 4.6 as the Skill Manager. We also use a supervised Verifier based on reference answers to distinguish successful trajectories from failed ones and provide the verification results to the Skill Manager.

The training set contains:

• 1 DSBio Spatial Neighbor task;
• 8 DABstep Highest Cost tasks;
• 6 DABstep Routing tasks.

These tasks are used to generate and improve the corresponding Skills.

The test-set results are shown below. All numbers come from a single experimental run, and our analysis focuses primarily on concrete examples and failure patterns.

Within the scope of this small-scale experiment and the selected tasks, these results suggest that Skills can organize analytical conventions into reusable procedures and improve Codex’s performance on unseen tasks of the same type.

How Do Skills Improve Codex’s Data Analysis Capabilities?

1. Making Implicit Analytical Conventions Explicit

As discussed in our previous evaluation, many data analysis errors occur because the prompt does not fully specify the metrics, conventions, or background knowledge required by the task.

DABstep provides a typical example.

In its fee rules, aci: [] and merchant_category_code: [] do not represent empty sets in the conventional sense. Instead, they mean that the rule applies to all ACI or MCC values. Similarly, is_credit: null does not mean that the value is unknown. It means that the rule applies to both credit and debit transactions.

A Skill can make these implicit conventions explicit.

As a result, Codex no longer needs to guess the meaning of these fields while solving each task. Instead, it can follow the matching, fee-accumulation, and output rules defined in the Skill.

This improvement is particularly visible in the Highest Cost tasks. Without a Skill, Codex often selects only the single rule with the highest fee. After using the Skill, it is more likely to accumulate all applicable rules for each candidate ACI or MCC, as required by the task.

2. Clarifying the Actual Objective of the Task

Codex already has strong data processing and calculation capabilities. However, it may still misunderstand the actual objective of a task, causing its final result to deviate from the expected answer.

The DSBio Spatial Neighbor task is closer to a realistic scientific data analysis workflow. It requires constructing neighborhood relationships from spatial cell data and identifying which pair of cell types is most likely to occur near each other in space.

According to the task’s evaluation criteria, the final result must be a pair consisting of two different cell types. Without a Skill, however, Codex frequently selects same-type pairs such as macrophage_macrophage.

Its outputs without a Skill are shown below:

These results show that Codex successfully performed the neighborhood calculation but failed to recognize that the task only compares pairs of different cell types.

From the failed training trajectory, the Skill extracted several transferable rules:

• Use obsm["spatial"] as the source of spatial coordinates.
• Compare the number of observed neighborhood relationships with the number expected under a random baseline.
• Only compare pairs of different cell types, excluding same-type pairs such as macrophage_macrophage.

These rules clarify that the task is asking for the strongest spatial association between two different cell types.

The test set changes the neighborhood parameter and also introduces data from a new tissue region. These results suggest that the generated Skill extracted at least part of the transferable analytical procedure rather than merely reusing the final answer from the training task.

3. Turning Complex Business Workflows into Executable Procedures

Real-world business analysis usually consists of a complete workflow rather than a single calculation. If any intermediate step is misunderstood, the final answer may be incorrect.

For example, the solution procedure used in our experiments for the DABstep Card Scheme Routing task includes the following nine steps:

1. Convert the natural-language time range into the corresponding date range in the data tables.
2. Filter all transactions belonging to the specified merchant within that period.
3. Calculate monthly transaction volume and fraud rate from those transactions.
4. Retrieve the merchant’s account type, settlement delay, MCC, and other relevant attributes.
5. Reprice the same transactions under each candidate card scheme.
6. Match every transaction against all applicable fee rules and accumulate the resulting fees.
7. Aggregate the total cost for each candidate scheme.
8. Select the lowest-cost or highest-cost option, depending on the question.
9. Return the final answer in the required format.

Without a Skill, Codex may skip some of these steps.

For example, it may calculate the cost using only transactions that originally used a given card scheme, instead of repricing the same set of transactions under every candidate scheme. The resulting numbers may appear reasonable, but they do not represent the comparison requested by the task.

The Skill turns this workflow into a fixed procedure and provides a supporting script, solve_routing.py. Codex therefore does not need to reconstruct the date parsing, wildcard handling, rule matching, and fee-accumulation logic for every task. Instead, it can follow a stable, reusable process.

4. Converting Failure Experience into a Checklist

Another important role of a Skill is to convert errors observed in failed trajectories into constraints that must be checked during future executions.

Many Codex failures are not entirely random. Similar mistakes tend to recur at similar points in the workflow.

For example:

• In DABstep, Codex may interpret wildcard fields as empty sets.
• In DSBio Spatial Neighbor, it may return a same-type cell pair as the final answer.

Once these errors have been identified, they can be written into the Skill as explicit checks for future tasks.

A Skill can require, for example:

• Always check whether a fee-rule field represents a wildcard before deciding that the rule does not apply.
• In spatial neighborhood analysis, only compare pairs of different cell types.

In this way, the Skill provides a task-specific checklist that helps Codex detect recurring errors before finalizing the result.

For rule-heavy and multi-step data analysis tasks, such checks can help reduce recurring errors during execution.

What Are the Limitations of Skills?

1. Skills Can Reinforce Incorrect Interpretations

A Skill converts previous experience into reusable rules. However, if the training tasks provide insufficient coverage or our own interpretation of the task is incomplete, the resulting Skill may also be incomplete or incorrect.

In the DABstep Highest Cost tasks, the Skill produced a substantial improvement on the training set. However, several MCC-related questions in the test set still failed.

After examining the trajectories, we found that Codex became more consistent in calculating MCC fees after using the Skill. Unfortunately, this also caused several different questions to converge on the same incorrect answer.

The underlying problem was that the Skill had not accurately captured the correct procedure for this class of questions.

Although it correctly emphasized that applicable fee rules must be accumulated, it did not sufficiently explain how the candidate set should be defined for certain MCC questions or how costs should be compared in the general case. Codex therefore converged on a consistent but incorrect solution.

A Skill may therefore do more than preserve correct behavior: it may also cause the same incorrect behavior to recur more consistently.

This risk becomes more pronounced when the training set is too small or covers only a narrow range of cases.

2. Skills Apply to Related Tasks, Not Every Task

A Skill is normally useful only for a particular class of tasks.

It may help Codex solve questions involving the same dataset, similar fields, and comparable analytical procedures, but it cannot automatically generalize to every data analysis problem.

During our experiments, we found that Skills generated from the training set were effective on the test set only when the two sets followed similar task distributions.

For example, the DSBio Spatial Neighbor Skill learned a spatial cell-neighborhood analysis procedure from the training task. It was therefore effective on similar spatial neighborhood tasks in the test set. However, it would not necessarily help with other types of cell analysis tasks in DSBio.

At the same time, placing too many substantially different tasks into one Skill may introduce multiple unrelated or even conflicting sets of rules. This can dilute the Skill’s focus and make it less effective for any specific task.

A more practical approach is to divide Skills by task type or analytical workflow and select the most relevant Skill at execution time.

3. Skills Cannot Replace Human Validation

Skills can reduce repeated errors, but they cannot guarantee that the final answer is correct.

Human validation is still required when:

• the question itself is ambiguous;
• field definitions are unclear;
• the training tasks provide insufficient coverage;
• the test task differs substantially from the training tasks.

This is especially important in scientific analysis and business decision-making, where final conclusions should not rely exclusively on a Skill.

A more appropriate use of Skills is to help Codex follow common rules and standardized procedures while reducing repeated mistakes. Critical metrics and final conclusions should still be reviewed by humans.

How Can We Use Skills More Effectively for Data Analysis?

1. Make Implicit Conventions Explicit

Many data analysis errors arise from conventions that are not stated in the prompt but must still be followed during the calculation.

A good Skill should first make these conventions explicit.

For example, in payment analysis, it should state that:

• aci: [] and merchant_category_code: [] mean that the rule applies to all values, rather than to no values.
• is_credit: null matches both credit and debit transactions.
• Multiple applicable fee rules must be accumulated instead of selecting only the highest- or lowest-cost rule.

The same principle applies to other tasks.

The more precisely a Skill describes its implicit assumptions and rules, the less Codex needs to guess and the more likely it is to complete the task successfully.

2. Structure the Skill as a Fixed Workflow

We recommend organizing a practical data analysis Skill into three sections:

Many Skills fail not because they contain no useful rules, but because field definitions, procedures, and output requirements are mixed together.

In such cases, Codex may understand the general direction but still omit important steps during execution.

A clearly structured workflow makes it easier for Codex to complete the task in a consistent order.

3. Convert Common Errors into Concrete Rules

A Skill should describe not only the correct procedure but also the mistakes that commonly occur.

However, generic reminders such as “check data quality” or “verify the result carefully” are usually of limited value. Codex cannot easily infer what specifically needs to be checked.

A more effective approach is to express each failure pattern as an explicit and executable rule.

For example:

• Do not interpret aci: [] or merchant_category_code: [] as empty sets; these fields mean that all values are applicable.
• Do not use statistics calculated over the entire year when the task requires monthly transaction volume or monthly fraud rate.

These instructions are more precise than general warnings and are easier for the agent to follow.

The more concrete the failure checklist is, the more effectively Codex can avoid previously observed mistakes when solving similar tasks.

4. Pair Complex Skills with Supporting Scripts

For some tasks, a Markdown-based Skill alone is insufficient.

When a task involves multi-table joins, interval matching, wildcard rules, transaction-level repricing, or complex matrix calculations, it is often better to include a reusable supporting script and explain how it should be used.

For example, the solve_routing.py script included in the DABstep Routing Skill standardizes:

• date-range parsing;
• fee-rule matching;
• settlement-delay matching;
• transaction-level fee calculation.

This prevents Codex from rewriting the same complex logic for every task.

The same approach is useful for scientific computing. The Skill can define the formulas, parameters, fields, and units, while the script executes the calculation consistently.

This combination reduces omitted calculations, incorrect implementations, and inconsistent output formatting.

Conclusion

In this small-scale, single-run experiment, Skills reduced several recurring errors and improved Codex’s accuracy on the selected unseen tasks.

In realistic data analysis scenarios, we believe the ideal division of responsibilities is:

• Humans define the problem, confirm the meaning of the data, and validate the conclusions.
• Skills preserve domain rules, reusable procedures, and known failure patterns.
• Codex reads the data, performs the calculations, produces intermediate results, and organizes the final report.

Under this collaboration model, Skills can make Codex behave more like a reliable data analysis assistant.

They cannot guarantee correctness on every task, nor can they replace human validation. However, for data analysis tasks with explicit rules, repeated workflows, and frequent errors caused by field definitions or background conventions, Skills provide a practical way to improve Codex’s data analysis performance.

DataCOPE is now available as an open-source component of the DataMind project. If you are also working on data analysis agents, Skill augmentation, automated analytical workflows, or reusable Skill generation for your own tasks, we invite you to explore our GitHub repository and read our paper:

👉 GitHub: https://github.com/zjunlp/DataMind/tree/main/datacope/general

👉 Paper: https://arxiv.org/abs/2606.06416

We welcome users to try DataCOPE, share feedback, and contribute suggestions. If you find the project useful, please consider supporting us with a GitHub Star.

If you find this blog useful, please consider citing it as follows:

@misc{qiu2026skills_improve_codex_data_analysis,
      title={{Can Skills Improve Codex’s Data Analysis Capabilities?}},
      author={Zhisong Qiu and Chen Yan and Jiafu Wei},
      year={2026},
      publisher={Zenodo},
      doi={10.5281/zenodo.21292469},
      url={https://doi.org/10.5281/zenodo.21292469},
      note={Technical blog}
}

@misc{qiu2026unsupervisedskilldiscoveryagentic,
      title={Unsupervised Skill Discovery for Agentic Data Analysis}, 
      author={Zhisong Qiu and Kangqi Song and Shengwei Tang and Shuofei Qiao and Lei Liang and Huajun Chen and Shumin Deng},
      year={2026},
      eprint={2606.06416},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.06416}, 
}