khursheed/datacard-ci

DataCard CI

Turn a claim about your dataset into a check you can review and run.

Status: experimental model v0.2, hardened runtime v0.2.1. Intended for supervised trials; not production-certified. The latest update adds CPU-only verification, strict input/output validation and reproducible model pins. It does not claim new model accuracy.

Dataset documentation can fall behind the data. A README says IDs are unique, a split has a fixed number of rows, or training and test sets do not overlap. After the next update, is that still true?

I built DataCard CI to explore a small part of that problem. It is a fine-tuned Qwen3-0.6B LoRA adapter that reads one quoted claim and the dataset's split and column names, then proposes a structured check. You review the interpretation before a separate Python verifier checks the CSV files.

The goal is to make a few routine data checks easier to set up, while keeping the result traceable to the original claim and the files actually checked.

Where it might help

• Dataset updates: compare an exact documented row count with the latest file.
• Duplicate IDs: check whether an ID column is unique within a split.
• Train/test overlap: check whether the same ID appears in both splits. This catches ID overlap, not every form of data leakage.
• Claims that need more evidence: separate computable statements from consent, rights or provenance claims that need human evidence, and unclear claims that need rewriting.

For example, given “All sample_id values in fit are unique” and a schema containing that column, the intended proposal is:

{"kind":"computable","check":{"op":"unique","split":"fit","column":"sample_id"}}

This is a proposal, not a finding that the data passed. The prediction helper validates its structure and returns an unapproved claim. Once a person reviews it, the verifier can report PASS, FAIL or ERROR for the supplied files, or ABSTAIN when the claim cannot be checked.

Try the existing model

You do not need to retrain it. There is no hosted demo or API supplied by this repository; inference runs on the machine you choose. The updated helper defaults to CPU. GPU execution requires device="cuda" (CLI: --device cuda). CPU can be slower and still needs enough RAM for the base model. This model repository is public; no author token is needed to download it. Keep tokens in your environment or secret store, never in a shared notebook.

In a fresh Python environment with PyTorch installed, install the packages in requirements-colab.txt. The versions are the ones used for this experiment. Colab already supplies PyTorch; its optional preinstalled torchao package may need removing as described in the notebook.

Download this pinned release and try a single claim:

import sys
from huggingface_hub import snapshot_download

release = snapshot_download(
    "khursheed/datacard-ci",
    revision="ebba05fac41cb7f1cd2fb82cec4211922292951c",
    allow_patterns=["*.py", "adapter_config.json", "adapter_model.safetensors"],
)
sys.path.insert(0, release)
from predict_v2 import predict

proposal = predict(
    "All sample_id values in fit are unique.",
    {"fit": ["sample_id", "value", "category"],
     "holdout": ["sample_id", "value", "category"]},
    adapter=release,
    device="cpu",
)
print(proposal)  # Check the interpretation; approval remains false.

The helper also downloads the pinned Qwen base model. After caching both models, use local_files_only=True (CLI: --offline) to prevent Hub downloads. The updated loader has CPU mock tests; a fresh full-weight run and a memory/latency benchmark on the intended machine are still required before deployment. Malformed or unsupported output raises an error; it should not be silently accepted. Review the quote, operation and parameters before using datacard_ci.py to verify your local CSV bytes. Approval is a local flag, not authenticated sign-off.

The verifier currently accepts up to 10 CSV splits, each at most 2,000,000 bytes, 20,000 rows and 100 columns. It also limits a batch to 100 claims and the documentation/claims JSON to 64,000 bytes each. It compares IDs as exact strings and treats blank IDs as errors. A passing result applies only to the named files and check.

Start without loading an LLM

If you already know the check you need, write and review it directly. The deterministic verifier works without model weights, a GPU, an API token or third-party Python packages.

From a complete source checkout, run the included, hand-reviewed toy example:

python3 verify_cli.py --card examples/reviewed/card.txt --claims examples/reviewed/claims.json --csv fit=examples/reviewed/fit.csv
python3 -m unittest -q

The command prints a JSON evidence report. Exit 0 means every reviewed check passed; 1 means a failure, abstention or pending review; 2 means input/file validation failed. An empty claim list cannot pass. Do not copy the toy example's approval onto a generated claim. Reports include your documentation text; keep confidential reports local.

What changed without more training

• Strict JSON parsing rejects duplicate keys, non-finite numbers, excessive nesting and unexpected fields.
• Quote, schema, file and batch limits reject oversized work. Prompt length is checked before loading model weights.
• Remote adapter revisions are pinned; CPU is the default and offline operation is available once files are cached.
• The offline verifier command treats unreviewed and inconclusive results as non-success.
• 35 CPU tests pass. The installed command and toy fixture were exercised. Model-loading tests use mocks, not a fresh full-weight inference run.

Replaying the saved 100 test outputs reproduces 86 exact answers and 7 unsupported proposals. The runtime accepts 98 structurally valid, grounded proposals; that is not 98 correct answers. Schema validation cannot catch all plausible misinterpretations. See the replay report and readiness notes.

Next: prove usefulness before buying compute

The next useful input is a real workflow, not a bigger training run. Share a non-sensitive claim or recurring dataset problem in Hugging Face Discussions. The example intake guide explains what to include.

We need permission-cleared, independently reviewed examples from real documentation, held out by source dataset. Compare the current model with manual checks and a simple rules baseline, then run it alongside the existing process with human review. Only retrain if those failures justify a concrete data change. Before deployment, validate full-weight inference, memory, latency and failure handling on the actual target machine. READINESS.md records the remaining gates and compute limits.

What I have measured so far

v0.2 matched the expected structured answer on 86 of 100 synthetic test examples, compared with 60 for v0.1. It produced 99 schema-valid outputs and correctly abstained on 35 of 40 examples that needed abstention.

There were still 14 wrong answers, including estimated counts interpreted as exact counts and mistakes around uniqueness and header wording. Seven outputs met our definition of an unsupported proposal. These results come from a small project-authored synthetic test, not an independent benchmark or evidence of production reliability.

The training dataset contains 1,200 training, 100 validation and 100 test examples for v2, plus the earlier v1 corpus. The examples are AI-authored and programmatically validated, not independently human-annotated. The test has now been inspected; further tuning needs a new blind test.

What I want to learn from you

If keeping dataset documentation accurate is a pain point, I would love to hear what actually breaks in your workflow. What do you check manually today? Which mistake would you want caught before you train a model or publish a dataset?

Try a claim, inspect the proposal, and share what worked or failed in Discussions. A useful report includes a non-sensitive or invented claim, split and column names, the expected check, and the output you received. Please leave out credentials, personal records and confidential data.

I am especially interested in whether this saves any work compared with writing the check directly, and which missing checks would make it useful. Feature ideas are welcome; this is an experiment, with no promise of support or a hosted service.

Boundaries and compute

DataCard CI does not yet read a whole dataset card or support missing-value, range or category checks. The new verifier command can be called from your existing CI pipeline; automatic claim extraction and a managed CI integration are not included. It cannot establish consent, fairness or provenance. Its verifier runs a fixed set of operations, never model-generated Python or SQL.

The training schemas all use two splits and three columns with the ID first. Generalization to other layouts and real documentation remains unproven. Human review is required even when the output is valid JSON.

The GPU training notebook is for reproducing the experiment. Training is off by default and the notebook does not publish anything. If you enable training, it consumes your chosen runtime's compute. The script's time checks do not shut down a cloud machine or guarantee a billing cap; stop paid runtimes when finished. Trying the model on your own hardware does not use the author's API credentials.

Experiment record

The details below make the result inspectable and reproducible. The release archives and their manifests preserve the original experiment snapshot; this README has since been rewritten for clarity. Model weights and evaluation results have not changed. Runtime v0.2.1 files have a separate runtime-manifest.json; the original release-manifest.json belongs to the archived training snapshot.

Training settings and complete evaluation

Training

• Base Qwen/Qwen3-0.6B, pinned revision c1899de289a04d12100db370d81485cdf75e47ca.
• Hardware: Tesla T4 GPU.
• 1,200 original AI-authored synthetic examples, 60 training sentence templates, 20 identifier/domain families; MIT licensed. Programmatically validated, not independently human-annotated. No scraped card prose or private records.
• Fresh LoRA from the same base as v0.1: rank 8, alpha 16, dropout 0.05, q_proj/v_proj.
• Batch 1, accumulation 4; 1 epoch(s) / 300 optimizer steps completed. Development-selected checkpoint: epoch 1, step 300.
• Peak learning rate 0.0002, 20-step warmup and linear decay with 10% floor; float32 weights with float16 autocast and gradient scaling; seed 42.
• Completion-only loss, no truncation, maximum observed sequence 311 tokens.
• Run time 1149.9 seconds including comparisons and reload evaluation.
• Serialized weights and reloaded generations verified.

Evaluation

100 development examples evaluate the checkpoint (and select between epochs if two are requested); a separate frozen 100-example test is evaluated afterward. This release used 1 epoch(s). Template and domain families are disjoint. All three models use the same v0.2 prompt and test inputs. These are synthetic examples authored in the same project, not an independent real-world benchmark. The v0.1 40-example evaluation is now a diagnostic set and was not reused here.

Metric	Base	v0.1	v0.2
Raw JSON	2/100	100/100	100/100
Valid check schema	2/100	96/100	99/100
Valid schema references	2/100	90/100	98/100
Exact target (strict JSON)	0/100	60/100	86/100
Exact target (single fence allowed)	25/100	60/100	86/100
Correct classification	42/100	85/100	91/100
Correct abstention	0/40	25/40	35/40
Unsupported proposals (lower is better)	25/100	20/100	7/100

Strict scores require raw JSON. Semantic scores allow one enclosing Markdown fence only. Classification is scored separately from check validity. Symmetric split-disjoint arguments are treated as equivalent. Unsupported proposals include invented schema fields/operators and executable checks for expected abstentions; this count is not a complete safety assessment. All raw generations are included.

v0.2 class	Exact
row_count	16/20
unique	16/20
split_disjoint	19/20
attestation	20/20
ambiguous	15/20

Promotion gate passed: True. The gate requires better exact accuracy than v0.1 with no regression in unsupported proposals or correct abstention. It does not certify production readiness.

Real-data integration fixture

real-data-evidence/evidence.json records passing, deliberately failing, and missing-ID tests using the UCI Wine Quality red table. IDs and splits were added for the fixture and are not original entity identifiers. This tests the verifier, not the model's generalization. The table is not used for model training. Cortez et al. (2009), Wine Quality, UCI, https://doi.org/10.24432/C56S3T, CC BY 4.0. See the included attribution and transformation record.

Reviewed failures

The 14 final-test errors comprise five estimated counts incorrectly treated as exact, four row-count claims with header-exclusion wording misinterpreted, four uniqueness errors, and one split-disjointness claim incorrectly abstained on. Seven outputs met the unsupported-proposal definition. Human review remains required; the synthetic test does not establish real-card generalization.

License and limitations

Original adapter, code and synthetic examples: MIT. Base weights: Qwen Apache-2.0. UCI-derived fixture material: CC BY 4.0, attributed separately. No base weights are redistributed. No CardBench prose was incorporated; source-specific reuse permissions and claim annotation are still needed for future real-card training.

The model can misinterpret claims or abstain unnecessarily. Scores cover only three operators and five label categories. All generated schemas use two splits and three columns with an ID field first; different layouts remain untested. Review evaluation.json, DATASET_V2.md and per-example errors before use. The original v0.1 remains in repository history and its release archive.