Abhisek Behera
AI & ML interests
Recent Activity
Organizations
I verified your numbers against the raw data and they reproduce exactly, including the per-pair smoke counts and the celery+kombu cell.
On your question: I think 263 rows is enough to report, but not enough to conclude. It supports an aggregate smoke number with honest error bars and a per-pair count table, but not per-pair claims, since 4 of the 8 pairs carry fewer than 25 rows each. So smoke should stay a small-n secondary axis, not a headline.
The celery+kombu finding is a genuine bright spot: both failure modes inside one near-balanced cell means the sealed split can at least ask the smoke question, even if the answer comes with wide intervals.
The bigger gap this points at: the sealed split has only 79 post-resolve failures in total, all from one family. That is what I will try to improve in a future release: more post-resolve failure cases from more than one unseen family, with the lookup ceiling computed privately before anything is published as a test.
Thanks for pushing on this axis. The dataset's evaluation story is much sharper because of this thread.
Thanks, I like both of your ideas.
The two-layer label idea is good. One field for the guessed cause (from error patterns), and a separate field for the confirmed cause (checked by a human). This way, nobody confuses a guess with a verified fact. I will use this structure.
The visibility rule is also a good idea. The record itself should say what the agent is allowed to see: the goal, the Python version, the resolver output, the package list, and the wheel info. And what it must not see: anything from the smoke test. With this rule written down, the test is fair and nobody can argue about cheating.
About the competition: it is an interesting idea. One thing worth mentioning: I designed a final test set with 9 new packages, but I have not run it yet. So the results do not exist anywhere. No model or bot could have seen them. That kind of unseen data is what a fair challenge would need.
The task you described makes sense: predict pass or fail, guess the cause, suggest a fix, and score against the real result.
Let me think about all this. Thanks again for the detailed suggestions.
This is a very useful list, thank you. Let me sort it into what the dataset already has and what it doesn't.
Already in v1.0.0:
- Tool and environment details: exact uv version and exact CPython build are recorded per experiment (in
runtime). One caveat: the environment is fixed by design, everything ran on Linux x86_64, so the OS fields never vary. They confirm the setup rather than describe differences. Expanding to more OS types (Windows, macOS) is something I may explore in the future, and the schema already has the fields ready for it. - The exact input spec: both packages, versions, Python version (in
spec). - The full resolved package graph, not just the two headline packages (in
installed_environment). - Every stage's exact command, exit code, stdout/stderr, plus a timeout flag and peak memory (in
stages). - Retry count, normalized error text next to the raw output, and full wheel provenance: ABI/platform tags, sha256, source URL.
Not covered today, but I will try to include these in a future v2 release:
- The smoke-test code snippet itself (only its output is captured)
- Explicit timeout and resource limit values
- Confirming a failure reproduces across retries
- A "likely cause" label: ABI mismatch, removed symbol, missing optional dependency, build backend, platform wheel
Your likely-cause idea is the most interesting one to me. Since all raw tracebacks are preserved, labels like that could in principle be derived from the existing records without rerunning anything.
The benchmark framing also makes sense: an agent proposes a requirements change and gets scored against known real outcomes before anything is installed. That's much closer to real developer work than a synthetic task. Thanks for laying it out so clearly.
Well said. That's exactly why every record here keeps the full environment state: the resolved package set, installed wheels with ABI/platform tags, lockfile hash, and per-stage stdout/stderr.
One use I'd love to see: as an eval or reward signal for coding agents. An agent proposes a requirements change, and you score it against known real outcomes without installing anything. The 3,049 resolved-but-broken cases are the interesting ones, since they look fine to any metadata-only check.
Our own DepLab agent already works this way: models trained on these outcomes, with resolver facts kept separate from ML risk. Happy to hear what fields would make the dataset more useful for agent workflows.
Thank you for doing this properly. You are right on every point, and I appreciate that you verified the tallies reproduce before critiquing the framing.
I agree the full-set accuracy mixes two tasks. Predicting resolution failures that uv already reports is the easy part, and the 90.0% headline benefits from the validation split being 79.1% resolution failures. The number that matters for the dataset's core claim is the resolved-only subset, and I should have reported it separately from the start. I will update the dataset card to show resolved-only metrics beside full-set metrics.
You are also right that the sealed split is 4 pairs, not 6 packages crossed, and that all 79 post-resolve failures come from the celery/kombu/billiard family. So the honest statement is: the post-resolve signal was validated on one unseen family, not on unseen packages broadly. I will correct that wording too.
Your question, which development pairs produce the most import failures: pandas+seaborn is the largest (354 import failures plus 92 smoke-test failures out of 588 resolved rows, 75.9% post-resolve failure rate), then matplotlib+seaborn (264 import failures, 51.0%), flask+jinja2 (238, 46.1%), and numpy+opencv-python (196, where every resolved row fails import). Overall the development resolved-only subset is 15,080 rows: 12,031 pass, 2,448 import failures, 601 smoke-test failures, so always-pass scores 79.8% there.
Two pairs are degenerate in the other direction: fastapi+starlette (all 338 resolved rows fail the smoke test) and numpy+opencv-python (all 196 resolved rows fail import). A constant predictor is unbeatable on those, so they would make poor holdouts too. The most informative holdout candidates are the mixed pairs where neither constant baseline wins: matplotlib+seaborn (300 pass vs 312 post-resolve failures, almost exactly balanced), flask+jinja2 (291 vs 249), and pandas+seaborn (142 vs 446).
For the next release: (1) resolved-only accuracy reported beside full-set accuracy, (2) a re-cut sealed split holding out at least one mixed family, likely matplotlib+seaborn, and (3) the reserved 9-package final test set run with that design. If you see a better holdout choice in these numbers, I am genuinely interested.
Every experiment installed a pair of Python packages into a fresh environment and tested three stages: resolution → import → runtime smoke test.
The finding that motivated this: resolvers only read published metadata. In this data, ~3,000 environments resolved successfully and still failed at import or smoke-test time. That gap is invisible to pip and uv, and this dataset makes it measurable.
What's inside:
- 21,490 development + 3,432 sealed validation experiments
- 41 unique packages, Python 3.8–3.14, Linux x86_64
- The 6 validation packages (boto3, botocore, s3transfer, celery, kombu, billiard) never appear in training — validation measures generalization to entirely unseen packages
- Full per-stage logs: exact commands, exit codes, stdout/stderr
- Normalized error text + Python exception types for every failure
- Wheel metadata with Python/ABI/platform tag compatibility decisions
- PEP 751 lockfile hashes
- 142 engineered features per experiment, ready for training
Outcome labels: pass, resolution_failure, import_failure, smoke_test_failure.
I used it to train two models for DepLab (a structured model + a fine-tuned ModernBERT on release notes). On the sealed validation set (entirely unseen packages) the structured model reached 90.0% accuracy and 100% precision when predicting failures.
CC BY 4.0. Load it directly:
from datasets import load_dataset
ds = load_dataset("Abhisek987/deplab-dependency-compatibility", "experiments")Dataset: Abhisek987/deplab-dependency-compatibility
Zenodo DOI: https://doi.org/10.5281/zenodo.21729353
Code: https://github.com/Abhisek12378/DepLab
If you're working on ML for developer tools or software engineering research, I'd love to hear what you build with it.
The idea I wanted to test: in a projective-dynamics solver, could you replace each hand-derived local constraint projection with a learned one, while keeping the analytic parts (rotations, the global solve) exactly as they are? It's one tiny network, shared across every element and across constraint types through material tokens. A new material isn't a new network, just a new token row. Fluids fall out of the same idea, with water treated as one more token.
A few things held up in testing: one tied projector matched five separate per-material solutions, the neural fluid tracked the exact analytic solver closely on a dam-break sim, and a learned warm-start trimmed solver iterations without touching correctness.
Try it here:
(Projective-dynamics)
Dam Break Demo:
https://quazim0t0-neural-physics-engine-demo.hf.space/
FPS Shooter Demo:
https://quazim0t0-neural-combat-evolved.hf.space/
(Raytracing)
Voxel World:
https://quazim0t0-neural-world.static.hf.space/index.html
(Rigid Body)
https://quazim0t0-ashendepths.static.hf.space/
Model Repo:
Quazim0t0/neural-physics-engine
Quazim0t0/neural-raytracing
Quazim0t0/physgait-weights
Thanks John, this is a great workflow, especially the part about having the model summarize what did and did not work, then feeding those summaries back in as context. That "what broke and what fixed it" record is basically the signal my dataset tries to capture, just at the dependency-upgrade level. Drift really is the quiet killer. Appreciate you sharing the prompt link too, I'll take a look.
Nice!
Drift is always, always really troublesome. If there were no drift, Python would be heaven. Well, it does have weaknesses like speed, GIL constraints, and indentation, but...
It’s not that I’m seriously working on it as research; it’s just a bit of everyday wisdom, but when I debug Spaces, if they are meant to be open/public, I first feed the code into ChatGPT. Then I deliberately debug it within the chat, whether by vibe coding or not, and at the end I have it summarize what didn’t work and what made it work.
Then, if I feed the generated file into ChatGPT and use it as-is, or feed several of them in together, add context, and have it summarize them into another Markdown document before attaching it, it tends to refer to them fairly well.😅
Nice!
Drift is always, always really troublesome. If there were no drift, Python would be heaven. Well, it does have weaknesses like speed, GIL constraints, and indentation, but...
It’s not that I’m seriously working on it as research; it’s just a bit of everyday wisdom, but when I debug Spaces, if they are meant to be open/public, I first feed the code into ChatGPT. Then I deliberately debug it within the chat, whether by vibe coding or not, and at the end I have it summarize what didn’t work and what made it work.
Then, if I feed the generated file into ChatGPT and use it as-is, or feed several of them in together, add context, and have it summarize them into another Markdown document before attaching it, it tends to refer to them fairly well.😅
Good question. It is for training and evaluating models that fix Python code after a dependency upgrade breaks it. If you have ever bumped numpy or pandas and watched old code stop working, this is the data to teach a model to make those fixes automatically. Each example pairs the broken code with the fix and a short note on the API change that caused it. There is a live demo in the collection if you want to see it fix real code.
I built an open dataset for exactly this problem. DepDoctor is 6,204 examples of Python code broken by a dependency upgrade, each paired with the fix and a short note on the API change that caused it. It is a mixture of real cases mined from public GitHub commits and synthetic cases generated from a database of known breaking changes.
A few things I tried to get right:
- 935 "leave it alone" examples, to teach a model restraint, not just what to change.
- Honest evaluation: a fine-tuned Qwen2.5-Coder-7B gets 62% of fixes fully correct. I report that, not just the 97% text-similarity score that hides the truth.
- The main failure mode, over-editing, is measured and explained rather than buried.
Dataset, fine-tuned model, and a live demo are all open in one place:
https://huggingface.co/collections/Abhisek987/depdoctor
Feedback welcome, especially from anyone working on code repair or API migration.