qwen2.5-coder-1.5b-ml-engineer

A LoRA adapter for Qwen/Qwen2.5-Coder-1.5B-Instruct that writes machine-learning engineering Python that runs. It was trained on 6,979 rows drawn entirely from a filtered third-party corpus, with zero data authored by this project, for 250 optimizer steps.

It is a research checkpoint. It earned publication under a rule fixed before any number existed, and the caveats below are part of that publication rather than a footnote to it. Read them.

The headline, with its interval

Measured on a 133-task execution suite: a task passes only if the generated code imports, runs in a sandboxed subprocess, and its assertions about values and shapes hold. Vocabulary earns nothing. Greedy, unbatched, 512 max new tokens, under the frozen system prompt quoted below.

pass@1 0.4436 (59/133) against the base model's 0.3158 (42/133).

Δ +0.1278, paired bootstrap 95% CI [+0.0376, +0.2180], McNemar exact p = 0.0076.

Never quote the point estimate without the interval. The lower bound is +0.0376: an effect roughly a third the size of the headline is fully consistent with this measurement. The suite's realized minimum detectable effect at the observed discordance is 0.125, so this instrument cannot resolve anything much smaller than what it just measured.

McNemar detail: 27 tasks this adapter passes and base fails, 10 the reverse, 37 discordant of 133. Only discordant tasks carry information in a paired comparison.

The rule that admitted it, written to disk before any generation

A checkpoint is PUBLISHABLE only if, on the 133-task union under frozen_007: its execution_pass_at_1 exceeds base's, AND the paired per-task bootstrap 95% CI for (candidate - base) excludes zero, AND McNemar's exact test on the discordant pairs gives p < 0.05.

Nine readings were taken and the rule was applied once, to all of them. Exactly two returned PUBLISHABLE, and both are this arm. Every other candidate — including the adapter this project previously published — is at or below base.

reading	pass@1 (n=133)	Δ vs base	95% CI	McNemar p	verdict
013_D_step_250	0.4436 (59)	+0.1278	[+0.0376, +0.2180]	0.0076	PUBLISHABLE
013_D_step_50	0.4436 (59)	+0.1278	[+0.0451, +0.2105]	0.0046	PUBLISHABLE
base (reference)	0.3158 (42)	—	—	—	reference
013_A_prime_step_250	0.2707 (36)	-0.0451	[-0.1278, +0.0376]	0.3771	NOT_PUBLISHABLE
013_C_step_250	0.2707 (36)	-0.0451	[-0.1278, +0.0376]	0.3616	NOT_PUBLISHABLE
008g_step_375	0.2030 (27)	-0.1128	[-0.1955, -0.0301]	0.0135	NOT_PUBLISHABLE
013_A_step_250	0.1805 (24)	-0.1353	[-0.2105, -0.0602]	0.0014	NOT_PUBLISHABLE
008e_step_250	0.1579 (21)	-0.1579	[-0.2481, -0.0752]	0.0008	NOT_PUBLISHABLE

Frozen-44 and expansion-89 subsets are reported separately because every historical number in this project is a frozen-44 figure. This adapter scores 0.3182 on the frozen 44 and 0.5056 on the 89-task expansion, against base's 0.2045 and 0.3708.

Why step 250, and why that is not a ranking

Step 250 was 013's pre-registered primary step, fixed before any number existed. It is published for that reason and no other. The step-50 reading has the identical point estimate (0.4436), so nothing here was chosen for scoring better. Best-of-N selection is banned by 014's pre-registration: with eight readings, cherry-picking the best manufactures roughly +0.05 of pure noise, about half this suite's entire detectable effect. Read the table above as a report, not a leaderboard.

Base vs this adapter, in detail

Five figures, generated at publish time from the same two artifacts as every number above. Each figure is followed by the numbers it draws, so nothing in this section depends on an image loading or on a reader being able to see one.

The headline, drawn

Base passes 42 of 133 tasks (0.3158); this adapter passes 59 of 133 (0.4436). The difference is +0.1278, paired bootstrap 95% CI [+0.0376, +0.2180]. The whisker sits on the adapter's bar because it is the interval of the difference and not of the adapter's own rate: a true effect that would put this adapter at 0.3534 is fully consistent with what was measured.

Every category improves

All seven categories improve and none regresses. Sorted by gain, with the task counts behind each rate:

category	n	base pass@1	this adapter	Δ
general_python	19	0.3158 (6/19)	0.5263 (10/19)	+0.2105
modeling	19	0.3158 (6/19)	0.5263 (10/19)	+0.2105
eda	19	0.2632 (5/19)	0.4211 (8/19)	+0.1579
time_series	19	0.1579 (3/19)	0.3158 (6/19)	+0.1579
data_engineering	19	0.2105 (4/19)	0.2632 (5/19)	+0.0526
deep_learning	19	0.6842 (13/19)	0.7368 (14/19)	+0.0526
feature_engineering	19	0.2632 (5/19)	0.3158 (6/19)	+0.0526
all seven	133	0.3158 (42/133)	0.4436 (59/133)	+0.1278

Resampling the same paired per-task differences stratified within category — each category's tasks resampled inside that category — gives the same +0.1278 with a 95% CI of [+0.0451, +0.2105] (SE 0.0445), which also excludes zero. The gain is not carried by one category.

What got worse. Of the 133 tasks, ten are ones base passes and this adapter fails. They are named rather than netted out: dl_linear_backward, eda_top_correlated_pairs, md_beats_majority_baseline, dex_numeric_column_names, dlx_accuracy_from_logits, fex_minmax_scale_columns, fex_interaction_terms, mdx_confusion_counts, mdx_probability_of_label, tsx_split_at_timestamp. A comparison that reports only what improved is not a comparison.

Every scored candidate, against base

Δ against base with its 95% CI, best first: 013_D_step_250 +0.1278 [+0.0376, +0.2180]; 013_D_step_50 +0.1278 [+0.0451, +0.2105]; 013_A_prime_step_250 -0.0451 [-0.1278, +0.0376]; 013_C_step_250 -0.0451 [-0.1278, +0.0376]; 008g_step_375 -0.1128 [-0.1955, -0.0301]; 013_A_step_250 -0.1353 [-0.2105, -0.0602]; 008e_step_250 -0.1579 [-0.2481, -0.0752]. Only two of the seven clear zero, and both are this arm. 008e_step_250 is the project's previous release (Hak5/qwen2.5-coder-1.5b-ml-engineer-lora) and it sits furthest below base, at -0.1579. In the figure colour carries the verdict and nothing else; every row is named in text.

The dose-response, with the seed spread left in

Owned-gold share against union pass@1: 0.0000 → 0.4436 (arm D, seed 42); 0.0921 → 0.2707 (arm C, seed 42); 0.2252 → 0.1805 (arm A, seed 42); 0.2252 → 0.2707 (arm A', seed 43); base 0.3158 with no training at all. The two points at the same share are the same mixture at two seeds, drawn as two separate marks 0.090 apart rather than averaged, because that spread is most of the headline effect and a chart that hid it would make this result look sturdier than it is.

Only the discordant pairs carry information

27 tasks this adapter passes and base fails; 10 tasks the reverse; 96 tasks concordant, where both pass or both fail. Only the 37 discordant pairs carry information in a paired comparison, and they are what the interval and the McNemar test are computed on. It is also why this instrument cannot resolve much below 0.125.

Read this before you use it

1. Single seed. The seed-to-seed spread is most of the claimed effect.

This adapter was trained once, at seed 42. In the same run, arms 013_A_step_250 and 013_A_prime_step_250 are the same configuration differing only by seed and scored 0.1805 and 0.2707 — 0.090 apart on this very metric, against a headline effect of +0.1278.

The bootstrap interval above resamples TASKS, not seeds, so it does not contain that variance and cannot. A second seed has not been run for this configuration. This is the single most important caveat on this release: the honest reading is that a zero-owned-gold mixture produced a large measured gain once, at one seed, on one suite.

2. This model is NOT better on term coverage.

On the project's own dev-mirror scorer — 0.55 * required-term coverage + 0.20 * safety + 0.25 * AST parse — this checkpoint moved the aggregate by +0.0079 (0.5711 → 0.5789), which is base to within noise, while feature_engineering fell -0.123. It trades vocabulary coverage for executability. Anyone expecting a uniformly better model should not use it.

3. The evaluation suite is ours, not a public benchmark.

133 tasks, 89 of them authored after these adapters existed, with a realized minimum detectable effect of 0.125.

Measured decontamination of those tasks against the training data: max prompt 5-gram Jaccard 0.0093 against a published null max of 0.0217 (the frozen prompts scored pairwise against each other), and entry-point name overlap of 1 of 89 (round_half_up, weighted 0.0 in scoring). Nothing was rejected by the decontamination rules.

That is a measurement of hygiene, not of generality. These numbers are not comparable to HumanEval, MBPP or DS-1000 and must not be quoted beside figures from them. The suite is also deliberately adversarial in places — roughly a quarter of the tasks are written so that the plausible, keyword-shaped answer fails — so the absolute pass@1 is not a general capability figure for anything.

4. The frozen release suite has never been read.

The project keeps a 35-task release suite that may be read once. That read has never happened: run 011 attempted it and was refused on its own preconditions, so its one permitted read remains unspent. There is therefore no held-out reading of this checkpoint at all. Every number here comes from suites that existed and were visible while decisions were being made.

5. A decode-length inconsistency exists in this project's history.

Run 012 generated at 768 max new tokens; runs 013 and 014 generated at 512 max new tokens. All numbers on this card are internally consistent at 512 and every comparison above is paired inside a single session, but older published figures from this project are not directly comparable to them.

6. The adapter this project published earlier is measurably worse than base.

Hak5/qwen2.5-coder-1.5b-ml-engineer-lora is the same project's earlier release. On this suite it scores 0.1579, a Δ of -0.1579 against base, with a 95% CI of [-0.2481, -0.0752] that excludes zero and McNemar exact p = 0.0008. It is measurably worse than the base model at writing code that runs. Readers deserve to know that before treating either release as an endorsement of the other.

7. This is a short run.

250 optimizer steps ≈ 0.5732 epochs over 6,979 rows. Not a long run, and not a converged one. Whether the effect survives more training is unmeasured.

8. Everything else you would expect.

It inherits every limitation and bias of the base model, cannot see images, cannot run code, and is a research checkpoint rather than a production component. Results degrade without the system prompt it was trained and evaluated under.

The uncomfortable finding: more owned synthetic gold, worse executable code

013 varied one thing — the share of this project's own generated, execution-verified "gold" conversations in the training mixture — and 014 scored the arms on the union.

owned-gold share	reading	pass@1 (n=133)
0.0000	013_D_step_250	0.4436
0.0921	013_C_step_250	0.2707
0.2252	013_A_step_250 (seed 42)	0.1805
0.2252	013_A_prime_step_250 (seed 43)	0.2707
—	base (no training)	0.3158

The arm with no owned gold at all is the only one above base. Both arms carrying gold are below it, and the seed-43 replicate of the highest-gold arm lands +0.0902 from its seed-42 twin, which is the same variance warned about in caveat 1. This is a four-point monotone pattern at one seed each, not a controlled dose-response study: read it as the reason this checkpoint exists, not as a law about synthetic data.

Training

base	Qwen/Qwen2.5-Coder-1.5B-Instruct @ 2e1fd397ee46e1388853d2af2c993145b0f1098a
method	LoRA, rank 16, alpha 32, dropout 0.0, fp16
targets	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
trainable	18,464,768 parameters
rows	6,979 (0 owned, gold share 0.0000)
steps	250 (≈ 0.5732 epochs)
optimiser	adamw_torch, cosine, 60 warmup steps
learning rate	0.0001
batch	2 x 8 accumulation = 16 effective
sequence	1536 tokens, loss on completion only
clip / weight decay	0.3 / 0.01
precision	fp16 (no quantisation)
hardware	1 x NVIDIA Tesla T4 (Kaggle)
seed	42 (one seed; see caveat 1)

Data and attribution

This adapter contains no data authored by this project. Its 6,979 training rows are drawn entirely from the filtered third-party bulk tier of the project's SFT corpus. That makes upstream attribution more important here, not less: everything the model learned, it learned from the sources below.

source	repository	revision	licence	what was taken	in this adapter
owned_sft	Hakob-Harutyunyan/FineTuning-LLM (this repository)	sft_generator_v1	Apache-2.0	All generated, executable-verified EDA and feature-engineering conversations (Tier-1 gold).	0 rows (gold_family_cap 0)
opencodeinstruct	nvidia/OpenCodeInstruct	8f3ba5bafe4d6e8db46082cf7ae6741bc370604d	CC-BY-4.0	input/output pairs passing the 005 quality gate (test score >= 0.8, judge minimum >= 4, style budget).	bulk tier; not attributed per source
glaive_v3	glaiveai/glaive-code-assistant-v3	31a2e16324e6712f212d4361a768fc49295becff	Apache-2.0	question/answer pairs tagged with an ML category.	bulk tier; not attributed per source
opc_sft_stage2	OpenCoder-LLM/opc-sft-stage2	7d28f40d579edd7c24402d17d0c7639f991e6f8d	MIT	instruction/output pairs (all four configs) tagged with an ML category.	bulk tier; not attributed per source
codefeedback_filtered	m-a-p/CodeFeedback-Filtered-Instruction	a08c213a9748c66c15d0225814be80a2e77adf4a	Apache-2.0	Python query/answer pairs tagged with an ML category.	bulk tier; not attributed per source
magicoder_oss	ise-uiuc/Magicoder-OSS-Instruct-75K	5f839b1f368a76b161028bb9edff055db34022b2	MIT	Python problem/solution pairs tagged with an ML category.	bulk tier; not attributed per source
vezora_tested143k	Vezora/Tested-143k-Python-Alpaca	4cde7d9aa634ddecc3ecc796fa99e6a114b7edfe	Apache-2.0	execution-verified Alpaca pairs tagged with an ML category.	bulk tier; not attributed per source
jupyter_agent	data-agents/jupyter-agent-dataset	10a8dd461bb6c580fa25b83cd3b2f6810f1d6385	Apache-2.0	notebook trajectories converted to single-turn question -> code+answer pairs where a clean conversion exists.	bulk tier; not attributed per source

Aggregate corpus licence: CC-BY-4.0 — CC-BY-4.0 is the strictest source licence in the corpus (nvidia/OpenCodeInstruct), so the corpus is distributed under it. Base model Qwen/Qwen2.5-Coder-1.5B-Instruct is Apache-2.0. Vezora/Tested-143k-Python-Alpaca is used as filtered by "Vezora's CodeTester".

Rows had to carry exactly one parseable Python block and were tagged by capability from imports and API calls in the code rather than prompt keywords. Training prompts were decontaminated against the project's evaluation suites, HumanEval and MBPP (normalised exact match or word 5-gram Jaccard ≥ 0.8).

Licence verification is OUTSTANDING

Do not read this section as a clearance. The licence strings above are recorded constants in this project's corpus builder that were never re-verified against the live dataset cards; the project's own source-registration task is still open. Two further things are true and unresolved:

• Several upstream sets are themselves LLM-generated, including the Alpaca-lineage and OSS-Instruct data. A repository licence attached to such a set does not necessarily describe the provenance of the generated content inside it.
• The pinned revisions above are what was ingested, but whether each card still states the licence recorded here at those revisions has not been rechecked.

What is known is recorded above. What is unverified is named here. This release is private-by-default pending that verification, and nothing on this card asserts that publication is cleared.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    revision="2e1fd397ee46e1388853d2af2c993145b0f1098a",
    torch_dtype="float16",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "Hak5/qwen2.5-coder-1.5b-ml-engineer")
tokenizer = AutoTokenizer.from_pretrained("Hak5/qwen2.5-coder-1.5b-ml-engineer")

messages = [
    {"role": "system", "content": (
        "You are a rigorous ML engineer. Produce executable Python, inspect data before modeling, prevent leakage, justify validation, and return reusable artifacts."
    )},
    {"role": "user", "content": "Join event and reading tables as-of each event "
      "time, treating readings older than 15 minutes as stale."},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
generated = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(generated[0], skip_special_tokens=True))

The system turn above is byte-identical to the one this adapter was trained and evaluated under. Every number on this card was measured with it; results degrade without it. Decoding is greedy at 512 new tokens, which is the condition all the numbers above hold at.

Provenance of this card

Generated from artifacts, not written by hand. Every number above was read out of score_only_014.json and pre_registration_014.json (run 014) and 013's arms/D/arm_record.json, and compared against the readings this publisher was built on; a mismatch fails the publish instead of shipping the card. metrics.json beside these weights carries the readings, the contrasts, the verdicts, the pre-registration verbatim, and a sha256 for every published file.

Adapter bytes: step_250 from arm D, r16 / alpha 32 over 7 projections, verified before staging. Suite digest 5e67291b8d51ff05369c21b771427abb72d898fef7392a97738bec3faa8578f2.

Figures: chart_headline.png, chart_categories.png, chart_all_candidates.png, chart_dose_response.png, chart_discordance.png, rendered at publish time from the same readers as the prose and uploaded beside these weights. Each is followed on this card by the numbers it draws.

Card generated on 2026-07-30.