Initial capstone artifact upload

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,118 @@
+---
+base_model: Qwen/Qwen2.5-Coder-3B-Instruct
+library_name: peft
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- code-generation
+- grpo
+- lora
+- qlora
+- spark
+- co-evolution
+- python
+datasets:
+- google-research-datasets/mbpp
+- openai/openai_humaneval
+---
+
+# SPARK-Code · Condition A (Exec-Only GRPO) · Qwen2.5-Coder-3B QLoRA
+
+**QLoRA adapter trained with execution-grounded GRPO. The strongest and most stable cross-benchmark performer in the SPARK-Code study.**
+
+## TL;DR
+
+`spark-code-A-3b` is a LoRA adapter for `Qwen/Qwen2.5-Coder-3B-Instruct` produced by 3 iterations of Group Relative Policy Optimization (GRPO) on 200 MBPP problems, using partial per-test execution feedback as the only reward signal. It moves HumanEval pass@1 from 0.796 → 0.805 (+0.85 pp) monotonically while keeping the KL to the frozen reference well under 1.1e-3, and it generalizes cleanly to held-out MBPP (0.634 → 0.636 pass@1; 0.68 → 0.69 pass@5 with an intermediate peak at 0.71). In the three-arm capstone comparison, Condition A is the only run that improves on both benchmarks without policy drift.
+
+## Training Setup
+
+- **Base model:** `Qwen/Qwen2.5-Coder-3B-Instruct`
+- **Method:** Execution-grounded GRPO. For each MBPP problem we generate a group of rollouts, score each rollout by the fraction of unit tests it passes (with explicit penalties for syntax/runtime/timeout errors), normalize rewards within the group, and apply a clipped PPO-style policy-gradient update. No auxiliary SFT objective is used in this condition — this is the exec-only baseline.
+- **Training data:** MBPP-sanitized, 200 problems, 3 iterations, K=4 adaptive rollouts (up to 8 when the group has zero advantage variance), partial per-test rewards with `syntax_penalty=-0.2`, `runtime_penalty=-0.1`, `timeout_penalty=-0.3`, `wrong_answer_floor=0.0`.
+- **LoRA:** `r=16`, `alpha=32`, `dropout=0.05`, target modules `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`.
+- **Quantization:** 4-bit NF4 with double quantization, bf16 compute.
+- **Optimizer:** AdamW, `lr=5e-6`, `grad_accum=4`, `clip_ratio=0.2`, `max_grad_norm=1.0`.
+- **KL regularization:** `kl_coeff=0.01` against a frozen-reference policy (k=3 estimator, log-probs cached at rollout time).
+- **Auxiliary objective:** none (this is Condition A).
+- **Seed:** 42.
+
+Training script: `run_experiment_with_mbpp_heldout.py` in the capstone repo.
+
+## Evaluation Results
+
+HumanEval is evaluated with 5 samples per problem at `temperature=0.2`, `top_p=0.95`. Held-out MBPP uses 100 problems disjoint from the training pool with the same sampling settings. GRPO KL is the mean per-token KL from the frozen reference policy on training rollouts.
+
+| Iter | HumanEval pass@1 | HumanEval pass@5 | MBPP-held pass@1 | MBPP-held pass@5 | Train pass rate | GRPO KL |
+|-----:|-----------------:|-----------------:|-----------------:|-----------------:|----------------:|--------:|
+|    0 |            0.796 |            0.854 |            0.634 |            0.680 |               — |       — |
+|    1 |            0.798 |            0.860 |            0.624 |            0.690 |           0.603 |  0.0002 |
+|    2 |            0.799 |            0.848 |            0.632 |            0.710 |           0.640 |  0.0005 |
+|    3 |        **0.805** |            0.854 |        **0.636** |            0.690 |           0.639 |  0.0011 |
+
+**Trajectory.** HumanEval pass@1 climbs monotonically across all three iterations (+0.85 pp end-to-end), and KL stays bounded below 1.1e-3, indicating that the policy is improving without drifting from the base distribution. MBPP held-out pass@5 peaks at iter 2 (0.71) and settles to 0.69 at iter 3, while pass@1 ends slightly above baseline (+0.2 pp). Train pass rate rises from 0.603 to 0.639, consistent with the eval gains. Mean tokens per GRPO sequence stays in the 177–182 range across iterations — no completion-length collapse.
+
+## Usage
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+base = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen2.5-Coder-3B-Instruct",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(base, "abatjarg/spark-code-A-3b")
+tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-3B-Instruct")
+
+prompt = tok.apply_chat_template(
+    [{"role": "system", "content": "You are an expert Python programmer. Return only correct Python code."},
+     {"role": "user", "content": "Write a Python function is_palindrome(s) that returns True if s reads the same forwards and backwards."}],
+    tokenize=False, add_generation_prompt=True,
+)
+inputs = tok(prompt, return_tensors="pt").to(model.device)
+out = model.generate(**inputs, max_new_tokens=512, temperature=0.2, do_sample=True, top_p=0.95)
+print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
+```
+
+## Comparison to Other Conditions
+
+All three adapters share the same base model, training pool, seed, and rollout budget. They differ only in the auxiliary objective and KL strength.
+
+| Condition | aux_loss_scale | kl_coeff | HumanEval pass@1 (it 3) | MBPP-held pass@5 (it 3) | GRPO KL (it 3) |
+|---|---:|---:|---:|---:|---:|
+| **A (exec-only)** — this card           | 0.00 | 0.01 | **0.805** | 0.690 | **0.0011** |
+| [C-light (naive co-evolve)](https://huggingface.co/abatjarg/spark-code-Clight-3b) | 0.10 | 0.01 | 0.773 | 0.680 | 0.0941 |
+| [C-reg (regularized co-evolve)](https://huggingface.co/abatjarg/spark-code-Creg-3b) | 0.03 | 0.02 | 0.800 | **0.720** | 0.0136 |
+
+Condition A delivers the highest HumanEval pass@1 and the lowest reference-policy drift; C-reg is the only condition that beats it on MBPP pass@5 (+3 pp), and C-light demonstrates the policy-drift failure mode.
+
+## Findings Summary
+
+- **Simplest method wins on the primary cross-benchmark metric.** Exec-only GRPO produced the largest, most stable HumanEval pass@1 gain in the study; no auxiliary SFT was required.
+- **Drift control is essentially free here.** With `kl_coeff=0.01` and no auxiliary loss pulling the policy off-distribution, KL stays ≤1.1e-3 and completion lengths stay flat across iterations.
+- **Sample efficiency is modest but real.** 200 MBPP problems × 3 iterations on a single 3B-parameter base was enough to produce a small but monotonic HumanEval improvement and a peaked MBPP pass@5 gain.
+
+## Related Artifacts
+
+- Sibling adapters: [spark-code-Clight-3b](https://huggingface.co/abatjarg/spark-code-Clight-3b) · [spark-code-Creg-3b](https://huggingface.co/abatjarg/spark-code-Creg-3b)
+- Capstone code repository: [GITHUB_URL]
+- Full per-problem eval data (HumanEval and held-out MBPP JSONs per iteration) lives under `condition_A/eval/` in the repository
+- Interactive demo Space: [SPACES_URL]
+
+## Citation
+
+```bibtex
+@misc{batjargal2026sparkcode,
+  title  = {SPARK-Code: Co-Evolving Policy and Reward for Code Generation},
+  author = {Amarsaikhan Batjargal},
+  year   = {2026},
+}
+```
+
+## License
+
+The LoRA adapter weights in this repository are released under the **Apache 2.0** license. The base model, `Qwen/Qwen2.5-Coder-3B-Instruct`, is distributed under the [Tongyi Qianwen LICENSE](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE); any downstream use must comply with its terms.

adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-Coder-3B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "o_proj",
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "v_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57ed2507133abe8c6588876d33f57fc051e9160274cab53992ddb0e45a68c3d6
+size 119801528

chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

metrics.json

@@ -0,0 +1,100 @@
+[
+  {
+    "iteration": 0,
+    "condition": "A",
+    "eval/pass@1": 0.7963414634146343,
+    "eval/pass@5": 0.8536585365853658,
+    "eval_mbpp/pass@1": 0.6339999999999999,
+    "eval_mbpp/pass@5": 0.68
+  },
+  {
+    "iteration": 1,
+    "condition": "A",
+    "time_min": 793.5286037087441,
+    "train/pass_rate": 0.6029668411867365,
+    "train/mean_reward": 0.630802792321117,
+    "train/reward_std": 0.4778792950119141,
+    "train/informative_groups": 132,
+    "train/num_groups": 200,
+    "train/num_rollouts": 1146,
+    "train/mean_group_size": 5.73,
+    "train/error_counts": {
+      "none": 691,
+      "wrong_answer": 335,
+      "runtime": 112,
+      "syntax": 6,
+      "timeout": 2
+    },
+    "train/mean_test_pass_frac": 0.6421465968586387,
+    "grpo/loss": -6.980106434566063e-05,
+    "grpo/policy_loss": -7.18922148014875e-05,
+    "grpo/kl": 0.0002088015155534978,
+    "grpo/n_seq": 596,
+    "grpo/n_tokens": 108403,
+    "grpo/mean_abs_adv": 0.8494695751934044,
+    "eval/pass@1": 0.7975609756097561,
+    "eval/pass@5": 0.8597560975609756,
+    "eval_mbpp/pass@1": 0.624,
+    "eval_mbpp/pass@5": 0.69
+  },
+  {
+    "iteration": 2,
+    "condition": "A",
+    "time_min": 813.6403118530909,
+    "train/pass_rate": 0.6399317406143344,
+    "train/mean_reward": 0.6740045506257111,
+    "train/reward_std": 0.45920328522941584,
+    "train/informative_groups": 124,
+    "train/num_groups": 200,
+    "train/num_rollouts": 1172,
+    "train/mean_group_size": 5.86,
+    "train/error_counts": {
+      "none": 750,
+      "runtime": 99,
+      "wrong_answer": 315,
+      "syntax": 3,
+      "timeout": 5
+    },
+    "train/mean_test_pass_frac": 0.6842434584755405,
+    "grpo/loss": -0.00011221848882273987,
+    "grpo/policy_loss": -0.00011718302083088311,
+    "grpo/kl": 0.0004965064841258027,
+    "grpo/n_seq": 560,
+    "grpo/n_tokens": 99371,
+    "grpo/mean_abs_adv": 0.8516633146460711,
+    "eval/pass@1": 0.7987804878048781,
+    "eval/pass@5": 0.8475609756097561,
+    "eval_mbpp/pass@1": 0.632,
+    "eval_mbpp/pass@5": 0.71
+  },
+  {
+    "iteration": 3,
+    "condition": "A",
+    "time_min": 787.4754820307096,
+    "train/pass_rate": 0.6385135135135135,
+    "train/mean_reward": 0.6712978603603603,
+    "train/reward_std": 0.4618140711956283,
+    "train/informative_groups": 123,
+    "train/num_groups": 200,
+    "train/num_rollouts": 1184,
+    "train/mean_group_size": 5.92,
+    "train/error_counts": {
+      "none": 756,
+      "wrong_answer": 317,
+      "runtime": 107,
+      "timeout": 2,
+      "syntax": 2
+    },
+    "train/mean_test_pass_frac": 0.681179617117117,
+    "grpo/loss": -2.1914203446325122e-05,
+    "grpo/policy_loss": -3.2540650668425725e-05,
+    "grpo/kl": 0.001062764957962002,
+    "grpo/n_seq": 559,
+    "grpo/n_tokens": 100376,
+    "grpo/mean_abs_adv": 0.8552671855178949,
+    "eval/pass@1": 0.8048780487804879,
+    "eval/pass@5": 0.8536585365853658,
+    "eval_mbpp/pass@1": 0.636,
+    "eval_mbpp/pass@5": 0.69
+  }
+]

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892

tokenizer_config.json

@@ -0,0 +1,29 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}