Update env0 mobile Qwen3.5-9B SFT adapter env0-mobile-pr828-qwen35-bf16-lora-8k-h100-20260625T084605Z

by bingran-you - opened 18 days ago

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+241

-154

Files changed (4) hide show

README.md +201 -141
adapter_config.json +37 -10
adapter_model.safetensors +2 -2
tokenizer_config.json +1 -1

README.md CHANGED Viewed

@@ -1,147 +1,207 @@
 ---
-library_name: peft
 base_model: Qwen/Qwen3.5-9B
-datasets:
-- benchflow/general-agent-qwen35-9b-azure-gpt54mini-sft
 tags:
 - lora
-- peft
-- sft
-- general-agent
-- qwen
 ---
-# Qwen3.5-9B General-Agent SFT LoRA Adapter
-`v0.0.1` is the completed LoRA SFT release for the Prime `general-agent` reproduction using the full, non-prequantized `Qwen/Qwen3.5-9B` base checkpoint. It does **not** include the base weights; load this adapter on top of `Qwen/Qwen3.5-9B`.
-This release intentionally excludes the next QLoRA run that is currently in progress. That run will be documented and tagged separately after its training and eval finish.
-## Release Summary
-| Field | Value |
-| --- | --- |
-| Release tag | `v0.0.1` |
-| Adapter repo | `benchflow/benchflow-qwen35-9b` |
-| Base checkpoint | `Qwen/Qwen3.5-9B` |
-| Base checkpoint form | Full, non-quantized source checkpoint; frozen during LoRA SFT |
-| Adapter type | LoRA / PEFT |
-| Source completed run | `general-agent-qwen35-9b-sft-seq2048-fresh-20260624T131847Z` |
-| W&B project | `general-agent-qwen35-9b-sft-seq2048-fresh-20260624T131847Z` |
-| HF training artifacts | `benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-9b-sft-seq2048-fresh-20260624T131847Z` |
-| Published at | `2026-06-24 22:27:07 UTC` |
-## Research Reproduction Scope
-The goal of this adapter is to reproduce the SFT-stage lift from Prime Intellect's `general-agent` work as closely as possible while using a smaller student model that can train on one H100. The stack keeps the Prime-style task and verifier path:
-- Source tasks: open-source `PrimeIntellect-ai/research-environments/environments/general_agent` task corpus.
-- Teacher trace generation: `general-agent-solver-rlm` + Azure GPT-5.4-mini through native Verifiers / `vf-eval --save-results` artifacts.
-- SFT trainer: Prime-RL SFT.
-- Student: full, non-quantized `Qwen/Qwen3.5-9B` loaded in BF16 with LoRA adapters.
-- Eval: `general-agent-solver-local` through native `vf-eval --save-results` on the same held-in task sets before and after SFT.
-## Data Recipe
-| Field | Value |
-| --- | --- |
-| Dataset | `benchflow/general-agent-qwen35-9b-azure-gpt54mini-sft` |
-| Dataset rows | `4414` |
-| Original source task count | `4417` |
-| Teacher model | Azure GPT-5.4-mini |
-| Teacher harness | Prime/Verifiers `general-agent-solver-rlm` |
-| Artifact format | Native `vf-eval --save-results` trajectories converted to Prime-RL `messages` + `tool_defs` SFT rows |
-| Excluded source tasks | `dog_breeding_t1`, `skydiving_center_t1`, `skydiving_center_t2` |
-| Exclusion reason | Stable Azure content-filter blocks during teacher trace generation |
-| Full teacher sweep artifact | `benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-daytona-teacher-full4417-tunnel8-20260624T015706Z` |
-| Data validation | Prime SFT JSONL validator rejected non-leading system messages and leakage fields before training |
-## Training Parameters
-| Field | Value |
-| --- | --- |
-| Trainer | Prime-RL SFT |
-| Model loaded for SFT | `Qwen/Qwen3.5-9B` full BF16 base weights |
-| Quantization | None for the completed `v0.0.1` LoRA run |
-| Adapter | LoRA |
-| LoRA rank | `16` |
-| LoRA alpha | `32` |
-| LoRA dropout | `0.0` |
-| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
-| Trainable params | about `29.1M` |
-| Adapted base params | about `5.30B` |
-| Total base params loaded | about `9.44B` |
-| Sequence length | `2048` |
-| Global batch size | `8` |
-| Micro batch size | `1` |
-| Pack function | `cat` |
-| Shuffle | `true` |
-| Seed | `0` |
-| Optimizer | `AdamW` |
-| Learning rate | `5e-5` |
-| Weight decay | `0.01` |
-| Betas | `0.9`, `0.999` |
-| Grad norm clip | `1.0` |
-| Scheduler | Linear |
-| Warmup steps | `20` |
-| Decay steps | `180` |
-| Minimum LR | `0.0` |
-| Max steps | `200` |
-| Checkpoint interval | `20` |
-| Keep last | `3` |
-| Keep interval | `100` |
-| Save format | `safetensors` |
-| Loss mask | Assistant messages only; system, user, and tool messages are context-only |
-## Training Result
-| Metric | Value |
-| --- | ---: |
-| Completed step | `200` |
-| Final loss | `0.11897` |
-| `loss/nan_count` | `0` |
-| Peak GPU memory | about `40.8 GiB` |
-| Final adapter | `adapter_model.safetensors` in this repo |
-The initial `data.seq_len=8192` Prime-RL BF16 LoRA attempt OOMed on one H100. The completed `v0.0.1` run used `data.seq_len=2048`, system CUDA 12.8 `nvcc`/`ptxas`, and `g++-12` for the required FLA/TileLang kernels.
-## Evaluation Results
-All evaluations below use native Verifiers `vf-eval --save-results`, `general-agent-solver-local`, serving context length `4096`, `--enable-auto-tool-choice`, and `--tool-call-parser qwen3_xml`. Dynamic vLLM LoRA loading was not reliable for this stack, so eval served a merged local checkpoint built from this adapter plus `Qwen/Qwen3.5-9B`.
-| Task set | Base pass rate | LoRA SFT pass rate | Delta | Notes |
-| --- | ---: | ---: | ---: | --- |
-| Held-in 5 smoke | `1/5 = 20.00%` | `2/5 = 40.00%` | `+20.00 pp` | First serving/eval smoke |
-| Held-in 20 | `11/20 = 55.00%` | `13/20 = 65.00%` | `+10.00 pp` | Recovered `3d_print_shop_t1`, `accounting_firm_t1` |
-| Held-in 36 | `20/36 = 55.56%` | `23/36 = 63.89%` | `+8.33%` | No regressions; recovered `3d_print_shop_t1`, `accounting_firm_t1`, `allergy_clinic_t0` |
-| Held-in 50 assembled | `27/50 = 54.00%` | `30/50 = 60.00%` | `+6.00%` | Latest wider held-in result; final 14-task slice had no net delta |
-| Held-in 50 final 14-task slice | `7/14 = 50.00%` | `7/14 = 50.00%` | `+0.00%` | Recovered `animation_studio_t0`; regressed `antiquarian_bookshop_t0` |
-Evaluation artifact prefixes:
-- Held-in 5 smoke: `benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-smoke4096-20260624T152150Z`
-- Held-in 20 comparison: `benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-heldin20-compare-20260624`
-- Held-in 36 comparison: `benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-heldin36-compare-20260624`
-- Held-in 50 final 14-task run: `benchflow/env0-experiment-trajectories/experiments/general-agent/general-agent-qwen35-eval-heldin50-gap-20260624T190517Z`
-## Loading
-```python
-from peft import PeftModel
-from transformers import AutoModelForCausalLM, AutoTokenizer
-base = AutoModelForCausalLM.from_pretrained(
-    "Qwen/Qwen3.5-9B",
-    torch_dtype="auto",
-    trust_remote_code=True,
-)
-model = PeftModel.from_pretrained(base, "benchflow/benchflow-qwen35-9b")
-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B", trust_remote_code=True)
-```
-## Caveats
-- This is an SFT-stage reproduction artifact, not the full Prime paper recipe with the original teacher and student model stack.
-- The trainable dataset has `4414` rows rather than `4417` because three Azure teacher prompts were blocked by content filtering.
-- The latest held-in50 assembled lift is positive but modest at `+6.00 pp`; gains are concentrated in a small number of tasks rather than broad across-the-board recovery.
-- The next QLoRA seq8192 experiment is excluded from `v0.0.1` and should receive its own update/tag only after it completes.

 ---
 base_model: Qwen/Qwen3.5-9B
+library_name: peft
+pipeline_tag: text-generation
 tags:
+- base_model:adapter:Qwen/Qwen3.5-9B
 - lora
+- transformers
 ---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.18.0

adapter_config.json CHANGED Viewed

@@ -1,19 +1,46 @@
 {
-  "peft_type": "LORA",
-  "task_type": "CAUSAL_LM",
   "base_model_name_or_path": "Qwen/Qwen3.5-9B",
-  "r": 16,
-  "lora_alpha": 32.0,
-  "lora_dropout": 0.0,
   "bias": "none",
   "target_modules": [
-    "down_proj",
-    "gate_proj",
-    "k_proj",
     "o_proj",
     "q_proj",
     "up_proj",
-    "v_proj"
   ],
-  "modules_to_save": null
 }

 {
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
   "base_model_name_or_path": "Qwen/Qwen3.5-9B",
   "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
   "target_modules": [
     "o_proj",
+    "v_proj",
+    "k_proj",
     "q_proj",
     "up_proj",
+    "gate_proj",
+    "down_proj"
   ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
 }

adapter_model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fa60aae6e769faad42c7bb6c11a4685c7713da24e60175603e2926c943eaa7cd
-size 58233888

 version https://git-lfs.github.com/spec/v1
+oid sha256:cb01c366d71c056bfc133717e7e782c1e4b67c3452cb2adde51b59c305031852
+size 232818064

tokenizer_config.json CHANGED Viewed

@@ -21,7 +21,7 @@
     "vision_bos_token": "<|vision_start|>",
     "vision_eos_token": "<|vision_end|>"
   },
-  "pad_token": "<|im_end|>",
   "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",

     "vision_bos_token": "<|vision_start|>",
     "vision_eos_token": "<|vision_end|>"
   },
+  "pad_token": "<|endoftext|>",
   "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",