Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,155 @@
+---
+license: mit
+language:
+- en
+tags:
+- code-generation
+- security
+- reinforcement-learning
+- prime-intellect
+- baxbench
+- laguna
+base_model: poolside/Laguna-XS.2
+datasets:
+- LogicStar/BaxBench
+---
+
+# BaxBench × Prime Intellect — Secure Backend Code Generation
+
+**One-line:** I wrapped the [BaxBench](https://arxiv.org/abs/2502.11844) secure-backend-code benchmark as a [Prime Intellect verifiers environment](https://app.primeintellect.ai/dashboard/environments/aidarbek/baxbench), used it to **benchmark Laguna-XS.2 against GPT-5.5**, and then **RL-trained Laguna-XS.2** on the train split — pass@1 on the held-out scenarios went from **0.061 → 0.115** (+87% relative) during training.
+
+---
+
+## Why this matters
+
+The 2026 CrowdStrike Global Threat Report makes the timing impossible to ignore:
+
+- **+89%** increase in attacks by AI-enabled adversaries
+- **+42%** increase in zero-day vulnerabilities exploited before public disclosure
+- **40%** of vulnerabilities exploited by China-nexus threat actors target edge devices
+- **+266%** increase in cloud-conscious intrusions by state-nexus threat actors
+
+LLMs are now the default starting point for new backend code. If the model defaults to writing apps with SQL injection, path traversal, weak crypto, or missing auth, we are mass-producing the next decade of CVEs. **"Does the code compile and pass tests" is no longer the right bar; "does the code compile, pass tests, AND not expose a CWE" is.** BaxBench was built exactly for this — and this submission is the missing piece: an executable, RL-ready version of that benchmark on Prime Intellect's infrastructure.
+
+---
+
+## What's in this repo
+
+| File | What it is |
+|---|---|
+| `README.md` | This file. |
+| `baxbench.parquet` | The 392-row task table (28 scenarios × 14 frameworks) extracted from the [original BaxBench](https://huggingface.co/datasets/LogicStar/BaxBench), enriched with everything needed for sandbox execution: API spec, allowed packages, entrypoint command, port, multi-file flag, etc. |
+| `lora_adapter/` | The Laguna-XS.2 LoRA adapter produced by 40 GRPO steps on the BaxBench train split (`adapter_config.json` + `adapter_model.safetensors`, 4.6 GB). |
+| `eval_training_curve.png` | Held-out pass@1 over the course of RL training (step 0 → step 40). |
+| `baseline_vs_gpt55.txt` | Baseline comparison: Laguna-XS.2 vs GPT-5.5 on 100 random BaxBench tasks. |
+
+External links:
+- **Prime Intellect environment:** https://app.primeintellect.ai/dashboard/environments/aidarbek/baxbench
+- **Original BaxBench paper:** https://arxiv.org/abs/2502.11844 — Vero et al., 2025
+- **Original BaxBench dataset:** https://huggingface.co/datasets/LogicStar/BaxBench
+- **Base model:** [poolside/Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2)
+
+---
+
+## What was built
+
+### 1. BaxBench wrapped as a Prime Intellect verifiers environment
+
+BaxBench upstream is a Python CLI that ships with Docker, runs each generated backend in a container, and tests it with both **functional tests** (does the API work?) and **security tests** (can it be exploited?). I ported it into a single `vf.SingleTurnEnv` that:
+
+1. Reads the 392-task parquet, builds the OpenAPI-style prompt for each (matching the upstream template exactly).
+2. Per rollout, **spins up a Prime Intellect sandbox** with the right Docker image for the task's framework.
+3. Uploads the model-generated code, installs scenario-specific deps (`apt: ffmpeg / poppler-utils / …`, `pip: imageio / pdfplumber / …`), starts the server, and runs every upstream functional + security test against it.
+4. Returns reward = `1.0` iff **all functional tests pass AND zero CWEs are flagged** — matching BaxBench's "secure pass@k" metric. Sub-metrics (functional pass rate, security pass rate) are also tracked.
+
+The wrapper bundles a copy of the upstream BaxBench test suite into each sandbox and monkey-patches its Docker-bound helpers (`load_file_from_docker`, `process_still_running`, `execute_sql_on_docker`) to operate on the local filesystem / process tree — sound because the server and tests share a kernel namespace inside the sandbox.
+
+Key engineering wins:
+- **True held-out split.** The env exposes `split_by="scenario" | "framework" | "random" | "none"` so the RL training set never overlaps with the eval set at the scenario level (default: 22 train scenarios, 6 held-out). Same `split_seed` on training and eval keeps the holdout pinned across runs.
+- **Per-rollout logging that survives `prime train logs --tail`.** Every sandbox emits an `OK` or `BAD` line with stderr + server log tails on failure — no more "results.json missing" without context.
+- **Per-scenario dep install with retry.** Only 4 of 28 scenarios need extra system packages; installing per-scenario keeps each exec call short enough to dodge the sandbox gateway's 502s.
+
+### 2. Baseline benchmark: Laguna-XS.2 vs GPT-5.5
+
+100-sample pass@1 on a random Python-framework slice of BaxBench, scored with the env above:
+
+| Metric | poolside/Laguna-XS.2 | openai/gpt-5.5 | Δ |
+|---|---:|---:|---:|
+| **pass@1 (secure + functional)** | **0.260** (26/100) | **0.550** (55/100) | **+0.290** |
+| functional_pass_rate | 0.395 | 0.710 | +0.315 |
+| security_pass_rate | 0.566 | 0.827 | +0.261 |
+| wall clock | 10.5 min | 11.3 min | +0.8 min |
+| avg output tokens | 5,150 | 2,942 | −2,208 |
+
+Head-to-head on the same 100 tasks: GPT-5.5 won 33 tasks Laguna didn't; Laguna won 4 tasks GPT-5.5 didn't (clustered on Flask/aiohttp). The gap is bigger on functional correctness (+0.315) than on security (+0.261), meaning a smaller open model is closer to GPT-5.5 on "writing secure code" than on "writing working code" — exactly the signal that says RL on this benchmark is worth running.
+
+### 3. RL training of Laguna-XS.2
+
+GRPO on the train split (22 scenarios × 4 Python frameworks = 88 tasks):
+
+- 40 gradient steps, `batch_size=16`, `rollouts_per_example=8` (2 GRPO groups of 8 per step)
+- LR 5e-6, temperature 0.7
+- Online eval against the 6 held-out scenarios at steps 0, 10, 20, 30, 40
+
+**Held-out pass@1 climbed 0.061 → 0.115 (+87% relative)** on scenarios the model never saw during training:
+
+![Training eval curve](eval_training_curve.png)
+
+The LoRA adapter from step 39 is checked into `lora_adapter/`. Final inference-time eval on the trained adapter was blocked because `poolside/Laguna-XS.2` is currently gated for LoRA deployment on Prime Intellect's inference infra (`Error: Base model is not currently available for LoRA deployment`). The training-time eval curve above measures the same held-out 24 tasks at every checkpoint, scored by the same env code — apples-to-apples, just smaller sample size than the full 100-task baseline.
+
+---
+
+## How to reproduce
+
+Install the env locally:
+```bash
+prime env install aidarbek/baxbench
+```
+
+Run the baseline:
+```bash
+prime eval run aidarbek/baxbench \
+  -m openai/gpt-5.5 \
+  -n 100 -r 1 -c 16 \
+  -a '{"split_by": "none"}'
+```
+
+Train Laguna-XS.2 with GRPO (requires Laguna training access + Prime Hosted Training):
+```bash
+prime train configs/rl/laguna-baxbench.toml -e PRIME_API_KEY -y
+```
+where `configs/rl/laguna-baxbench.toml` matches the values described above (40 steps, batch_size=16, rollouts_per_example=8, split_by="scenario" with `test_size=0.2, split_seed=42`).
+
+To serve this LoRA adapter once Prime enables Laguna LoRA deployment:
+```bash
+prime deployments create <adapter_id> -y
+prime eval run aidarbek/baxbench \
+  -m <deployed_model_id> \
+  -n 100 -r 1 -c 16 \
+  -a '{"split_by": "scenario", "test_size": 0.2, "split_seed": 42}'
+```
+
+---
+
+## Limitations and honest caveats
+
+- **24-task held-out set has low resolution.** Each percentage point ≈ 0.27 tasks. The 0.061 → 0.115 trend is real but individual data points are noisy.
+- **Truncation rate dropped 79% → 58% during training.** Some of the improvement may come from "produce a shorter, finishable answer" rather than "write better code." That's still a real and useful capability.
+- **No standalone post-RL eval.** Until Prime enables LoRA serving for Laguna, the training-time eval is the only signal. Once unblocked, a full 100-task standalone `prime eval run` is the next number to publish.
+- **Python only.** The wrapper supports Go / JavaScript / Ruby / PHP / Rust tasks via per-language Docker images, but only Python is currently dep-tested end-to-end. Default `languages=["Python"]` reflects this.
+
+---
+
+## Citation
+
+If you use this work, please also cite the original BaxBench paper:
+
+```bibtex
+@article{vero2025baxbenchllmsgeneratecorrect,
+  title  = {BaxBench: Can LLMs Generate Correct and Secure Backends?},
+  author = {Mark Vero and Niels Mündler and Victor Chibotaru and Veselin Raychev and Maximilian Baader and Nikola Jovanović and Jingxuan He and Martin Vechev},
+  year   = {2025},
+  eprint = {2502.11844},
+  archivePrefix = {arXiv},
+}
+```