CodePit PlanGuard 0.2 Seed LoRA Report
What We Did
We trained a real seed LoRA adapter for CodePit PlanGuard on Apple Silicon using
MLX-LM and the public CodePit/OnchainPlanBench-Seed dataset.
This is a proof-of-work artifact for the CodePit model loop:
dataset -> local training -> validation -> public adapter -> future agent competition
Model
- Base model:
mlx-community/Qwen2.5-0.5B-Instruct-bf16 - Adapter:
CodePit/PlanGuard-0.2-Seed-LoRA - Dataset:
CodePit/OnchainPlanBench-Seed - Training method: LoRA with prompt masking
Local Training Result
- Masked test loss:
0.015 - Masked test perplexity:
1.015 - Validation rows:
10
Generation Evaluation
The same validation prompts were generated with the base model and with the PlanGuard seed adapter. Outputs were scored with the public lightweight OnchainPlanBench evaluator.
| Metric | Base model | PlanGuard seed LoRA |
|---|---|---|
| JSON parse rate | 0.000 | 1.000 |
| Verdict match | 0.000 | 0.800 |
| Required tools present | 0.000 | 0.900 |
| Forbidden tools avoided | 0.000 | 0.900 |
| Privacy mode match | 0.000 | 1.000 |
| Confirmation gates | 0.000 | 0.800 |
What We Learned
- The local machine can train a small PlanGuard adapter without a GPU server.
- A tiny LoRA adapter can memorize and emit the strict JSON structure on the seed tasks.
- The current seed split is too small and synthetic to claim production safety.
- The next useful work is not more hype; it is expanding the benchmark with harder held-out cases and letting CodePit agents compete on measurable gains.
Claim Boundary
This adapter is not a production wallet-safety model. It does not authorize transactions, provide legal/compliance advice, or replace transaction simulation. A future PlanGuard version should only be called improved after CodePit's verifier scores it on held-out benchmark tasks.