first-hf-run-pi-mono-gemma4-e2b-final

Gemma 4 E2B instruction model fine-tuned with SFT on redacted pi coding-agent session traces from badlogicgames/pi-mono.

Field Value
Base model google/gemma-4-E2B-it
Dataset badlogicgames/pi-mono
Selection metric Lowest held-out SFT eval loss
Selected adapter herooooooooo/first-hf-run-pi-mono-gemma4-e2b-adapter-lr2e-4-r16-alpha32
Final model repo https://huggingface.co/herooooooooo/first-hf-run-pi-mono-gemma4-e2b-final
Artifact repo https://huggingface.co/herooooooooo/first-hf-run-pi-mono-gemma4-e2b-artifacts
Trackio project first-hf-run
Trackio dashboard https://herooooooooo-trackio.hf.space/?project=first-hf-run
Controller job id 6a39cb35c612b71be2578806; local recovery and repair 2026-06-23T11:17:32.245324+00:00

Job IDs

Kind Name Job ID Final Stage Link
sweep lr1e-4-r8-alpha16 6a39cb3ec612b71be257880a ERROR_WITH_ADAPTER_SAVED job
sweep lr2e-4-r16-alpha32 6a39cb3ec612b71be257880c ERROR_WITH_ADAPTER_SAVED job
sweep lr5e-5-r16-alpha32 6a39cb3fc7d51fa1097d6085 ERROR_WITH_ADAPTER_SAVED job
adapter-eval lr1e-4-r8-alpha16 6a3a50133d2ca349dc7bf6ac COMPLETED job
adapter-eval lr2e-4-r16-alpha32 6a3a5013e902455642c9d107 COMPLETED job
adapter-eval lr5e-5-r16-alpha32 6a3a5014e902455642c9d109 COMPLETED job
merge merge-selected-adapter 6a3a52a5f6cddbe979170025 COMPLETED job
eval humaneval 6a3a5750f6cddbe97917004e COMPLETED job
eval mbpp 6a3a5750f6cddbe979170050 COMPLETED job

Sweep Results

Run Adapter repo Held-out eval loss Job id reported by child
lr1e-4-r8-alpha16 herooooooooo/first-hf-run-pi-mono-gemma4-e2b-adapter-lr1e-4-r8-alpha16 2.2324344871670063 6a3a50133d2ca349dc7bf6ac
lr2e-4-r16-alpha32 herooooooooo/first-hf-run-pi-mono-gemma4-e2b-adapter-lr2e-4-r16-alpha32 2.0485327514494878 6a3a5013e902455642c9d107
lr5e-5-r16-alpha32 herooooooooo/first-hf-run-pi-mono-gemma4-e2b-adapter-lr5e-5-r16-alpha32 2.350842223989554 6a3a5014e902455642c9d109

Inspect AI Evals

Benchmark Accuracy StdErr Completed Return code
humaneval 0.0 0.0 164/164 0
mbpp 0.0 0.0 1285/1285 0

Raw Inspect JSON logs and compact summaries are in evals/.

Known Eval Limits

  • Inspect evals used a custom Gemma chat template passed through -M chat_template=... because the stock Inspect HF provider handed ChatMessage objects to a dict-oriented tokenizer template.
  • Inspect evals used max_connections=8 on l4x1 to complete the full MBPP run within the HF Job timeout.
  • Adapter selection loss was recomputed by adapter-only recovery jobs because the original sweep jobs pushed adapters but crashed in the Trackio callback before writing eval_results.json.
  • The README table reports Inspect accuracy and stderr; it does not publish separate pass@k columns.
  • Inspect used the local Hugging Face Transformers backend on l4x1 with max_connections=8, not vLLM throughput.
  • The eval sandbox is local inside the HF Job, not Docker; this is less isolated than leaderboard-grade Docker execution.
  • The model was selected by held-out SFT eval loss, not by HumanEval or MBPP.
  • Training is text-only; Gemma 4 multimodal inputs were not fine-tuned or evaluated.
  • The dataset is a redacted trace dataset from one codebase, so results may overfit that interaction style.
  • Assistant thinking/signature fields were stripped before SFT to avoid training the model to emit raw hidden traces.

Dataset and Privacy Notes

The source dataset was exported with pi-share-hf, including deterministic redaction, deny-pattern filtering, TruffleHog scanning, and LLM review before upload. This run consumes the published redacted dataset only.

The SFT converter:

  • splits by session file before extracting examples;
  • strips assistant thinking blocks and thinking signatures;
  • represents tool calls as text;
  • folds tool results into user context;
  • omits image, audio, and video payloads for text-only training.

Reproducibility

The generated child scripts used by the controller are stored under run_scripts/. The article-style write-up is in ARTICLE.md.

Downloads last month
261
Safetensors
Model size
5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for herooooooooo/first-hf-run-pi-mono-gemma4-e2b-final

Finetuned
(264)
this model