action: checkpoint

details: All scripts updated to google/gemma-4-E4B-it. Anti-parroting fixes: LoRA r=4, epochs=1. prep_dataset_rich.py created with 12-batch multi-perspective design covering 13 input variables.


details: Ran prep_dataset_rich.py sequentially on 1 A10G. Timed out. Switched to parallel approach.


details: Refactored prep_dataset_rich.py to use Modal .map() — 12 batches on 12 separate A10Gs concurrently. Estimated ~15 min, ~$15 cost.


details: 12 GPU jobs launched. Batches generating concurrently. Model: google/gemma-4-E4B-it, temperature=0.8.


action: audit

details: Fixed eval.py and README.md Gemma 3 references. Extracted v1 history to REPORT_v1.md. Updated all file tables. 18 files inventoried, 3 deprecated marked.


action: research

details: Found google/gemma-4-E4B-it-qat-q4_0-unquantized — QAT-trained float model, fine-tunable. Better GGUF quality after quantize. Documented as v3 option in REPORT.md and RUNBOOK.md.


action: plan

details: Documented parallel training setup. Track A: Standard E4B. Track B: QAT-unquantized. Both will run on the dataset simultaneously.


details: Verified user's fast dataset generation run. Budget stands at $7.21 spent, $92.79 remaining.


details: Fast dataset generation nearing completion at 96/120. Outputting context-aware reasoning for TPU overhangs.


action: documentation

details: Added explicit Agent Protocol to BUDGET.md and RUNBOOK.md to enforce activity.jsonl usage for future agents.


details: Eval set generation in progress at 11/80. Seeing context-aware settings for PLA stringing.


action: train

details: Started Track A (Standard E4B) fine-tune and push to kylebrodeur/microfactory-node-lora-v2. Modal App ID: ap-6XiWWsyXzFOK0zAWskvLW4


details: Started Track B (QAT-unquantized) fine-tune and push to kylebrodeur/microfactory-node-lora-v3-qat. Modal App ID: ap-idunQc5EsF0tIuhCv6KSGJ


details: Track A (Standard E4B) fine-tuning completed successfully. Loss: ~2.069. Adapter pushed to kylebrodeur/microfactory-node-lora-v2 (35MB).


details: Track B (QAT-unquantized) fine-tuning completed successfully. Loss: ~1.751. Adapter pushed to kylebrodeur/microfactory-node-lora-v3-qat (35MB).


action: eval

details: Started parallel evaluation for Track A (Standard E4B) and Track B (QAT). Running on 80 held-out examples.


details: Both evaluation tracks hit the 1800s (30m) Modal timeout while generating the TUNED responses. Increased timeout to 3600s.


details: Bumped eval_modal.py timeout to 7200s (2 hours) to be absolutely safe against further timeouts.


details: Refactored eval_modal.py to use .map() to run the BASE and TUNED evaluations on separate A10G GPUs concurrently, cutting the total evaluation time exactly in half (from ~30 mins to ~15 mins).


details: Refactored eval_modal.py to chunk the dataset into sizes of 20, mapping across 8 A10G GPUs (4 chunks x 2 models) to drastically reduce eval time to under 8m.


details: Refactored eval_modal.py again to find the perfect balance: instead of 8 GPUs per track (which risks quota limits and heavy cold-start penalties), it uses 2 GPUs per track. Each GPU evaluates both BASE and TUNED sequentially for 40 examples. This guarantees under 8m execution while minimizing instance boots.


details: Fixed PermissionError in eval_modal.py. When moving the file reading logic to the local entrypoint during the sharding refactor, the path was incorrectly left as the container mount path ('/root/sft.eval.jsonl'). Updated to read from the local data directory.


details: Successfully launched both Track A and Track B evaluations in parallel. The chunked evaluation logic is functioning, and the baseline evaluation is processing the chunks.


details: Both evaluations finished perfectly under the 8m mark. TUNED matched BASE perfectly with 100% valid JSON and 100% spine-safe parameters. Most importantly, TUNED provided uniquely tailored reasoning and varied temperature adjustments based on context instead of collapsing to a single templated output like in v1. The Well-Tuned badge is officially secured.


action: cleanup

details: Verified all Modal apps have been stopped. Documented benign PEFT and PyTorch warnings to REPORT.md to prevent future confusion. Completed full pipeline validation.


details: Completed serving/deployment research. Created SERVING.md covering Ollama publishing (simplified to Merge→GGUF→Ollama), Modal hosting feasibility (confirmed YES, designed modal_serve.py), and Gradio model switching design (dropdown + llm_zerogpu_lora.py). Fixed stale E2B→E4B in llm_zerogpu.py.


action: serving

details: Created gguf_pipeline_modal.py — full merge→GGUF pipeline on Modal. GPU for merge, CPU for llama.cpp build+convert. No local setup needed. One command per track.


details: Created modal_serve.py — FastAPI endpoint on Modal GPU with OpenAI-compatible /v1/chat/completions. Auto-scales to zero. Separate $100 serving budget.


details: Created core/llm_zerogpu_lora.py — LoRA-aware ZeroGPU backend. Added _apply_model_choice(), MODEL_OPTIONS, MODEL_LORA_MAP to app.py. Rolled back UI placement changes per user request (another agent handling UI). Left clear handoff note in SERVING.md.


details: GGUF pipeline running on Modal (ap-ZYdn9niRL6ywRgXPYcIjTz). llama.cpp building at 11%. Merge step completed, convert step in progress.


details: Modal inference API deploying (ap-60wirJOd35PZl1ZIKakD9v). Installing dependencies. Fixed two Modal API deprecations: container_idle_timeout->scaledown_window, allow_concurrent_inputs->@modal.concurrent.


details: All three serving items implemented: 1) gguf_pipeline_modal.py for Ollama GGUF on Modal, 2) modal_serve.py for Modal inference API, 3) core/llm_zerogpu_lora.py + app.py backend for Gradio model switcher. UI placement deferred to other agent per user request.


details: Modal inference API deployed successfully at https://kylebrodeur--microfactory-node-inference-serve.modal.run. Image built in 71s, app deployed in 75s.


action: deploy

details: Deploy preflight 10/10 green. Pushed to build-small-hackathon/microfactory-lab. Factory reboot triggered. Space rebuilding. Note: Gradio warns about build_job arg count (model_choice has default, UI agent will wire dropdown).


details: Killed old v2-only GGUF run. Launched both v2 and v3 GGUF pipelines in parallel with distinct --name flags (microfactory-node-v2, microfactory-node-v3-qat). Reusing cached llama.cpp image — no rebuild.


details: Space now defaults to LoRA v3 (QAT E4B) — best training loss (1.75), best quantization quality. Pre-warmed at startup. Model options: v3 → v2 → Base → Modal API.


details: Pulled latest (UI agent docs). Deploy preflight 10/10 green. Pushed to Space, factory reboot triggered. Model switcher fully wired with Modal API backend.


details: Pulled UI agent updates (theme, app). Deploy 10/10 green. Space rebuilding. GGUF pipelines re-running with full deps.


details: Removed all emojis from buttons. Used CSS mask-image to inject SVG icons (bolt/shuffle/anchor) as ::before pseudo-elements. Constrained LORA dropdown to 200px fixed width. Set all buttons to scale=0. Verified with Playwright smoke test - 0 emojis in app (1 from HF org badge only).