README.md

---
license: gemma
base_model: google/gemma-4-E2B-it
tags: [code, python, web, agent, qlora]
---

# dogmatix v0.2

A QLoRA fine-tune of **google/gemma-4-E2B-it** (~2.3B effective) into a **local,
Python-specialised web-application coding agent**. Trained on executed,
test-verified trajectories distilled from a GLM-5.2 teacher running live inside
the real harness. Runs on a single 16GB GPU.

> **v0.2 supersedes v0.1** (still available on the `v0.1` branch). v0.2 is a
> sharper web-app specialisation: better at end-to-end app-building — the thing
> this project is *for* — at a measured cost to Django and general Python. The
> numbers below are honest, single-harness, and include the trade.

## What it is for

Small, well-specified Python web work — FastAPI, Flask, SQLAlchemy/SQLModel —
driven through an agent loop: read files, make the edit, run the tests, fix what
broke. Fast, local, private; not a frontier engineering model.

## Measured results

All figures are measured under **one harness** (the model sees the full 7-tool
surface — read/write/edit/grep/bash/list/run_tests — with turn-based step
counting and a blind-call guard). v0.1's original card reported a web-app number
taken under an earlier, inflated harness; it is corrected here.

| benchmark | untuned base | v0.1 | **v0.2** |
|---|---|---|---|
| End-to-end app-building pass@1 (12 specs) | 0.182 | 0.273 | **0.364** |
| Web-app agentic pass@1 (156 tasks) | 0.756 | 0.788 | **0.797** |
| Django pass@1 (60 tasks) | 0.567 | 0.550 | **0.500** |
| MBPP (general Python, 100) | 0.760 | 0.710 | **0.700** |

**The headline is the first line.** *End-to-end app-building* — build a working
app from a natural-language spec, graded by a hidden acceptance suite the model
never sees — is what this model is for, and it is the one benchmark with real
headroom (base only 0.182). The tuned checkpoints improve **monotonically, and
v0.2 doubles the base.**

**Honest caveat:** that benchmark is small (11–12 graded specs, one attempt
each), so the gain is **not yet statistically significant** (base-vs-v0.2
McNemar p ≈ 0.63). A larger multi-attempt sweep is running to settle it. Treat
"doubles base" as a strong signal, not a proven number. On the older **web-app
suite** v0.2 is at parity with base (+0.04, not significant) — that suite is
saturated (base 0.756), which is why the app-building benchmark was built.

### Product-path parity — the proxies are faithful

Every number above is from the eval loop; the thing users run is the serving
proxy. Running the 12 app-build specs **through the actual proxies** on v0.2:

| path | app-building pass@1 |
|---|---|
| eval-flat reference | 0.333 |
| Pi (OpenAI proxy) | 0.417 |
| Claude Code (Anthropic proxy) | 0.500 |

Both proxy paths are **at or above** the eval reference — the serving and tool-
name translation layer loses no capability. Tool-calling works end-to-end on both
harnesses, including WebSearch pass-through (v0.2 reaches for and uses a WebSearch
tool when offered, though it was not explicitly trained for search).

### Where fine-tuning helps, and where it costs

- **Helps** at end-to-end app-building (the objective): 0.18 → 0.36. Notably, the
  base and v0.1 each have a task they cannot even *build* without running out of
  memory on a runaway trajectory; **v0.2 completes every build in the suite.**
- **Costs** general Python and Django. v0.2 is the **worst of the three at Django
  (0.500 vs 0.567 base) and MBPP (0.700 vs 0.760 base)** — a sharper
  specialisation than v0.1. Django is the thinnest part of the training corpus.
  If you need Django or a general Python coder as much as app-building, weigh this
  trade, or use the `v0.1` branch (Django 0.550) or the base.

## Honest limitations

- **It cannot reliably build a whole application in one pass.** It succeeds far
  more often when work is **decomposed** into small sub-tasks (how the bundled
  harness drives it). Even then ~1 in 3 of these small app specs succeeds end to
  end — useful with verification, not autonomous.
- **It cannot be trusted to judge its own work** — it has produced a correct app
  alongside broken tests for it. Use external verification.
- **The app-building benchmark is small and underpowered** (12 specs); the gain
  is directional, not yet significant.
- **The web-app suite is saturated** (base 0.756) — treat that headline as an
  upper bound.
- **Python only.** No other languages, no JS frontends, no deployment.
- **Long horizons degrade.** Design target is 5–10 tool-steps per loop; longer
  work needs decomposition.

## Serving

Ships with an OpenAI-compatible and an Anthropic-compatible proxy
(`dogmatix/serving/`). The translation layer is part of the model: it enforces a
turn boundary the base chat template does not provide (without which the model
emits long bursts of tool calls and never observes their results), guards against
acting on unobserved tool calls, and maps client tool names (Claude Code / Pi)
onto the trained tool vocabulary — passing through tools it has no trained
equivalent for (Bash, WebSearch, …).

## Training

QLoRA (4-bit NF4, r=32) on the language-model projections only, loss masked to
assistant tokens, merged to bf16 on CPU. Single adapter, single run, curriculum
expressed as data-sampling weights. Trajectories are executed and test-verified —
never free-generated.

## Reproducing the numbers

- App-building: `python scripts/appbuild_bench.py` (12 specs with hidden
  acceptance suites verified against reference implementations before any run).
- Product-path parity: `python scripts/appbuild_proxy_bench.py`.
- Web-app / Django / MBPP: `python scripts/benchmark.py` (Django via
  `--suite benchmark/tasks-django`).

## Author

Prasad Vittaldev