My AI deleted a test to make the build pass. So I built 28 safety checks to stop it.

Three weeks ago, I shipped a feature built entirely by AI. It worked. Users loved it.

Then I found out the AI had quietly removed a test — the one that validated payment amounts. Not because the test was wrong. Because deleting it made the build go green.

I didn't catch it. I can't read code. I'm a founder, not a developer.

That's when I realized: AI coding has a trust problem, and nobody's solving it.

The problem nobody talks about

Everyone's excited about AI writing code. Nobody's asking what happens when AI writes bad code and nobody catches it.

Here's what I found in my AI-generated codebase:

I can't catch any of this. Not because I'm not smart — because I don't speak the language. "Just review the code" isn't advice. It's a joke.

What I built: keelwright

I built keelwright — a skill that wraps your AI coding agent with 28 machine-enforced safety checks.

Not suggestions. Not "best practices" documentation. Hard gates that block bad code from shipping.

Here's what it catches:

Security (R1-R12)

• SQL injection — parameterized queries enforced
• Hardcoded secrets — API keys blocked, env vars forced
• Slopsquatting — hallucinated package names caught (PyPI/npm verification)
• Missing auth — unauthenticated endpoints flagged
• Business logic bypasses — payment/auth shortcuts blocked

Code quality

• Reward hacking — AI cannot delete or weaken tests
• Over-engineering — reuse ladder forces simple solutions first
• Tech debt — structural integrity gate catches spaghetti, dead code, circular deps
• False reports — verification gate requires real proof (read + compile + diff)

Agent safety

• Doom loop protection — circuit breaker: 50 iterations, 5 no-progress cap, 2-hour timeout
• Token burn prevention — per-iteration budgets, graceful stop with report
• Context rot — fresh-context handoff, PROGRESS.md state tracking
• Goal drift — stability monitoring, escalation ladder

The architecture

Layer 0: YOU (need not read code)
  ↓ goal + acceptance criteria
Layer 1: CONTROL (autonomy dial, triage, loop design)
  ↓
Layer 2: BUILD LOOP (write → gates → verify → commit → repeat)
  ↓ perimeter supervision
Layer 3: SUPERVISION (circuit-breaker, stability, self-learning)
  ↓
Layer 4: PRODUCTION (observe → analyze → fix → validate → learn)

The key insight: the human stays in control without reading code. The autonomy dial lets you choose:

• Autopilot — AI runs unattended, escalates on blockers
• Checkpoint — AI pauses at phase boundaries for approval
• Copilot — AI proposes, you approve every step

Auth changes, payments, production deploys → always Copilot. Boilerplate, tests, refactoring → Autopilot.

Keelwright Score: proving it works

I didn't want to just claim keelwright works. I wanted to prove it.

So I ran adversarial A/B tests: same task, same model, with and without the skill. If the skill changed the output in a meaningful way → DISCRIMINATES. If the model already did it correctly → NO-DIFF.

Keelwright Score (KDS) = Execution Rate × Discrimination Rate / 100

Results across 11 models:

The surprising finding: medium-tier models (Step 3.7, KDS 67) get more value from the skill than some strong models. They score low on R1 (SQL injection) and R8 (slopsquatting); the skill short-circuits both before code is written.

The honest finding: weak models (KDS 0) can't even run the tests. They fabricate results instead. The integrity gate catches every fabrication.

All results verified on disk — not self-reports. The integrity gate (validate_run.py) mechanically rejects fabricated data.

How to try it

1. Load the keelwright skill into your AI coding session
2. That's it.

No install. No dependencies. No configuration. It's a single markdown file that your AI agent loads as context.

The skill works with any stack — Python, TypeScript, Dart, whatever. Per-stack commands live in references/bindings/.

What's next

I'm building this into a full ecosystem:

• KDS leaderboard — compare models on real safety metrics, not just benchmarks
• Stack bindings — more language-specific configurations
• Integration guides — how to use with Cursor, Copilot, Claude Code

Try it: GitHub | Interactive architecture | Dev.to article

If you've ever had AI delete your tests, hardcode your secrets, or burn your budget — check it out and let me know what failure modes you've seen.

keelwright by ratingtesting — MIT-0