HONEST_SUBMISSION.md

Honest Submission Guardrails

Jawbreaker should compete hard without gaming the hackathon.

What We Can Claim

• Backyard AI: Jawbreaker is built for a real, specific safety problem: helping someone pause before clicking, replying, or sending money.
• Real user story: the motivating user is a friend's grandmother who had already been affected by scam messages; public docs omit identity and private message details.
• Small-model constraint: the deployed model is under 32B parameters.
• Gradio Space: the app runs as a Gradio app under the hackathon organization.
• OpenBMB targeting: MiniCPM is central to the deployed scam analysis path.
• Codex usage: the GitHub repo has Codex-attributed commits and the build logs document Codex's role.
• Custom UI: the app uses a custom Gradio interface, not the default demo look.
• Off the Grid: the app does not call an external LLM API; it loads the open MiniCPM base model and Jawbreaker LoRA adapter directly in the app runtime. Hugging Face ZeroGPU is the demo compute layer, not a hosted model API.
• Eval work: the repo includes hand-curated synthetic/sanitized eval cases plus generated synthetic train/dev/test data.
• Public data page: sanitized/synthetic eval and training artifacts are published at build-small-hackathon/jawbreaker-scam-defense-data.
• Well-Tuned: a MiniCPM LoRA adapter was trained on Modal, published to Hugging Face, and evaluated before deployment.
• Modal usage: Modal was used for training and eval; it is not the live inference backend.
• Safety guardrails: model JSON is validated, unsafe actions are checked in eval, and a deterministic fallback handles malformed model output.

What We Should Not Claim Unless Completed

• Do not claim real-world user validation until the target person has actually tried the app.
• Do not turn the grandmother story into a measured outcome claim unless we have a quote, permission, or concrete post-use feedback.
• Do not claim the generated synthetic dataset is real user data.
• Do not claim benchmark superiority unless we publish the eval command, dataset, and result.
• Do not claim llama.cpp as the deployed runtime unless the live Space actually uses the llama.cpp path.
• Do not confuse transport with model dependency. The hosted Space is legitimate judge-facing ZeroGPU compute; the Off the Grid claim is about avoiding hosted/external LLM APIs.
• Do not hide fallback behavior. If MiniCPM fails and Jawbreaker uses deterministic safety fallback, document that as a reliability layer.

Training Rules

• Training data and eval data must stay separated.
• eval/scam_eval.jsonl is the hand-curated eval set and should not be used as training data.
• training/data/test.jsonl and eval/generated_eval.jsonl are holdout data and should not be used for LoRA training.
• A LoRA adapter should only be deployed if it improves valid JSON and safety metrics without increasing dangerous false negatives.
• The current deploy candidate is build-small-hackathon/jawbreaker-minicpm5-1b-lora-v8, which reached 579/632 risk accuracy on the hard guarded eval with zero dangerous-as-safe misses, zero dangerous-as-needs-check misses, zero safe overcalls, zero invalid predictions, zero model errors, and zero unsafe action violations.
• If synthetic training mostly teaches format-following, say that. Do not describe it as learning from real scam victims.
• If Modal is used, include the Modal command, training output, and whether an adapter was actually published.

Demo Rules

• The demo can use sanitized examples.
• If a real family story is used, remove names, phone numbers, addresses, account details, and links.
• The demo should show the actual Space behavior, not a scripted mockup.
• If the Space is slow because of ZeroGPU or MiniCPM loading, say so in field notes rather than pretending it is instant.

Winning Without Cheating

The strongest honest story is:

Jawbreaker is a narrow, polished Backyard AI tool using a fine-tuned OpenBMB MiniCPM LoRA adapter under the 32B limit. It turns a suspicious message into a clear scam DNA breakdown and a safe next step for a non-technical person. The repo shows evals, safety fallbacks, Modal training, Codex traces, field notes, and no-external-LLM-API runtime evidence. Claims are limited to what is actually built and measured.