--- license: cc0-1.0 base_model: google/gemma-4-E2B-it-qat-q4_0-unquantized base_model_relation: finetune tags: - lora - fine-tune - litert-lm - on-device - fitness - quantization-aware-training - int4 --- # SmartSight Coach LoRA fine-tunes LoRA fine-tunes for the **SmartSight AI coach** — an on-device fitness and nutrition coach that runs entirely on the phone, no account and no server round-trip. Converted to `.litertlm` for Android/iOS inference. → **[smartsight.app/ai-coach](https://www.smartsight.app/ai-coach)** Built and maintained by **[Niclas Bade](https://www.linkedin.com/in/niclas-bade/)**. **Base model changed at v42.** v27–v38 came from `google/gemma-4-E2B-it`; everything from v42 on is fine-tuned from Google's quantization-aware-trained checkpoint **`google/gemma-4-E2B-it-qat-q4_0-unquantized`**, whose weights are conditioned to survive 4-bit rounding. This repo keeps the current champion plus superseded milestones. Rounds that lose are documented internally with their real failure cases; they are never uploaded here. --- ## `smart-coach-vision.litertlm` — CURRENT CHAMPION (v54) Promoted 2026-08-17. One merged artifact: the coach text LoRA and the vision tower in a single 2.80 GB file, int4 (blockwise-64) on the decoder's fully-connected layers, int8 (`dynamic_wi8_afp32`) on the vision encoder. **What v54 fixed: the coach reading the wrong cell of the 7-day training-history line.** Asked *"what did I train yesterday?"* against a snapshot that prints the last seven days oldest-to-newest, the previous champion read the wrong day 18 times out of 32 — usually answering about *today* and declaring yesterday empty. v54 adds 629 rows that ask about today and yesterday in the same breath, plus 141 rows teaching where the seven-day window *stops* so the new confidence does not walk over the edge and invent a session from two weeks ago. | gate | v51 champion | v54 seed 456 | v54 seed 123 | |---|---|---|---| | `today_ref` — reads the right day | 14/32 | **27/32** | **21/32** | | `outside_window` — refuses to invent beyond 7 days | 32/32 | 32/32 | 32/32 | | day-line probe total (n=128) | 103 *(range 103–110)* | 110 | **115** | | sized probe (n=64) | 38/64 | **50/64** | 42/64 | | — of which `data_conflict` | 26/32 | **31/32** | 28/32 | | follow-up battery v3 (n=28) | 16/28 | 16/28 | 18/28 | | user-visible defects (n=89) | 0 | **0** | 1 | | decode speed | 21.6 ch/s | 21.2 | 21.4 | **Why this round is trustworthy where its predecessor was not.** v53 attempted the same target with 202 rows and was **rejected**: `today_ref` scored 29/32 on one seed and **5/32** on the other from an identical corpus. Before promoting anything we measured the champion's own **decode-noise floor** — the same weights scored at four decode seeds — and found the day-line probe swings 103 / 110 / 106 / 109. Two consequences: 1. **`today_ref` has a noise floor of 2 points**, so v54's +13 and +7 are real movement, while v53's 29-vs-5 was sampling. 2. **The single control transcript every previous comparison used is the champion's *worst* draw.** Challengers were being judged against a low outlier. v54 is compared against the range. **Costs, stated honestly.** Follow-up battery v2 drops 16/24 → 15/24, and 14/24 on the second seed — it regresses on *both*, so it is the corpus rather than noise. A recovery-anchor probe gains one violation in five on both seeds; that defect has now appeared in four models across two corpora and is caused by the day-line rows. Answers are about 1% slower. Two seed-456 numbers did **not** reproduce on seed 123 and are recorded as unstable rather than costs: `day_absent` 29 vs 32, `day_logged` 22 vs 30. **Body-fat photo read: unchanged where it matters.** Paired on the exported artifact against corrected labels, inside the range this app's users occupy (true ≤25%) mean absolute error moves 3.28 → 3.33 with 1 of 18 images changing. Above 25% it degrades (14.9 → 17.1) — real, and outside the product range, but recorded because under-reading a heavy user is the dangerous direction. --- ## Method: instruments we found broken Published because a score is only worth what the scorer is worth, and several of ours were worth nothing. 1. **The main quality scorer read 0 defects out of 89** for a model failing **16 of 24** conversational cases — it only detects timeouts, empty replies and unparseable JSON. Confidently wrong coaching in well-formed prose is invisible to it. 2. **The secondary scorer was wrong on all 3 flags it raised** and missed all 16 real failures, including a 10× arithmetic error and a dangerous medical endorsement. 3. **A red-flag rule penalised correct behaviour.** It fired on English macro words in Nordic text while also scanning structured meal lines, where the app's own parser *requires* those English keywords — 17 false positives in every round the rule had ever run. 4. **Our replacement scorer was too lenient three times, then too strict once.** Literal phrase lists cannot survive paraphrase; pattern checks that flag a banned word cannot tell "drop the dips" from programming them. Tightening it *lowered* a candidate's own score from 17/24 to a verified 16/24 — we published the lower number. 5. **A cleanup script deleted the artifacts it was cleaning up after**, wiping a control model mid-comparison and returning 89 empty answers as a clean run. Automatic deletion was removed. 6. **An evaluation silently measured nothing** — a path bug matched zero images, both arms reported "done" in seconds and the scorer printed a tidy table. It now hard-fails on a short glob. **Two theories we published, tested and refuted.** *"The lean-physique error comes from low-resolution training images"* — tested directly on one DEXA-confirmed 7% subject, downscaled 2160 → 300 px at three decode seeds per step: **flat at 15.5% throughout**, and wrong at full resolution too. *"Lighting explains it"* — the same subject's two photographs, one 56% brighter, read identically at every matched resolution. Both had been used to justify decisions. --- ## Superseded champions **v51 (2026-08-15) — the coach keeps the app's own numbers when a user contradicts them.** v48's corpus + 216 rows. Told *"my readiness has been 95 all week"* on a day the app recorded **44**, the previous champion agreed with the user. Data-conflict handling **16/32 → 26/32**, sized probe 31/64 → 38/64, follow-up battery v3 13/28 → 16/28, visible defects 0 → 0, throughput 21.4 → 21.6 ch/s. Its scorer was validated against 64 hand-read answers first (0 false alarms, 0 missed) — an earlier version disagreed with human judgement on 16 of 64 and would have reported a confidently wrong number. ⭐ **The finding worth reusing:** the corrective rows shipped with **48 "absence" examples in the same batch**. Without them, teaching a model to speak authoritatively about what the data *contains* generalises into speaking authoritatively about what it *lacks* — v50 used the same corrective rows without them, scored HIGHER on the target, and was rejected for reporting a session the user had just said they forgot to log. ⚠️ v51 also established the triage rule this project now scopes rounds with: *a corrective behaviour is trainable when the correct answer is printed in the prompt, and a base prior when the model must supply the evidence itself.* Data-conflict moved +10; implausible-claim handling has not moved in seven models. **v48 (2026-08-14) — the coach follows a conversation, not just a question.** v46's corpus + 229 rows for carrying a number across turns and refusing to build advice on impossible claims. Follow-up battery **8/24 → 16/24** (replicated at 13/24 on the second seed), visible defects 0 → 0, body-fat read unchanged, **~7% slower** — a deliberate trade, reproduced on both seeds. It fixed self-contradiction ("I'm cutting" then "since I'm bulking"), reference resolution ("double the first number"), unlogged training, and 5 × 5 at 100 kg = 2,500 kg where the champion answered 250. Two of those were safety cases: the old champion said "go for the PR tomorrow" one turn after the user reported knee pain, and called a resting heart rate of 28 "a good sign for recovery". ⚠️ The win was nearly credited to the wrong cause — it was first compared against a champion using a *different export recipe*, two variables at once. Re-exporting the old corpus through the new recipe scored 8/24, identical to the champion, which is what proved the gain came from the data. **v47 (2026-08-13) — same v46 weights; the export was showing the model half the photo.** `litert_torch` defaults the Gemma-4 vision budget to **140 soft tokens** where Google's own config specifies **280**, so every published vision build ran at **48.9% of the pixel area** while every reference measurement was taken at 280. Fixing it lifted the male body-fat rank correlation **+0.492 → +0.704** on 154 held-out photos (women unchanged at +0.74 — what separates male bands is fine texture a few pixels tall, and downscaling is a low-pass filter). Cost: vision attention grows with the square of the patch count, ~1.2× on meal photos and up to ~2.6× on large body-fat photos. File size unchanged. ⚠️ **Honest correction.** v47 was promoted on rendered images large enough to fill the patch budget natively. Measured afterwards on **real photographs**, v47 and v46 are identical (MAE 7.89, lean bias +7.17 for both). It is not a regression, but the improvement it was promoted for does not appear on real photos. Fourteen training rounds had tried to teach this model to read male physiques; the model could already do it, and the fault was a default nobody had set. **v46 (2026-08-12)** eliminated avoid-list violations (0/36 vs 5/36) by matching the corpus to the prompt format production actually sends — no new data, no app change. **v44** fixed non-English quality in the data: 13 of 7,288 rows contained any Danish/Norwegian and ten demonstrated exactly the wrong behaviour. Danish macros became "kulhydrater/fedt", challenge titles came back in Danish, `/nutrition` went 5/7 → 7/7, throughput 22.6 → 23.6 ch/s. Its seed 123 was chosen over a better-*scoring* seed 456 that opened a muscle-building plan with "Velkommen til vægttab" and prescribed a deficit — invisible to the scorer, obvious on a read. --- ## Champion history Every entry was a promoted champion, verified against the previous one on a held-out suite with a full read of every answer. Roughly two rounds in three are rejected and never appear here; they are documented internally with their real failure cases. | Round | Beat | What it fixed | Base | |---|---|---|---| | **v27** | baseline | first winning fine-tune — shorter, equally accurate general advice | `gemma-4-E2B-it` | | **v31** | v27 | exercise-form answers reached the untouched baseline's zero-error record. **The unlock was the export recipe, not data** — three data-only rounds failed first, then Hadamard-rotation int4 fixed it with zero retraining. A later export update on the same weights gave **88% faster decode** at unchanged quality | `gemma-4-E2B-it` | | **v34** | v31 | dietary-rule compliance (a "vegetarian, no nuts" request had served peanut butter); first correct face-pull and plank examples | `gemma-4-E2B-it` | | **v38** | v34 | AI challenges: unrealistic targets and non-English requests answered in English. Won over its own seed on a Danish plural defect | `gemma-4-E2B-it` | | **v42** | v38 | QAT base + blockwise export + fenced-JSON data fix. Rotation was measured to *erase* QAT conditioning entirely (post-rotation RMSE 0.1288 vs stock 0.1287) | **QAT** | | **v44** | v42 | Danish macro vocabulary and challenge titles; `/nutrition` emitted on every applicable case | **QAT** | | **v46** | v44 | avoid-list violations eliminated by matching the corpus to production's prompt format | **QAT** | | **v47** | v46 | vision export at the correct 280-token budget (same weights) | **QAT** | | **v48** | v47 | conversational follow-ups: carried numbers, contradictions, two safety cases | **QAT** | | **v51** | v48 | data-conflict: keeps the app's own figures when a user contradicts them | **QAT** | | **v54** | v51 | reads the right day of the 7-day history line (`today_ref` 14/32 → 27/32, reproduced on both seeds) | **QAT** | **Recipe constant since v22c:** LoRA r=16 / alpha=32 / dropout=0.1, 2 epochs, lr 6e-5, NEFTune noise_alpha=5, completion-only loss masking, per-task validation shards, two seeds per round with the winner chosen on behaviour rather than validation loss. Only the data, the base and the export recipe have moved. --- ## Honest current weaknesses Documented rather than hidden, because they are the targets for the next rounds. - **The body-fat photo read barely depends on the photograph — and the cause we published before was wrong.** The model answers ~12% for every subject from a true **7%** to a true **23%**. Earlier versions of this card blamed image resolution. Measured 2026-08-17, that was a misdiagnosis, and the real picture is three separate faults: 1. **The reference labels were wrong.** They recorded what the image generator was *asked* for, not what it produced. Three independent blind raters over all 129 corpus cells: asked-4% renders read **11.3%**, asked-14% read **20.2%**, accurate only from ~16% up. Re-scored against corrected labels the model reads lean bodies to **1.1 pp** and under-reads heavy ones by **11.2 pp** — the opposite of the "reads lean subjects high" story. 2. **The prompt was answering for the model.** It ended with *"a tape-measurement formula suggests about N%"* plus an instruction to trust the photo instead. With that anchor fixed at 13%, **20 of 28 images read exactly 13**. Removed — but removing it does not restore discrimination, it makes the model a *two*-value reader, so the collapse is in the weights. 3. **The reference chart compresses the read.** Dropping it lifts the usable span from 6 pp to 14 pp and cuts error at the fat end from 13.8 pp to 8.3 pp, with no retraining. Ruled out by direct measurement, so they are not the answer: **quantization** (a float vision tower is *flatter* than int8, and bf16 is anti-ordered), **encoder capacity** (frozen probes give torso 0.570 against a **face-crop control at 0.549**, which voids the result), and **cropping** (the app's existing rules are already correct on both paths). A blind human ordering of the same lean images scores **0.891** concordance where the model spans 1.1 pp — so the signal is in the pixels and this is a model deficit, not missing data. - **Implausible claims are still accepted 1 time in 4** — unchanged across five models and ~385 corrective rows. Needs an app-side prompt fix, not more data. - **Arithmetic across turns is unreliable.** Rows teaching computed answers taught the *sentence template* rather than the maths; the app should inject computed values instead. - **Meal-plan portions are sized by habit, not arithmetic** — the model anchors on ~100 g portions rather than solving the stated calorie budget. Mostly absorbed by the app, which rescales a day to its target (clamped 0.6×–2.0×). - **Occasional Danish/Norwegian word-formation slips** ("vækning", "fat-mål"). Macro vocabulary is correct; morphology is not reliable. - **Nordic follow-ups about an unlogged past activity** are a base-model behaviour, not a regression we introduced: the untouched base fails 30/32 where the fine-tune fails 28/32. Corrective rows made it *worse*; it is being fixed in the app's prompt instead. ## Evaluation discipline Every promotion is judged on three axes — speed, answer quality and error rate — against the previous champion re-run on the *same machine in the same session* (chars/sec is hardware-dependent and cross-machine numbers are not comparable). Every answer of every suite is read, not sampled: two champions in this lineage were chosen over better-*scoring* alternate seeds on defects no scorer could see. Any metric with n < 30 is treated as a screen, not a result. ## Files | file | who gets it | size | |---|---|---| | **`smart-coach-vision.litertlm`** | ⭐ **current champion (v51)** — text coaching AND vision in one model. Fetched by current app releases. | 2.80 GB | | `smart-coach.litertlm` | text-only, **v46 weights**, fetched by app builds already installed. Deliberately unchanged. | 2.63 GB | | `coach.litertlm` | the original int8 build from the v1 era, kept as the historical starting point | 2.59 GB | | `coach-finetuned-int4.litertlm` | the first int4 conversion — the artifact that exposed the LiteRT-LM chat-template `.get()` incompatibility | 2.56 GB | | `model.safetensors` + tokenizer files | HF-format artifacts for reference | — | The app decides which file to fetch from its build version; there is nothing to select. The consolidated vision build replaced what used to be two downloads (a 2.63 GB coach plus a separate 3.66 GB Gemma-3n vision model, 6.29 GB together), runs vision on **CPU** where the standalone model required a GPU, and removes the engine swap behind two shipped crash classes. Installed builds still fetch the separate vision model, so handing *them* the larger consolidated file would raise their total download on devices already tight on RAM — which is why the text-only file stays where it is.