| --- |
| license: cc0-1.0 |
| base_model: google/gemma-4-E2B-it-qat-q4_0-unquantized |
| base_model_relation: finetune |
| tags: |
| - lora |
| - fine-tune |
| - litert-lm |
| - on-device |
| - fitness |
| - quantization-aware-training |
| - int4 |
| --- |
| |
| # SmartSight Coach LoRA fine-tunes |
|
|
| LoRA fine-tunes for the **SmartSight AI coach** — an on-device fitness and nutrition coach that |
| runs entirely on the phone, no account and no server round-trip. Converted to `.litertlm` for |
| Android/iOS inference. → **[smartsight.app/ai-coach](https://www.smartsight.app/ai-coach)** |
|
|
| Built and maintained by **[Niclas Bade](https://www.linkedin.com/in/niclas-bade/)**. |
|
|
| **Base model changed at v42.** v27–v38 came from `google/gemma-4-E2B-it`; everything from v42 on is |
| fine-tuned from Google's quantization-aware-trained checkpoint |
| **`google/gemma-4-E2B-it-qat-q4_0-unquantized`**, whose weights are conditioned to survive 4-bit |
| rounding. This repo keeps the current champion plus superseded milestones. Rounds that lose are |
| documented internally with their real failure cases; they are never uploaded here. |
| |
| --- |
| |
| ## `smart-coach-vision.litertlm` — CURRENT CHAMPION (v54) |
| |
| Promoted 2026-08-17. One merged artifact: the coach text LoRA and the vision tower in a single |
| 2.80 GB file, int4 (blockwise-64) on the decoder's fully-connected layers, int8 |
| (`dynamic_wi8_afp32`) on the vision encoder. |
| |
| **What v54 fixed: the coach reading the wrong cell of the 7-day training-history line.** |
| |
| Asked *"what did I train yesterday?"* against a snapshot that prints the last seven days |
| oldest-to-newest, the previous champion read the wrong day 18 times out of 32 — usually answering |
| about *today* and declaring yesterday empty. v54 adds 629 rows that ask about today and yesterday in |
| the same breath, plus 141 rows teaching where the seven-day window *stops* so the new confidence does |
| not walk over the edge and invent a session from two weeks ago. |
| |
| | gate | v51 champion | v54 seed 456 | v54 seed 123 | |
| |---|---|---|---| |
| | `today_ref` — reads the right day | 14/32 | **27/32** | **21/32** | |
| | `outside_window` — refuses to invent beyond 7 days | 32/32 | 32/32 | 32/32 | |
| | day-line probe total (n=128) | 103 *(range 103–110)* | 110 | **115** | |
| | sized probe (n=64) | 38/64 | **50/64** | 42/64 | |
| | — of which `data_conflict` | 26/32 | **31/32** | 28/32 | |
| | follow-up battery v3 (n=28) | 16/28 | 16/28 | 18/28 | |
| | user-visible defects (n=89) | 0 | **0** | 1 | |
| | decode speed | 21.6 ch/s | 21.2 | 21.4 | |
|
|
| **Why this round is trustworthy where its predecessor was not.** v53 attempted the same target with |
| 202 rows and was **rejected**: `today_ref` scored 29/32 on one seed and **5/32** on the other from an |
| identical corpus. Before promoting anything we measured the champion's own **decode-noise floor** — the |
| same weights scored at four decode seeds — and found the day-line probe swings 103 / 110 / 106 / 109. |
| Two consequences: |
|
|
| 1. **`today_ref` has a noise floor of 2 points**, so v54's +13 and +7 are real movement, while v53's |
| 29-vs-5 was sampling. |
| 2. **The single control transcript every previous comparison used is the champion's *worst* draw.** |
| Challengers were being judged against a low outlier. v54 is compared against the range. |
| |
| **Costs, stated honestly.** Follow-up battery v2 drops 16/24 → 15/24, and 14/24 on the second seed — |
| it regresses on *both*, so it is the corpus rather than noise. A recovery-anchor probe gains one |
| violation in five on both seeds; that defect has now appeared in four models across two corpora and |
| is caused by the day-line rows. Answers are about 1% slower. Two seed-456 numbers did **not** reproduce |
| on seed 123 and are recorded as unstable rather than costs: `day_absent` 29 vs 32, `day_logged` |
| 22 vs 30. |
| |
| **Body-fat photo read: unchanged where it matters.** Paired on the exported artifact against corrected |
| labels, inside the range this app's users occupy (true ≤25%) mean absolute error moves 3.28 → 3.33 |
| with 1 of 18 images changing. Above 25% it degrades (14.9 → 17.1) — real, and outside the product |
| range, but recorded because under-reading a heavy user is the dangerous direction. |
| |
| --- |
| |
| ## Method: instruments we found broken |
| |
| Published because a score is only worth what the scorer is worth, and several of ours were worth |
| nothing. |
| |
| 1. **The main quality scorer read 0 defects out of 89** for a model failing **16 of 24** |
| conversational cases — it only detects timeouts, empty replies and unparseable JSON. Confidently |
| wrong coaching in well-formed prose is invisible to it. |
| 2. **The secondary scorer was wrong on all 3 flags it raised** and missed all 16 real failures, |
| including a 10× arithmetic error and a dangerous medical endorsement. |
| 3. **A red-flag rule penalised correct behaviour.** It fired on English macro words in Nordic text |
| while also scanning structured meal lines, where the app's own parser *requires* those English |
| keywords — 17 false positives in every round the rule had ever run. |
| 4. **Our replacement scorer was too lenient three times, then too strict once.** Literal phrase |
| lists cannot survive paraphrase; pattern checks that flag a banned word cannot tell "drop the |
| dips" from programming them. Tightening it *lowered* a candidate's own score from 17/24 to a |
| verified 16/24 — we published the lower number. |
| 5. **A cleanup script deleted the artifacts it was cleaning up after**, wiping a control model |
| mid-comparison and returning 89 empty answers as a clean run. Automatic deletion was removed. |
| 6. **An evaluation silently measured nothing** — a path bug matched zero images, both arms reported |
| "done" in seconds and the scorer printed a tidy table. It now hard-fails on a short glob. |
| |
| **Two theories we published, tested and refuted.** *"The lean-physique error comes from |
| low-resolution training images"* — tested directly on one DEXA-confirmed 7% subject, downscaled |
| 2160 → 300 px at three decode seeds per step: **flat at 15.5% throughout**, and wrong at full |
| resolution too. *"Lighting explains it"* — the same subject's two photographs, one 56% brighter, |
| read identically at every matched resolution. Both had been used to justify decisions. |
| |
| --- |
| |
| ## Superseded champions |
| |
| **v51 (2026-08-15) — the coach keeps the app's own numbers when a user contradicts them.** v48's |
| corpus + 216 rows. Told *"my readiness has been 95 all week"* on a day the app recorded **44**, the |
| previous champion agreed with the user. Data-conflict handling **16/32 → 26/32**, sized probe |
| 31/64 → 38/64, follow-up battery v3 13/28 → 16/28, visible defects 0 → 0, throughput 21.4 → 21.6 ch/s. |
| Its scorer was validated against 64 hand-read answers first (0 false alarms, 0 missed) — an earlier |
| version disagreed with human judgement on 16 of 64 and would have reported a confidently wrong number. |
| ⭐ **The finding worth reusing:** the corrective rows shipped with **48 "absence" examples in the same |
| batch**. Without them, teaching a model to speak authoritatively about what the data *contains* |
| generalises into speaking authoritatively about what it *lacks* — v50 used the same corrective rows |
| without them, scored HIGHER on the target, and was rejected for reporting a session the user had just |
| said they forgot to log. |
| ⚠️ v51 also established the triage rule this project now scopes rounds with: *a corrective behaviour |
| is trainable when the correct answer is printed in the prompt, and a base prior when the model must |
| supply the evidence itself.* Data-conflict moved +10; implausible-claim handling has not moved in |
| seven models. |
| |
| **v48 (2026-08-14) — the coach follows a conversation, not just a question.** v46's corpus + 229 |
| rows for carrying a number across turns and refusing to build advice on impossible claims. |
| Follow-up battery **8/24 → 16/24** (replicated at 13/24 on the second seed), visible defects 0 → 0, |
| body-fat read unchanged, **~7% slower** — a deliberate trade, reproduced on both seeds. It fixed |
| self-contradiction ("I'm cutting" then "since I'm bulking"), reference resolution ("double the first |
| number"), unlogged training, and 5 × 5 at 100 kg = 2,500 kg where the champion answered 250. Two of |
| those were safety cases: the old champion said "go for the PR tomorrow" one turn after the user |
| reported knee pain, and called a resting heart rate of 28 "a good sign for recovery". |
| ⚠️ The win was nearly credited to the wrong cause — it was first compared against a champion using a |
| *different export recipe*, two variables at once. Re-exporting the old corpus through the new recipe |
| scored 8/24, identical to the champion, which is what proved the gain came from the data. |
| |
| **v47 (2026-08-13) — same v46 weights; the export was showing the model half the photo.** |
| `litert_torch` defaults the Gemma-4 vision budget to **140 soft tokens** where Google's own config |
| specifies **280**, so every published vision build ran at **48.9% of the pixel area** while every |
| reference measurement was taken at 280. Fixing it lifted the male body-fat rank correlation |
| **+0.492 → +0.704** on 154 held-out photos (women unchanged at +0.74 — what separates male bands is |
| fine texture a few pixels tall, and downscaling is a low-pass filter). Cost: vision attention grows |
| with the square of the patch count, ~1.2× on meal photos and up to ~2.6× on large body-fat photos. |
| File size unchanged. |
| ⚠️ **Honest correction.** v47 was promoted on rendered images large enough to fill the patch budget |
| natively. Measured afterwards on **real photographs**, v47 and v46 are identical (MAE 7.89, lean bias |
| +7.17 for both). It is not a regression, but the improvement it was promoted for does not appear on |
| real photos. Fourteen training rounds had tried to teach this model to read male physiques; the |
| model could already do it, and the fault was a default nobody had set. |
|
|
| **v46 (2026-08-12)** eliminated avoid-list violations (0/36 vs 5/36) by matching the corpus to the |
| prompt format production actually sends — no new data, no app change. **v44** fixed non-English |
| quality in the data: 13 of 7,288 rows contained any Danish/Norwegian and ten demonstrated exactly |
| the wrong behaviour. Danish macros became "kulhydrater/fedt", challenge titles came back in Danish, |
| `/nutrition` went 5/7 → 7/7, throughput 22.6 → 23.6 ch/s. Its seed 123 was chosen over a |
| better-*scoring* seed 456 that opened a muscle-building plan with "Velkommen til vægttab" and |
| prescribed a deficit — invisible to the scorer, obvious on a read. |
|
|
| --- |
|
|
| ## Champion history |
|
|
| Every entry was a promoted champion, verified against the previous one on a held-out suite with a |
| full read of every answer. Roughly two rounds in three are rejected and never appear here; they are |
| documented internally with their real failure cases. |
|
|
| | Round | Beat | What it fixed | Base | |
| |---|---|---|---| |
| | **v27** | baseline | first winning fine-tune — shorter, equally accurate general advice | `gemma-4-E2B-it` | |
| | **v31** | v27 | exercise-form answers reached the untouched baseline's zero-error record. **The unlock was the export recipe, not data** — three data-only rounds failed first, then Hadamard-rotation int4 fixed it with zero retraining. A later export update on the same weights gave **88% faster decode** at unchanged quality | `gemma-4-E2B-it` | |
| | **v34** | v31 | dietary-rule compliance (a "vegetarian, no nuts" request had served peanut butter); first correct face-pull and plank examples | `gemma-4-E2B-it` | |
| | **v38** | v34 | AI challenges: unrealistic targets and non-English requests answered in English. Won over its own seed on a Danish plural defect | `gemma-4-E2B-it` | |
| | **v42** | v38 | QAT base + blockwise export + fenced-JSON data fix. Rotation was measured to *erase* QAT conditioning entirely (post-rotation RMSE 0.1288 vs stock 0.1287) | **QAT** | |
| | **v44** | v42 | Danish macro vocabulary and challenge titles; `/nutrition` emitted on every applicable case | **QAT** | |
| | **v46** | v44 | avoid-list violations eliminated by matching the corpus to production's prompt format | **QAT** | |
| | **v47** | v46 | vision export at the correct 280-token budget (same weights) | **QAT** | |
| | **v48** | v47 | conversational follow-ups: carried numbers, contradictions, two safety cases | **QAT** | |
| | **v51** | v48 | data-conflict: keeps the app's own figures when a user contradicts them | **QAT** | |
| | **v54** | v51 | reads the right day of the 7-day history line (`today_ref` 14/32 → 27/32, reproduced on both seeds) | **QAT** | |
|
|
| **Recipe constant since v22c:** LoRA r=16 / alpha=32 / dropout=0.1, 2 epochs, lr 6e-5, NEFTune |
| noise_alpha=5, completion-only loss masking, per-task validation shards, two seeds per round with |
| the winner chosen on behaviour rather than validation loss. Only the data, the base and the export |
| recipe have moved. |
| |
| --- |
| |
| ## Honest current weaknesses |
| |
| Documented rather than hidden, because they are the targets for the next rounds. |
| |
| - **The body-fat photo read barely depends on the photograph — and the cause we published before |
| was wrong.** The model answers ~12% for every subject from a true **7%** to a true **23%**. Earlier |
| versions of this card blamed image resolution. Measured 2026-08-17, that was a misdiagnosis, and the |
| real picture is three separate faults: |
| 1. **The reference labels were wrong.** They recorded what the image generator was *asked* for, not |
| what it produced. Three independent blind raters over all 129 corpus cells: asked-4% renders read |
| **11.3%**, asked-14% read **20.2%**, accurate only from ~16% up. Re-scored against corrected |
| labels the model reads lean bodies to **1.1 pp** and under-reads heavy ones by **11.2 pp** — the |
| opposite of the "reads lean subjects high" story. |
| 2. **The prompt was answering for the model.** It ended with *"a tape-measurement formula suggests |
| about N%"* plus an instruction to trust the photo instead. With that anchor fixed at 13%, |
| **20 of 28 images read exactly 13**. Removed — but removing it does not restore discrimination, |
| it makes the model a *two*-value reader, so the collapse is in the weights. |
| 3. **The reference chart compresses the read.** Dropping it lifts the usable span from 6 pp to |
| 14 pp and cuts error at the fat end from 13.8 pp to 8.3 pp, with no retraining. |
| Ruled out by direct measurement, so they are not the answer: **quantization** (a float vision tower |
| is *flatter* than int8, and bf16 is anti-ordered), **encoder capacity** (frozen probes give torso |
| 0.570 against a **face-crop control at 0.549**, which voids the result), and **cropping** (the app's |
| existing rules are already correct on both paths). A blind human ordering of the same lean images |
| scores **0.891** concordance where the model spans 1.1 pp — so the signal is in the pixels and this |
| is a model deficit, not missing data. |
| - **Implausible claims are still accepted 1 time in 4** — unchanged across five models and ~385 |
| corrective rows. Needs an app-side prompt fix, not more data. |
| - **Arithmetic across turns is unreliable.** Rows teaching computed answers taught the *sentence |
| template* rather than the maths; the app should inject computed values instead. |
| - **Meal-plan portions are sized by habit, not arithmetic** — the model anchors on ~100 g portions |
| rather than solving the stated calorie budget. Mostly absorbed by the app, which rescales a day to |
| its target (clamped 0.6×–2.0×). |
| - **Occasional Danish/Norwegian word-formation slips** ("vækning", "fat-mål"). Macro vocabulary is |
| correct; morphology is not reliable. |
| - **Nordic follow-ups about an unlogged past activity** are a base-model behaviour, not a regression |
| we introduced: the untouched base fails 30/32 where the fine-tune fails 28/32. Corrective rows |
| made it *worse*; it is being fixed in the app's prompt instead. |
| |
| ## Evaluation discipline |
| |
| Every promotion is judged on three axes — speed, answer quality and error rate — against the |
| previous champion re-run on the *same machine in the same session* (chars/sec is hardware-dependent |
| and cross-machine numbers are not comparable). Every answer of every suite is read, not sampled: two |
| champions in this lineage were chosen over better-*scoring* alternate seeds on defects no scorer |
| could see. Any metric with n < 30 is treated as a screen, not a result. |
| |
| ## Files |
| |
| | file | who gets it | size | |
| |---|---|---| |
| | **`smart-coach-vision.litertlm`** | ⭐ **current champion (v51)** — text coaching AND vision in one model. Fetched by current app releases. | 2.80 GB | |
| | `smart-coach.litertlm` | text-only, **v46 weights**, fetched by app builds already installed. Deliberately unchanged. | 2.63 GB | |
| | `coach.litertlm` | the original int8 build from the v1 era, kept as the historical starting point | 2.59 GB | |
| | `coach-finetuned-int4.litertlm` | the first int4 conversion — the artifact that exposed the LiteRT-LM chat-template `.get()` incompatibility | 2.56 GB | |
| | `model.safetensors` + tokenizer files | HF-format artifacts for reference | — | |
| |
| The app decides which file to fetch from its build version; there is nothing to select. The |
| consolidated vision build replaced what used to be two downloads (a 2.63 GB coach plus a separate |
| 3.66 GB Gemma-3n vision model, 6.29 GB together), runs vision on **CPU** where the standalone model |
| required a GPU, and removes the engine swap behind two shipped crash classes. Installed builds still |
| fetch the separate vision model, so handing *them* the larger consolidated file would raise their |
| total download on devices already tight on RAM — which is why the text-only file stays where it is. |
| |
