card: v54 champion; v51 squished into superseded; body-fat weakness corrected

15911af verified about 10 hours ago

17.6 kB

license: cc0-1.0
base_model: google/gemma-4-E2B-it-qat-q4_0-unquantized
base_model_relation: finetune
tags:
  - lora
  - fine-tune
  - litert-lm
  - on-device
  - fitness
  - quantization-aware-training
  - int4

SmartSight Coach LoRA fine-tunes

LoRA fine-tunes for the SmartSight AI coach — an on-device fitness and nutrition coach that runs entirely on the phone, no account and no server round-trip. Converted to .litertlm for Android/iOS inference. → smartsight.app/ai-coach

Built and maintained by Niclas Bade.

Base model changed at v42. v27–v38 came from google/gemma-4-E2B-it; everything from v42 on is fine-tuned from Google's quantization-aware-trained checkpoint google/gemma-4-E2B-it-qat-q4_0-unquantized, whose weights are conditioned to survive 4-bit rounding. This repo keeps the current champion plus superseded milestones. Rounds that lose are documented internally with their real failure cases; they are never uploaded here.

`smart-coach-vision.litertlm` — CURRENT CHAMPION (v54)

Promoted 2026-08-17. One merged artifact: the coach text LoRA and the vision tower in a single 2.80 GB file, int4 (blockwise-64) on the decoder's fully-connected layers, int8 (dynamic_wi8_afp32) on the vision encoder.

What v54 fixed: the coach reading the wrong cell of the 7-day training-history line.

Asked "what did I train yesterday?" against a snapshot that prints the last seven days oldest-to-newest, the previous champion read the wrong day 18 times out of 32 — usually answering about today and declaring yesterday empty. v54 adds 629 rows that ask about today and yesterday in the same breath, plus 141 rows teaching where the seven-day window stops so the new confidence does not walk over the edge and invent a session from two weeks ago.

gate	v51 champion	v54 seed 456	v54 seed 123
`today_ref` — reads the right day	14/32	27/32	21/32
`outside_window` — refuses to invent beyond 7 days	32/32	32/32	32/32
day-line probe total (n=128)	103 (range 103–110)	110	115
sized probe (n=64)	38/64	50/64	42/64
— of which `data_conflict`	26/32	31/32	28/32
follow-up battery v3 (n=28)	16/28	16/28	18/28
user-visible defects (n=89)	0	0	1
decode speed	21.6 ch/s	21.2	21.4

Why this round is trustworthy where its predecessor was not. v53 attempted the same target with 202 rows and was rejected: today_ref scored 29/32 on one seed and 5/32 on the other from an identical corpus. Before promoting anything we measured the champion's own decode-noise floor — the same weights scored at four decode seeds — and found the day-line probe swings 103 / 110 / 106 / 109. Two consequences:

today_ref has a noise floor of 2 points, so v54's +13 and +7 are real movement, while v53's 29-vs-5 was sampling.
The single control transcript every previous comparison used is the champion's worst draw. Challengers were being judged against a low outlier. v54 is compared against the range.

Costs, stated honestly. Follow-up battery v2 drops 16/24 → 15/24, and 14/24 on the second seed — it regresses on both, so it is the corpus rather than noise. A recovery-anchor probe gains one violation in five on both seeds; that defect has now appeared in four models across two corpora and is caused by the day-line rows. Answers are about 1% slower. Two seed-456 numbers did not reproduce on seed 123 and are recorded as unstable rather than costs: day_absent 29 vs 32, day_logged 22 vs 30.

Body-fat photo read: unchanged where it matters. Paired on the exported artifact against corrected labels, inside the range this app's users occupy (true ≤25%) mean absolute error moves 3.28 → 3.33 with 1 of 18 images changing. Above 25% it degrades (14.9 → 17.1) — real, and outside the product range, but recorded because under-reading a heavy user is the dangerous direction.

Method: instruments we found broken

Published because a score is only worth what the scorer is worth, and several of ours were worth nothing.

The main quality scorer read 0 defects out of 89 for a model failing 16 of 24 conversational cases — it only detects timeouts, empty replies and unparseable JSON. Confidently wrong coaching in well-formed prose is invisible to it.
The secondary scorer was wrong on all 3 flags it raised and missed all 16 real failures, including a 10× arithmetic error and a dangerous medical endorsement.
A red-flag rule penalised correct behaviour. It fired on English macro words in Nordic text while also scanning structured meal lines, where the app's own parser requires those English keywords — 17 false positives in every round the rule had ever run.
Our replacement scorer was too lenient three times, then too strict once. Literal phrase lists cannot survive paraphrase; pattern checks that flag a banned word cannot tell "drop the dips" from programming them. Tightening it lowered a candidate's own score from 17/24 to a verified 16/24 — we published the lower number.
A cleanup script deleted the artifacts it was cleaning up after, wiping a control model mid-comparison and returning 89 empty answers as a clean run. Automatic deletion was removed.
An evaluation silently measured nothing — a path bug matched zero images, both arms reported "done" in seconds and the scorer printed a tidy table. It now hard-fails on a short glob.

Two theories we published, tested and refuted. "The lean-physique error comes from low-resolution training images" — tested directly on one DEXA-confirmed 7% subject, downscaled 2160 → 300 px at three decode seeds per step: flat at 15.5% throughout, and wrong at full resolution too. "Lighting explains it" — the same subject's two photographs, one 56% brighter, read identically at every matched resolution. Both had been used to justify decisions.

Superseded champions

v51 (2026-08-15) — the coach keeps the app's own numbers when a user contradicts them. v48's corpus + 216 rows. Told "my readiness has been 95 all week" on a day the app recorded 44, the previous champion agreed with the user. Data-conflict handling 16/32 → 26/32, sized probe 31/64 → 38/64, follow-up battery v3 13/28 → 16/28, visible defects 0 → 0, throughput 21.4 → 21.6 ch/s. Its scorer was validated against 64 hand-read answers first (0 false alarms, 0 missed) — an earlier version disagreed with human judgement on 16 of 64 and would have reported a confidently wrong number. ⭐ The finding worth reusing: the corrective rows shipped with 48 "absence" examples in the same batch. Without them, teaching a model to speak authoritatively about what the data contains generalises into speaking authoritatively about what it lacks — v50 used the same corrective rows without them, scored HIGHER on the target, and was rejected for reporting a session the user had just said they forgot to log. ⚠️ v51 also established the triage rule this project now scopes rounds with: a corrective behaviour is trainable when the correct answer is printed in the prompt, and a base prior when the model must supply the evidence itself. Data-conflict moved +10; implausible-claim handling has not moved in seven models.

v48 (2026-08-14) — the coach follows a conversation, not just a question. v46's corpus + 229 rows for carrying a number across turns and refusing to build advice on impossible claims. Follow-up battery 8/24 → 16/24 (replicated at 13/24 on the second seed), visible defects 0 → 0, body-fat read unchanged, ~7% slower — a deliberate trade, reproduced on both seeds. It fixed self-contradiction ("I'm cutting" then "since I'm bulking"), reference resolution ("double the first number"), unlogged training, and 5 × 5 at 100 kg = 2,500 kg where the champion answered 250. Two of those were safety cases: the old champion said "go for the PR tomorrow" one turn after the user reported knee pain, and called a resting heart rate of 28 "a good sign for recovery". ⚠️ The win was nearly credited to the wrong cause — it was first compared against a champion using a different export recipe, two variables at once. Re-exporting the old corpus through the new recipe scored 8/24, identical to the champion, which is what proved the gain came from the data.

v47 (2026-08-13) — same v46 weights; the export was showing the model half the photo. litert_torch defaults the Gemma-4 vision budget to 140 soft tokens where Google's own config specifies 280, so every published vision build ran at 48.9% of the pixel area while every reference measurement was taken at 280. Fixing it lifted the male body-fat rank correlation +0.492 → +0.704 on 154 held-out photos (women unchanged at +0.74 — what separates male bands is fine texture a few pixels tall, and downscaling is a low-pass filter). Cost: vision attention grows with the square of the patch count, ~1.2× on meal photos and up to ~2.6× on large body-fat photos. File size unchanged. ⚠️ Honest correction. v47 was promoted on rendered images large enough to fill the patch budget natively. Measured afterwards on real photographs, v47 and v46 are identical (MAE 7.89, lean bias +7.17 for both). It is not a regression, but the improvement it was promoted for does not appear on real photos. Fourteen training rounds had tried to teach this model to read male physiques; the model could already do it, and the fault was a default nobody had set.

v46 (2026-08-12) eliminated avoid-list violations (0/36 vs 5/36) by matching the corpus to the prompt format production actually sends — no new data, no app change. v44 fixed non-English quality in the data: 13 of 7,288 rows contained any Danish/Norwegian and ten demonstrated exactly the wrong behaviour. Danish macros became "kulhydrater/fedt", challenge titles came back in Danish, /nutrition went 5/7 → 7/7, throughput 22.6 → 23.6 ch/s. Its seed 123 was chosen over a better-scoring seed 456 that opened a muscle-building plan with "Velkommen til vægttab" and prescribed a deficit — invisible to the scorer, obvious on a read.

Champion history

Every entry was a promoted champion, verified against the previous one on a held-out suite with a full read of every answer. Roughly two rounds in three are rejected and never appear here; they are documented internally with their real failure cases.

Round	Beat	What it fixed	Base
v27	baseline	first winning fine-tune — shorter, equally accurate general advice	`gemma-4-E2B-it`
v31	v27	exercise-form answers reached the untouched baseline's zero-error record. The unlock was the export recipe, not data — three data-only rounds failed first, then Hadamard-rotation int4 fixed it with zero retraining. A later export update on the same weights gave 88% faster decode at unchanged quality	`gemma-4-E2B-it`
v34	v31	dietary-rule compliance (a "vegetarian, no nuts" request had served peanut butter); first correct face-pull and plank examples	`gemma-4-E2B-it`
v38	v34	AI challenges: unrealistic targets and non-English requests answered in English. Won over its own seed on a Danish plural defect	`gemma-4-E2B-it`
v42	v38	QAT base + blockwise export + fenced-JSON data fix. Rotation was measured to erase QAT conditioning entirely (post-rotation RMSE 0.1288 vs stock 0.1287)	QAT
v44	v42	Danish macro vocabulary and challenge titles; `/nutrition` emitted on every applicable case	QAT
v46	v44	avoid-list violations eliminated by matching the corpus to production's prompt format	QAT
v47	v46	vision export at the correct 280-token budget (same weights)	QAT
v48	v47	conversational follow-ups: carried numbers, contradictions, two safety cases	QAT
v51	v48	data-conflict: keeps the app's own figures when a user contradicts them	QAT
v54	v51	reads the right day of the 7-day history line (`today_ref` 14/32 → 27/32, reproduced on both seeds)	QAT

Recipe constant since v22c: LoRA r=16 / alpha=32 / dropout=0.1, 2 epochs, lr 6e-5, NEFTune noise_alpha=5, completion-only loss masking, per-task validation shards, two seeds per round with the winner chosen on behaviour rather than validation loss. Only the data, the base and the export recipe have moved.

Honest current weaknesses

Documented rather than hidden, because they are the targets for the next rounds.

The body-fat photo read barely depends on the photograph — and the cause we published before was wrong. The model answers ~12% for every subject from a true 7% to a true 23%. Earlier versions of this card blamed image resolution. Measured 2026-08-17, that was a misdiagnosis, and the real picture is three separate faults:
1. The reference labels were wrong. They recorded what the image generator was asked for, not what it produced. Three independent blind raters over all 129 corpus cells: asked-4% renders read 11.3%, asked-14% read 20.2%, accurate only from ~16% up. Re-scored against corrected labels the model reads lean bodies to 1.1 pp and under-reads heavy ones by 11.2 pp — the opposite of the "reads lean subjects high" story.
2. The prompt was answering for the model. It ended with "a tape-measurement formula suggests about N%" plus an instruction to trust the photo instead. With that anchor fixed at 13%, 20 of 28 images read exactly 13. Removed — but removing it does not restore discrimination, it makes the model a two-value reader, so the collapse is in the weights.
3. The reference chart compresses the read. Dropping it lifts the usable span from 6 pp to 14 pp and cuts error at the fat end from 13.8 pp to 8.3 pp, with no retraining. Ruled out by direct measurement, so they are not the answer: quantization (a float vision tower is flatter than int8, and bf16 is anti-ordered), encoder capacity (frozen probes give torso 0.570 against a face-crop control at 0.549, which voids the result), and cropping (the app's existing rules are already correct on both paths). A blind human ordering of the same lean images scores 0.891 concordance where the model spans 1.1 pp — so the signal is in the pixels and this is a model deficit, not missing data.
Implausible claims are still accepted 1 time in 4 — unchanged across five models and ~385 corrective rows. Needs an app-side prompt fix, not more data.
Arithmetic across turns is unreliable. Rows teaching computed answers taught the sentence template rather than the maths; the app should inject computed values instead.
Meal-plan portions are sized by habit, not arithmetic — the model anchors on ~100 g portions rather than solving the stated calorie budget. Mostly absorbed by the app, which rescales a day to its target (clamped 0.6×–2.0×).
Occasional Danish/Norwegian word-formation slips ("vækning", "fat-mål"). Macro vocabulary is correct; morphology is not reliable.
Nordic follow-ups about an unlogged past activity are a base-model behaviour, not a regression we introduced: the untouched base fails 30/32 where the fine-tune fails 28/32. Corrective rows made it worse; it is being fixed in the app's prompt instead.

Evaluation discipline

Every promotion is judged on three axes — speed, answer quality and error rate — against the previous champion re-run on the same machine in the same session (chars/sec is hardware-dependent and cross-machine numbers are not comparable). Every answer of every suite is read, not sampled: two champions in this lineage were chosen over better-scoring alternate seeds on defects no scorer could see. Any metric with n < 30 is treated as a screen, not a result.

Files

file	who gets it	size
`smart-coach-vision.litertlm`	⭐ current champion (v51) — text coaching AND vision in one model. Fetched by current app releases.	2.80 GB
`smart-coach.litertlm`	text-only, v46 weights, fetched by app builds already installed. Deliberately unchanged.	2.63 GB
`coach.litertlm`	the original int8 build from the v1 era, kept as the historical starting point	2.59 GB
`coach-finetuned-int4.litertlm`	the first int4 conversion — the artifact that exposed the LiteRT-LM chat-template `.get()` incompatibility	2.56 GB
`model.safetensors` + tokenizer files	HF-format artifacts for reference	—

The app decides which file to fetch from its build version; there is nothing to select. The consolidated vision build replaced what used to be two downloads (a 2.63 GB coach plus a separate 3.66 GB Gemma-3n vision model, 6.29 GB together), runs vision on CPU where the standalone model required a GPU, and removes the engine swap behind two shipped crash classes. Installed builds still fetch the separate vision model, so handing them the larger consolidated file would raise their total download on devices already tight on RAM — which is why the text-only file stays where it is.