# Issue log One entry per bug fix. The issue ID also appears in the comment at the fix site and in the regression test. | ID | Date | Summary | Commit | |---|---|---|---| | ISSUE-001 | 2026-10-28 | Modal AuthError in eval.py when .env has empty token lines | `05ddaa4` | | ISSUE-002 | 2026-10-29 | Tuned v3 reads "make that {zone}" after a source flag as the source, and guesses | fix `333f6f5` (v4 data); probe `fbc723f`; **fixed**: v4 keeps the flag 8/8 (v3 0/8) | | ISSUE-003 | 2026-11-01 | Tuned v3 is an hour off on conversions between two DST zones | fix `3b00c18` (v5 data); probe `40fbdfd`; **open, partly improved**: 58 → 68 of 80, short of the 70 bar | ## ISSUE-001: Modal AuthError in eval.py when .env has empty token lines - **Issue:** all eight v3 evals (`python eval.py ...`) failed at startup with `modal.exception.AuthError` "Token missing", although `modal run train.py` had worked minutes earlier. Found when the v3 eval batch returned no results. - **Root cause:** eval.py imports extract.py before starting the Modal app, and extract.py calls `load_dotenv()`. `.env` (copied from `.env.example`) had empty `MODAL_TOKEN_ID=` and `MODAL_TOKEN_SECRET=` lines, so both variables were set to `""`, and Modal used them instead of the `modal setup` login in `~/.modal.toml`. - **Fix:** `drop_empty_modal_tokens` in [eval.py](../eval.py) removes empty `MODAL_TOKEN_*` variables before Modal starts; `.env.example` now comments those lines out. - **Test:** `tests/test_eval.py::test_empty_modal_tokens_are_removed_so_the_modal_setup_login_applies` (fails without the fix). By hand: with empty `MODAL_TOKEN_*` lines in `.env`, run `python eval.py --model debudash/qwen3-4b-tz-convert-v3-n1500 --eval-set cases/ceiling.jsonl`; it starts the Modal app instead of raising AuthError. ## ISSUE-002: tuned v3 reads "make that {zone}" after a source flag as the source, and guesses **Status: fixed in v4** (`debudash/qwen3-4b-tz-convert-v4-n12000` @ `50b411fe`). Diagnosed with a failing probe set (`fbc723f`); the fix is a v4 data change, trained with the same hyperparameters as every other run. - **Issue:** on a fresh grader-style conversation ([cases/probe_live_conversation.jsonl](../cases/probe_live_conversation.jsonl)), tuned v3-n12000 got 1 of 4 turns right and the base model 3 of 4. After a correct `AMBIGUOUS: source_timezone` for "Convert 2026-06-12 09:30 CST to Europe/London", the user said "Actually make that Asia/Tokyo instead." v3 replied `2026-06-12T01:30:00+01:00`: it took Asia/Tokyo as the *source* (09:30 Tokyo in London), kept the old target, and carried that reading through the next two turns. Found on 2026-10-29 while recording the demo video's live base-vs-tuned run ([results/probe/live_conversation/](../results/probe/live_conversation/)). - **Probe:** [cases/probe_target_change.jsonl](../cases/probe_target_change.jsonl), 24 scenarios, 60 turns. Each opens with a flagged source (CST, IST, or EST in July) and varies only the second turn's wording; every turn is labelled by the oracle and the judge agrees with exact match on 100% of turns. Results in [results/probe/target_change/](../results/probe/target_change/). Both probes ran from scratch copies under `runs/` with the same content, so the meta files record `runs/...` as the eval set path: | Second-turn wording (after the flag) | Base: flag kept | Tuned v3: flag kept | Tuned v3: new target applied (turn 3) | |---|---|---|---| | "And in {tgt}?" (a training phrasing) | 2/4 | 4/4 | 3/4 | | "Change the target to {tgt}." | 2/4 | 4/4 | 3/4 | | "Actually, convert it to {tgt} instead." | 2/4 | 4/4 | 3/4 | | "Actually make that {tgt} instead." | 2/4 | **0/4** | | | "Make that {tgt}." | 2/4 | **0/4** | | | "I meant {src}." (source clarification, control) | 0/4 converted | 2/4 converted | | When the wording names the role, v3 keeps the flag 12 of 12 times and applies the new target (the 3 turn-3 and 2 control misses are the known 1-hour arithmetic errors). When it doesn't, v3 guesses 8 of 8 times. Overall exact match on the probe: tuned 78.3%, base 36.7%. - **Root cause (data):** in v1-v3 training data, a zone named right after a source flag is always a source clarification (30 `clarify` templates, e.g. "Actually, the timezone is {value}", "Please go with {value}"). "Make that ..." appears only in a date template ("Actually, make that {date}"), where it replaces the value just discussed. No template changes a zone without saying which component it fills. So the model learned to fill the flagged slot by position, not by the words. The spec says a message that doesn't make clear which component a value fills is AMBIGUOUS. The drift eval missed this because its target changes all use role-explicit wording, so v3's flag-then-change failures there are all arithmetic. - **Fix (v4 data, `333f6f5`):** (1) a new teacher template kind for zone changes that don't state a role ("Make that {zone}.", "Use {zone} instead."), labelled by the oracle as AMBIGUOUS both after a flag and after a clean request (decided 2026-10-29: the spec as written, no spec change); (2) more role-explicit correction wordings for target changes, so "actually" and "instead" don't signal "source"; (3) contrast triples from one flagged opening (source clarification, explicit target change, change with no role) so the role must come from the words; (4) the probe's wordings excluded from training. As built ([gen_train.py](../gen_train.py) `role_change_example`, [teacher.py](../teacher.py)): 37 role-free and 38 explicit-target templates passed a role gate (the gate call must find the role unstated, or the target named). Templates on the wrong side of the line were dropped for v4: role-free templates that say "meant" or "correction" (those clarify the flagged zone, like the probe's "I meant {src}." control), and two old clarify templates that read as role-free swaps ("Please go with {value}."). One explicit-target template that nearly copied a probe wording was dropped too. [data/train_v4.jsonl](../data/train_v4.jsonl): 12,000 examples, seed 14, 934 role_change conversations (7.8%); v1-v3 still regenerate byte for byte. - **Frontier models on the same probe** (2026-11-01, best strategy = CoT with reasoning on, 2 repeats, [results/probe/target_change/frontier/](../results/probe/target_change/frontier/); judge agrees with exact match on 100% of turns): role-free wordings keep the flag 6/8 and 5/8 for Sonnet 5.5 and 4/8 and 5/8 for Mistral Large 4; role-explicit target changes 12/12 for both. Overall exact match: Sonnet 96.7% / 95.0%, Mistral 91.7% / 91.7%. So the frontier models guess here too, less often than v3 (0/8). - **Ship rule (fixed before training):** v4 replaces v3 as the headline checkpoint only if the probe passes (8/8 and 12/12) and ceiling and drift stay within noise of v3 (8 and 10 points). Otherwise v3 stays, and v4 is reported as a measured attempt. - **Test:** `python eval.py --model --eval-set cases/probe_target_change.jsonl`, then `python scripts/report_probe.py `. v3 fails it (0/8 on the no-role wordings). The fix passes when v4 keeps the flag on 8/8 no-role wordings and still 12/12 on the role-explicit ones. By hand: run [cases/probe_live_conversation.jsonl](../cases/probe_live_conversation.jsonl); turn 2 must be `AMBIGUOUS: source_timezone`. - **Result (2026-11-01):** v4-n12000 (43.0 min on an H100, same hyperparameters) passes the ship rule. | | v3 | v4 | |---|---|---| | Probe: role-free change keeps the flag | 0/8 | **8/8** | | Probe: role-explicit target change keeps the flag | 12/12 | **12/12** | | Probe: overall exact match | 78.3% | **86.7%** | | Ceiling: spec-adherence / robustness | 72.0% / 84.5% | 72.0% / 79.6% | | Drift: conversations held / turns correct | 75.7% / 91.8% | **80.0% / 93.8%** | Remaining probe misses are wrong hours on turn 3 (after the source is clarified), the same value sets under every wording, so arithmetic (ISSUE-003), not role reading. Cost of the fix: on the ceiling set, 3 multi-fault scenarios now report `AMBIGUOUS: source_timezone` before an invalid date or time (priority order), plausibly because every role-free change in the contrast sets is labelled `AMBIGUOUS: source_timezone`; the robustness drop (-4.9) is within noise and otherwise wrong hours. Results: [results/probe/target_change/v4-n12000/](../results/probe/target_change/v4-n12000/), [results/eval/v4-n12000/](../results/eval/v4-n12000/), [results/drift/v4-n12000/](../results/drift/v4-n12000/). ## ISSUE-003: tuned v3 is an hour off on conversions between two DST zones **Status: open, partly improved.** v5 (`debudash/qwen3-4b-tz-convert-v5-n18000` @ `f90209fc`) lifts the DST probe from 58/80 to 68/80, two turns short of the pre-registered bar of 70/80, so the issue is not closed. - **Issue:** the tuned model's main remaining failure is a wrong hour (15 of its 30 failed ceiling turns; most of its drift failures, where a wrong hour then carries forward). Example from the demo script, convert_iana-03: "2025-12-26 at 07:00 America/New_York - what time is that in Pacific/Auckland?" Expected `2025-12-27T01:00:00+13:00`; v3 replied `2025-12-27T00:00:00+13:00` (offset right, hour 1 h early). Found while reviewing the demo script on 2026-11-01. - **Diagnosis:** over v3's ceiling and drift transcripts, the 27 turns where a wrong hour first appears: 26 have the right target offset, 18 are exactly 1 h off. Conversions fail 6/19 when both zones observe DST, 6/41 when only the source does, and 8/189 when neither does. The model labels the target offset correctly but applies the wrong side of a DST switch to the hour, most often when both zones switch on different dates or in opposite seasons. - **Root cause (data):** conversions between two DST zones are a small, unstratified share of the data: zones are sampled independently, so DST/DST pairs from different families, on dates spread over the year, are rare per (pair, season) cell, and anchor pairs (UTC first) are not targeted at them. - **Probe (test first):** [cases/probe_dst_pairs.jsonl](../cases/probe_dst_pairs.jsonl) ([scripts/gen_probe_dst_pairs.py](../scripts/gen_probe_dst_pairs.py), seed 31): 80 single conversions between two DST zones of different families, 40 northern/southern (opposite seasons) and 40 US/EU, in the eval set's phrasings, oracle-labelled, excluded from training. Base (full spec prompt): 0/80. **Tuned v3: 58/80 (72.5%)** (opposite hemispheres 27/40, US/EU 31/40); 15 of its 22 misses are exactly 1 h off, with the offset right in all but one. So v5 needs at least 87.5% under the ship rule. v4 (ISSUE-002 fix only): 52/80 (65.0%), within noise of v3. Frontier models at their best prompt (CoT, reasoning on, 2 repeats, [results/probe/dst_pairs/frontier/](../results/probe/dst_pairs/frontier/)): Sonnet 5.5 75/80 and 74/80, Mistral Large 4 79/80 and 80/80. Arithmetic is where reasoning helps them most. - **Fix (v5 data, same hyperparameters):** `arith_example` in [gen_train.py](../gen_train.py) samples DST-zone pairs stratified by family (70% different families, 15% same family, 15% one side without DST) on dates spread over 2020-2030, half as anchor pairs that show the UTC value first. v5 = the v4 mix at 2/3 weight plus arith at 1/3, N=18,000 (seed 15), so the v4 categories keep about their v4 counts. 5,867 arith conversations; 0 dropped for eval overlap; v1-v4 still regenerate byte for byte. - **Ship rule (fixed before training):** v5 becomes the headline checkpoint only if (1) the DST probe improves on v3 by at least 15 points (80 turns: about 2 standard errors is 11 points), (2) the ISSUE-002 probe still passes (8/8 and 12/12), and (3) ceiling spec-adherence and robustness and drift conversations held stay within noise (8 / 8 / 10 points) of the better of v3 and v4. Otherwise the headline is the latest version that passes its own rule, and v5 is reported as a measured attempt. - **Test:** `python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_dst_pairs.jsonl`. - **Result (2026-11-01):** v5-n18000 (78.0 min on an H100, same hyperparameters, final loss 0.0044). | | v3 | v4 | v5 | Ship rule | |---|---|---|---|---| | DST probe | 58/80 | 52/80 | **68/80** (opposite hemispheres 33/40, US/EU 35/40) | needs 70/80: **missed by 2** | | ISSUE-002 probe: role-free / role-explicit | 0/8, 12/12 | 8/8, 12/12 | **8/8, 12/12** | passes | | Ceiling: spec-adherence / robustness | 72.0% / 84.5% | 72.0% / 79.6% | 72.0% / **87.4%** | passes | | Drift: held / turns correct | 75.7% / 91.8% | 80.0% / 93.8% | **91.4% / 98.1%** | passes | 10 of v5's 12 probe misses are still exactly 1 h off, and convert_iana-03 (New York to Auckland) is still 1 h early. Clean single-turn spec-adherence did not move (72.0%), so the gain shows on DST pairs and in conversations (drift wrong hours 33 → 5) more than on the clean ceiling set. - **Decision (user, 2026-11-01):** the literal rule makes v4 the headline, but v5 is equal or better than v4 on every measure, so **v5 is the final model**. ISSUE-003 stays open: the bar set before training was not met and is not moved. Next step: more arithmetic data per (pair, season) cell, or let code do the arithmetic (the model outputs the structured request, the oracle computes the time). Results: [results/probe/dst_pairs/v5-n18000/](../results/probe/dst_pairs/v5-n18000/), [results/eval/v5-n18000/](../results/eval/v5-n18000/), [results/drift/v5-n18000/](../results/drift/v5-n18000/), [results/probe/target_change/v5-n18000/](../results/probe/target_change/v5-n18000/).