slm-training / docs /ISSUE_LOG.md
debudash's picture
Mirror slm-training @ 507b432 (course PDF excluded)
dc23a98 verified
|
Raw History Blame Contribute Delete
13.9 kB

Issue log

One entry per bug fix. The issue ID also appears in the comment at the fix site and in the regression test.

ID Date Summary Commit
ISSUE-001 2026-10-28 Modal AuthError in eval.py when .env has empty token lines 05ddaa4
ISSUE-002 2026-10-29 Tuned v3 reads "make that {zone}" after a source flag as the source, and guesses fix 333f6f5 (v4 data); probe fbc723f; fixed: v4 keeps the flag 8/8 (v3 0/8)
ISSUE-003 2026-11-01 Tuned v3 is an hour off on conversions between two DST zones fix 3b00c18 (v5 data); probe 40fbdfd; open, partly improved: 58 → 68 of 80, short of the 70 bar

ISSUE-001: Modal AuthError in eval.py when .env has empty token lines

  • Issue: all eight v3 evals (python eval.py ...) failed at startup with modal.exception.AuthError "Token missing", although modal run train.py had worked minutes earlier. Found when the v3 eval batch returned no results.
  • Root cause: eval.py imports extract.py before starting the Modal app, and extract.py calls load_dotenv(). .env (copied from .env.example) had empty MODAL_TOKEN_ID= and MODAL_TOKEN_SECRET= lines, so both variables were set to "", and Modal used them instead of the modal setup login in ~/.modal.toml.
  • Fix: drop_empty_modal_tokens in eval.py removes empty MODAL_TOKEN_* variables before Modal starts; .env.example now comments those lines out.
  • Test: tests/test_eval.py::test_empty_modal_tokens_are_removed_so_the_modal_setup_login_applies (fails without the fix). By hand: with empty MODAL_TOKEN_* lines in .env, run python eval.py --model debudash/qwen3-4b-tz-convert-v3-n1500 --eval-set cases/ceiling.jsonl; it starts the Modal app instead of raising AuthError.

ISSUE-002: tuned v3 reads "make that {zone}" after a source flag as the source, and guesses

Status: fixed in v4 (debudash/qwen3-4b-tz-convert-v4-n12000 @ 50b411fe). Diagnosed with a failing probe set (fbc723f); the fix is a v4 data change, trained with the same hyperparameters as every other run.

  • Issue: on a fresh grader-style conversation (cases/probe_live_conversation.jsonl), tuned v3-n12000 got 1 of 4 turns right and the base model 3 of 4. After a correct AMBIGUOUS: source_timezone for "Convert 2026-06-12 09:30 CST to Europe/London", the user said "Actually make that Asia/Tokyo instead." v3 replied 2026-06-12T01:30:00+01:00: it took Asia/Tokyo as the source (09:30 Tokyo in London), kept the old target, and carried that reading through the next two turns. Found on 2026-10-29 while recording the demo video's live base-vs-tuned run (results/probe/live_conversation/).

  • Probe: cases/probe_target_change.jsonl, 24 scenarios, 60 turns. Each opens with a flagged source (CST, IST, or EST in July) and varies only the second turn's wording; every turn is labelled by the oracle and the judge agrees with exact match on 100% of turns. Results in results/probe/target_change/. Both probes ran from scratch copies under runs/ with the same content, so the meta files record runs/... as the eval set path:

    Second-turn wording (after the flag) Base: flag kept Tuned v3: flag kept Tuned v3: new target applied (turn 3)
    "And in {tgt}?" (a training phrasing) 2/4 4/4 3/4
    "Change the target to {tgt}." 2/4 4/4 3/4
    "Actually, convert it to {tgt} instead." 2/4 4/4 3/4
    "Actually make that {tgt} instead." 2/4 0/4
    "Make that {tgt}." 2/4 0/4
    "I meant {src}." (source clarification, control) 0/4 converted 2/4 converted

    When the wording names the role, v3 keeps the flag 12 of 12 times and applies the new target (the 3 turn-3 and 2 control misses are the known 1-hour arithmetic errors). When it doesn't, v3 guesses 8 of 8 times. Overall exact match on the probe: tuned 78.3%, base 36.7%.

  • Root cause (data): in v1-v3 training data, a zone named right after a source flag is always a source clarification (30 clarify templates, e.g. "Actually, the timezone is {value}", "Please go with {value}"). "Make that ..." appears only in a date template ("Actually, make that {date}"), where it replaces the value just discussed. No template changes a zone without saying which component it fills. So the model learned to fill the flagged slot by position, not by the words. The spec says a message that doesn't make clear which component a value fills is AMBIGUOUS. The drift eval missed this because its target changes all use role-explicit wording, so v3's flag-then-change failures there are all arithmetic.

  • Fix (v4 data, 333f6f5): (1) a new teacher template kind for zone changes that don't state a role ("Make that {zone}.", "Use {zone} instead."), labelled by the oracle as AMBIGUOUS both after a flag and after a clean request (decided 2026-10-29: the spec as written, no spec change); (2) more role-explicit correction wordings for target changes, so "actually" and "instead" don't signal "source"; (3) contrast triples from one flagged opening (source clarification, explicit target change, change with no role) so the role must come from the words; (4) the probe's wordings excluded from training. As built (gen_train.py role_change_example, teacher.py): 37 role-free and 38 explicit-target templates passed a role gate (the gate call must find the role unstated, or the target named). Templates on the wrong side of the line were dropped for v4: role-free templates that say "meant" or "correction" (those clarify the flagged zone, like the probe's "I meant {src}." control), and two old clarify templates that read as role-free swaps ("Please go with {value}."). One explicit-target template that nearly copied a probe wording was dropped too. data/train_v4.jsonl: 12,000 examples, seed 14, 934 role_change conversations (7.8%); v1-v3 still regenerate byte for byte.

  • Frontier models on the same probe (2026-11-01, best strategy = CoT with reasoning on, 2 repeats, results/probe/target_change/frontier/; judge agrees with exact match on 100% of turns): role-free wordings keep the flag 6/8 and 5/8 for Sonnet 5.5 and 4/8 and 5/8 for Mistral Large 4; role-explicit target changes 12/12 for both. Overall exact match: Sonnet 96.7% / 95.0%, Mistral 91.7% / 91.7%. So the frontier models guess here too, less often than v3 (0/8).

  • Ship rule (fixed before training): v4 replaces v3 as the headline checkpoint only if the probe passes (8/8 and 12/12) and ceiling and drift stay within noise of v3 (8 and 10 points). Otherwise v3 stays, and v4 is reported as a measured attempt.

  • Test: python eval.py --model <checkpoint> --eval-set cases/probe_target_change.jsonl, then python scripts/report_probe.py <base results> <tuned results>. v3 fails it (0/8 on the no-role wordings). The fix passes when v4 keeps the flag on 8/8 no-role wordings and still 12/12 on the role-explicit ones. By hand: run cases/probe_live_conversation.jsonl; turn 2 must be AMBIGUOUS: source_timezone.

  • Result (2026-11-01): v4-n12000 (43.0 min on an H100, same hyperparameters) passes the ship rule.

    v3 v4
    Probe: role-free change keeps the flag 0/8 8/8
    Probe: role-explicit target change keeps the flag 12/12 12/12
    Probe: overall exact match 78.3% 86.7%
    Ceiling: spec-adherence / robustness 72.0% / 84.5% 72.0% / 79.6%
    Drift: conversations held / turns correct 75.7% / 91.8% 80.0% / 93.8%

    Remaining probe misses are wrong hours on turn 3 (after the source is clarified), the same value sets under every wording, so arithmetic (ISSUE-003), not role reading. Cost of the fix: on the ceiling set, 3 multi-fault scenarios now report AMBIGUOUS: source_timezone before an invalid date or time (priority order), plausibly because every role-free change in the contrast sets is labelled AMBIGUOUS: source_timezone; the robustness drop (-4.9) is within noise and otherwise wrong hours. Results: results/probe/target_change/v4-n12000/, results/eval/v4-n12000/, results/drift/v4-n12000/.

ISSUE-003: tuned v3 is an hour off on conversions between two DST zones

Status: open, partly improved. v5 (debudash/qwen3-4b-tz-convert-v5-n18000 @ f90209fc) lifts the DST probe from 58/80 to 68/80, two turns short of the pre-registered bar of 70/80, so the issue is not closed.

  • Issue: the tuned model's main remaining failure is a wrong hour (15 of its 30 failed ceiling turns; most of its drift failures, where a wrong hour then carries forward). Example from the demo script, convert_iana-03: "2025-12-26 at 07:00 America/New_York - what time is that in Pacific/Auckland?" Expected 2025-12-27T01:00:00+13:00; v3 replied 2025-12-27T00:00:00+13:00 (offset right, hour 1 h early). Found while reviewing the demo script on 2026-11-01.

  • Diagnosis: over v3's ceiling and drift transcripts, the 27 turns where a wrong hour first appears: 26 have the right target offset, 18 are exactly 1 h off. Conversions fail 6/19 when both zones observe DST, 6/41 when only the source does, and 8/189 when neither does. The model labels the target offset correctly but applies the wrong side of a DST switch to the hour, most often when both zones switch on different dates or in opposite seasons.

  • Root cause (data): conversions between two DST zones are a small, unstratified share of the data: zones are sampled independently, so DST/DST pairs from different families, on dates spread over the year, are rare per (pair, season) cell, and anchor pairs (UTC first) are not targeted at them.

  • Probe (test first): cases/probe_dst_pairs.jsonl (scripts/gen_probe_dst_pairs.py, seed 31): 80 single conversions between two DST zones of different families, 40 northern/southern (opposite seasons) and 40 US/EU, in the eval set's phrasings, oracle-labelled, excluded from training. Base (full spec prompt): 0/80. Tuned v3: 58/80 (72.5%) (opposite hemispheres 27/40, US/EU 31/40); 15 of its 22 misses are exactly 1 h off, with the offset right in all but one. So v5 needs at least 87.5% under the ship rule. v4 (ISSUE-002 fix only): 52/80 (65.0%), within noise of v3. Frontier models at their best prompt (CoT, reasoning on, 2 repeats, results/probe/dst_pairs/frontier/): Sonnet 5.5 75/80 and 74/80, Mistral Large 4 79/80 and 80/80. Arithmetic is where reasoning helps them most.

  • Fix (v5 data, same hyperparameters): arith_example in gen_train.py samples DST-zone pairs stratified by family (70% different families, 15% same family, 15% one side without DST) on dates spread over 2020-2030, half as anchor pairs that show the UTC value first. v5 = the v4 mix at 2/3 weight plus arith at 1/3, N=18,000 (seed 15), so the v4 categories keep about their v4 counts. 5,867 arith conversations; 0 dropped for eval overlap; v1-v4 still regenerate byte for byte.

  • Ship rule (fixed before training): v5 becomes the headline checkpoint only if (1) the DST probe improves on v3 by at least 15 points (80 turns: about 2 standard errors is 11 points), (2) the ISSUE-002 probe still passes (8/8 and 12/12), and (3) ceiling spec-adherence and robustness and drift conversations held stay within noise (8 / 8 / 10 points) of the better of v3 and v4. Otherwise the headline is the latest version that passes its own rule, and v5 is reported as a measured attempt.

  • Test: python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_dst_pairs.jsonl.

  • Result (2026-11-01): v5-n18000 (78.0 min on an H100, same hyperparameters, final loss 0.0044).

    v3 v4 v5 Ship rule
    DST probe 58/80 52/80 68/80 (opposite hemispheres 33/40, US/EU 35/40) needs 70/80: missed by 2
    ISSUE-002 probe: role-free / role-explicit 0/8, 12/12 8/8, 12/12 8/8, 12/12 passes
    Ceiling: spec-adherence / robustness 72.0% / 84.5% 72.0% / 79.6% 72.0% / 87.4% passes
    Drift: held / turns correct 75.7% / 91.8% 80.0% / 93.8% 91.4% / 98.1% passes

    10 of v5's 12 probe misses are still exactly 1 h off, and convert_iana-03 (New York to Auckland) is still 1 h early. Clean single-turn spec-adherence did not move (72.0%), so the gain shows on DST pairs and in conversations (drift wrong hours 33 → 5) more than on the clean ceiling set.

  • Decision (user, 2026-11-01): the literal rule makes v4 the headline, but v5 is equal or better than v4 on every measure, so v5 is the final model. ISSUE-003 stays open: the bar set before training was not met and is not moved. Next step: more arithmetic data per (pair, season) cell, or let code do the arithmetic (the model outputs the structured request, the oracle computes the time). Results: results/probe/dst_pairs/v5-n18000/, results/eval/v5-n18000/, results/drift/v5-n18000/, results/probe/target_change/v5-n18000/.