Download docs/ISSUE_LOG.md from debudash/slm-training: direct link, hf CLI and curl.
- Browser
- Download file 13.9 kB
-
https://huggingface.co/debudash/slm-training/resolve/main/docs/ISSUE_LOG.md
- Command line
-
hf download hf://debudash/slm-training/docs/ISSUE_LOG.md
-
curl -L -o ISSUE_LOG.md https://huggingface.co/debudash/slm-training/resolve/main/docs/ISSUE_LOG.md
Issue log
One entry per bug fix. The issue ID also appears in the comment at the fix site and in the regression test.
| ID | Date | Summary | Commit |
|---|---|---|---|
| ISSUE-001 | 2026-10-28 | Modal AuthError in eval.py when .env has empty token lines | 05ddaa4 |
| ISSUE-002 | 2026-10-29 | Tuned v3 reads "make that {zone}" after a source flag as the source, and guesses | fix 333f6f5 (v4 data); probe fbc723f; fixed: v4 keeps the flag 8/8 (v3 0/8) |
| ISSUE-003 | 2026-11-01 | Tuned v3 is an hour off on conversions between two DST zones | fix 3b00c18 (v5 data); probe 40fbdfd; open, partly improved: 58 → 68 of 80, short of the 70 bar |
ISSUE-001: Modal AuthError in eval.py when .env has empty token lines
- Issue: all eight v3 evals (
python eval.py ...) failed at startup withmodal.exception.AuthError"Token missing", althoughmodal run train.pyhad worked minutes earlier. Found when the v3 eval batch returned no results. - Root cause: eval.py imports extract.py before starting the Modal app, and extract.py calls
load_dotenv()..env(copied from.env.example) had emptyMODAL_TOKEN_ID=andMODAL_TOKEN_SECRET=lines, so both variables were set to"", and Modal used them instead of themodal setuplogin in~/.modal.toml. - Fix:
drop_empty_modal_tokensin eval.py removes emptyMODAL_TOKEN_*variables before Modal starts;.env.examplenow comments those lines out. - Test:
tests/test_eval.py::test_empty_modal_tokens_are_removed_so_the_modal_setup_login_applies(fails without the fix). By hand: with emptyMODAL_TOKEN_*lines in.env, runpython eval.py --model debudash/qwen3-4b-tz-convert-v3-n1500 --eval-set cases/ceiling.jsonl; it starts the Modal app instead of raising AuthError.
ISSUE-002: tuned v3 reads "make that {zone}" after a source flag as the source, and guesses
Status: fixed in v4 (debudash/qwen3-4b-tz-convert-v4-n12000 @ 50b411fe). Diagnosed with a failing probe set (fbc723f);
the fix is a v4 data change, trained with the same hyperparameters as every other run.
Issue: on a fresh grader-style conversation (cases/probe_live_conversation.jsonl), tuned v3-n12000 got 1 of 4 turns right and the base model 3 of 4. After a correct
AMBIGUOUS: source_timezonefor "Convert 2026-06-12 09:30 CST to Europe/London", the user said "Actually make that Asia/Tokyo instead." v3 replied2026-06-12T01:30:00+01:00: it took Asia/Tokyo as the source (09:30 Tokyo in London), kept the old target, and carried that reading through the next two turns. Found on 2026-10-29 while recording the demo video's live base-vs-tuned run (results/probe/live_conversation/).Probe: cases/probe_target_change.jsonl, 24 scenarios, 60 turns. Each opens with a flagged source (CST, IST, or EST in July) and varies only the second turn's wording; every turn is labelled by the oracle and the judge agrees with exact match on 100% of turns. Results in results/probe/target_change/. Both probes ran from scratch copies under
runs/with the same content, so the meta files recordruns/...as the eval set path:Second-turn wording (after the flag) Base: flag kept Tuned v3: flag kept Tuned v3: new target applied (turn 3) "And in {tgt}?" (a training phrasing) 2/4 4/4 3/4 "Change the target to {tgt}." 2/4 4/4 3/4 "Actually, convert it to {tgt} instead." 2/4 4/4 3/4 "Actually make that {tgt} instead." 2/4 0/4 "Make that {tgt}." 2/4 0/4 "I meant {src}." (source clarification, control) 0/4 converted 2/4 converted When the wording names the role, v3 keeps the flag 12 of 12 times and applies the new target (the 3 turn-3 and 2 control misses are the known 1-hour arithmetic errors). When it doesn't, v3 guesses 8 of 8 times. Overall exact match on the probe: tuned 78.3%, base 36.7%.
Root cause (data): in v1-v3 training data, a zone named right after a source flag is always a source clarification (30
clarifytemplates, e.g. "Actually, the timezone is {value}", "Please go with {value}"). "Make that ..." appears only in a date template ("Actually, make that {date}"), where it replaces the value just discussed. No template changes a zone without saying which component it fills. So the model learned to fill the flagged slot by position, not by the words. The spec says a message that doesn't make clear which component a value fills is AMBIGUOUS. The drift eval missed this because its target changes all use role-explicit wording, so v3's flag-then-change failures there are all arithmetic.Fix (v4 data,
333f6f5): (1) a new teacher template kind for zone changes that don't state a role ("Make that {zone}.", "Use {zone} instead."), labelled by the oracle as AMBIGUOUS both after a flag and after a clean request (decided 2026-10-29: the spec as written, no spec change); (2) more role-explicit correction wordings for target changes, so "actually" and "instead" don't signal "source"; (3) contrast triples from one flagged opening (source clarification, explicit target change, change with no role) so the role must come from the words; (4) the probe's wordings excluded from training. As built (gen_train.pyrole_change_example, teacher.py): 37 role-free and 38 explicit-target templates passed a role gate (the gate call must find the role unstated, or the target named). Templates on the wrong side of the line were dropped for v4: role-free templates that say "meant" or "correction" (those clarify the flagged zone, like the probe's "I meant {src}." control), and two old clarify templates that read as role-free swaps ("Please go with {value}."). One explicit-target template that nearly copied a probe wording was dropped too. data/train_v4.jsonl: 12,000 examples, seed 14, 934 role_change conversations (7.8%); v1-v3 still regenerate byte for byte.Frontier models on the same probe (2026-11-01, best strategy = CoT with reasoning on, 2 repeats, results/probe/target_change/frontier/; judge agrees with exact match on 100% of turns): role-free wordings keep the flag 6/8 and 5/8 for Sonnet 5.5 and 4/8 and 5/8 for Mistral Large 4; role-explicit target changes 12/12 for both. Overall exact match: Sonnet 96.7% / 95.0%, Mistral 91.7% / 91.7%. So the frontier models guess here too, less often than v3 (0/8).
Ship rule (fixed before training): v4 replaces v3 as the headline checkpoint only if the probe passes (8/8 and 12/12) and ceiling and drift stay within noise of v3 (8 and 10 points). Otherwise v3 stays, and v4 is reported as a measured attempt.
Test:
python eval.py --model <checkpoint> --eval-set cases/probe_target_change.jsonl, thenpython scripts/report_probe.py <base results> <tuned results>. v3 fails it (0/8 on the no-role wordings). The fix passes when v4 keeps the flag on 8/8 no-role wordings and still 12/12 on the role-explicit ones. By hand: run cases/probe_live_conversation.jsonl; turn 2 must beAMBIGUOUS: source_timezone.Result (2026-11-01): v4-n12000 (43.0 min on an H100, same hyperparameters) passes the ship rule.
v3 v4 Probe: role-free change keeps the flag 0/8 8/8 Probe: role-explicit target change keeps the flag 12/12 12/12 Probe: overall exact match 78.3% 86.7% Ceiling: spec-adherence / robustness 72.0% / 84.5% 72.0% / 79.6% Drift: conversations held / turns correct 75.7% / 91.8% 80.0% / 93.8% Remaining probe misses are wrong hours on turn 3 (after the source is clarified), the same value sets under every wording, so arithmetic (ISSUE-003), not role reading. Cost of the fix: on the ceiling set, 3 multi-fault scenarios now report
AMBIGUOUS: source_timezonebefore an invalid date or time (priority order), plausibly because every role-free change in the contrast sets is labelledAMBIGUOUS: source_timezone; the robustness drop (-4.9) is within noise and otherwise wrong hours. Results: results/probe/target_change/v4-n12000/, results/eval/v4-n12000/, results/drift/v4-n12000/.
ISSUE-003: tuned v3 is an hour off on conversions between two DST zones
Status: open, partly improved. v5 (debudash/qwen3-4b-tz-convert-v5-n18000 @ f90209fc) lifts the DST probe from
58/80 to 68/80, two turns short of the pre-registered bar of 70/80, so the issue is not closed.
Issue: the tuned model's main remaining failure is a wrong hour (15 of its 30 failed ceiling turns; most of its drift failures, where a wrong hour then carries forward). Example from the demo script, convert_iana-03: "2025-12-26 at 07:00 America/New_York - what time is that in Pacific/Auckland?" Expected
2025-12-27T01:00:00+13:00; v3 replied2025-12-27T00:00:00+13:00(offset right, hour 1 h early). Found while reviewing the demo script on 2026-11-01.Diagnosis: over v3's ceiling and drift transcripts, the 27 turns where a wrong hour first appears: 26 have the right target offset, 18 are exactly 1 h off. Conversions fail 6/19 when both zones observe DST, 6/41 when only the source does, and 8/189 when neither does. The model labels the target offset correctly but applies the wrong side of a DST switch to the hour, most often when both zones switch on different dates or in opposite seasons.
Root cause (data): conversions between two DST zones are a small, unstratified share of the data: zones are sampled independently, so DST/DST pairs from different families, on dates spread over the year, are rare per (pair, season) cell, and anchor pairs (UTC first) are not targeted at them.
Probe (test first): cases/probe_dst_pairs.jsonl (scripts/gen_probe_dst_pairs.py, seed 31): 80 single conversions between two DST zones of different families, 40 northern/southern (opposite seasons) and 40 US/EU, in the eval set's phrasings, oracle-labelled, excluded from training. Base (full spec prompt): 0/80. Tuned v3: 58/80 (72.5%) (opposite hemispheres 27/40, US/EU 31/40); 15 of its 22 misses are exactly 1 h off, with the offset right in all but one. So v5 needs at least 87.5% under the ship rule. v4 (ISSUE-002 fix only): 52/80 (65.0%), within noise of v3. Frontier models at their best prompt (CoT, reasoning on, 2 repeats, results/probe/dst_pairs/frontier/): Sonnet 5.5 75/80 and 74/80, Mistral Large 4 79/80 and 80/80. Arithmetic is where reasoning helps them most.
Fix (v5 data, same hyperparameters):
arith_examplein gen_train.py samples DST-zone pairs stratified by family (70% different families, 15% same family, 15% one side without DST) on dates spread over 2020-2030, half as anchor pairs that show the UTC value first. v5 = the v4 mix at 2/3 weight plus arith at 1/3, N=18,000 (seed 15), so the v4 categories keep about their v4 counts. 5,867 arith conversations; 0 dropped for eval overlap; v1-v4 still regenerate byte for byte.Ship rule (fixed before training): v5 becomes the headline checkpoint only if (1) the DST probe improves on v3 by at least 15 points (80 turns: about 2 standard errors is 11 points), (2) the ISSUE-002 probe still passes (8/8 and 12/12), and (3) ceiling spec-adherence and robustness and drift conversations held stay within noise (8 / 8 / 10 points) of the better of v3 and v4. Otherwise the headline is the latest version that passes its own rule, and v5 is reported as a measured attempt.
Test:
python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_dst_pairs.jsonl.Result (2026-11-01): v5-n18000 (78.0 min on an H100, same hyperparameters, final loss 0.0044).
v3 v4 v5 Ship rule DST probe 58/80 52/80 68/80 (opposite hemispheres 33/40, US/EU 35/40) needs 70/80: missed by 2 ISSUE-002 probe: role-free / role-explicit 0/8, 12/12 8/8, 12/12 8/8, 12/12 passes Ceiling: spec-adherence / robustness 72.0% / 84.5% 72.0% / 79.6% 72.0% / 87.4% passes Drift: held / turns correct 75.7% / 91.8% 80.0% / 93.8% 91.4% / 98.1% passes 10 of v5's 12 probe misses are still exactly 1 h off, and convert_iana-03 (New York to Auckland) is still 1 h early. Clean single-turn spec-adherence did not move (72.0%), so the gain shows on DST pairs and in conversations (drift wrong hours 33 → 5) more than on the clean ceiling set.
Decision (user, 2026-11-01): the literal rule makes v4 the headline, but v5 is equal or better than v4 on every measure, so v5 is the final model. ISSUE-003 stays open: the bar set before training was not met and is not moved. Next step: more arithmetic data per (pair, season) cell, or let code do the arithmetic (the model outputs the structured request, the oracle computes the time). Results: results/probe/dst_pairs/v5-n18000/, results/eval/v5-n18000/, results/drift/v5-n18000/, results/probe/target_change/v5-n18000/.