Post
31
Zero-shot Qwen3-8B on a 10-item honesty gate: 90%. After a tiny LoRA tune (194 examples, rank 8, 3 epochs): 94%.
Also after that same tune: a fact it got right 20/20 times before the tune, it now gets right 12/20.
Here's what happened. Two fine-tune jobs went out on Fireworks before I'd actually run a baseline — caught mid-session when asked directly "did we eval before, or just tune?" Answer was no, just tune. So: baseline first, honestly, after the fact.
Then a real infra wall: Fireworks won't let you download a trained LoRA's weights, and won't serve either of these two base models (Qwen3-8B, Llama-3.1-8B-Instruct) with a LoRA addon at all -- "does not support serverless addons." No export, no inference. So I reproduced both tunes locally on a rented L40S, same data, same hyperparameters, and ran the real before/after there instead.
Results, k=20 samples per item (not single-shot -- one ask isn't a measurement):
Qwen3-8B, honesty task (its own tune): 90.0% -> 94.0%. But a claim it nailed cold before the tune -- confidently, every single time -- it now gets wrong 40% of the time. Looks less like the model learning to discriminate better and more like the tune pushing its whole decision threshold toward skepticism. Net accuracy went up. What it's actually doing changed in a way the aggregate number hides.
Llama-3.1-8B, misbehavior-discrimination task (its own tune): 92.9% -> 96.8%, no regression anywhere, mostly from fixing the one item both base models failed completely. Same dataset, full fine-tune, different architecture (Qwen2.5-7B), a month ago: no measurable effect. Architecture + method mattered more than I'd assumed.
Adapters, raw eval data, and the full writeup (including the baseline gap and how it got caught) are up now.
SoulInPsyAbstract/qwen3-8b-binary-honesty-lora
SoulInPsyAbstract/llama31-8b-misbehavior-discriminator-lora
EXP-040 writeup (Qwen3-8B)
EXP-041 writeup (Llama-3.1-8B)
Also after that same tune: a fact it got right 20/20 times before the tune, it now gets right 12/20.
Here's what happened. Two fine-tune jobs went out on Fireworks before I'd actually run a baseline — caught mid-session when asked directly "did we eval before, or just tune?" Answer was no, just tune. So: baseline first, honestly, after the fact.
Then a real infra wall: Fireworks won't let you download a trained LoRA's weights, and won't serve either of these two base models (Qwen3-8B, Llama-3.1-8B-Instruct) with a LoRA addon at all -- "does not support serverless addons." No export, no inference. So I reproduced both tunes locally on a rented L40S, same data, same hyperparameters, and ran the real before/after there instead.
Results, k=20 samples per item (not single-shot -- one ask isn't a measurement):
Qwen3-8B, honesty task (its own tune): 90.0% -> 94.0%. But a claim it nailed cold before the tune -- confidently, every single time -- it now gets wrong 40% of the time. Looks less like the model learning to discriminate better and more like the tune pushing its whole decision threshold toward skepticism. Net accuracy went up. What it's actually doing changed in a way the aggregate number hides.
Llama-3.1-8B, misbehavior-discrimination task (its own tune): 92.9% -> 96.8%, no regression anywhere, mostly from fixing the one item both base models failed completely. Same dataset, full fine-tune, different architecture (Qwen2.5-7B), a month ago: no measurable effect. Architecture + method mattered more than I'd assumed.
Adapters, raw eval data, and the full writeup (including the baseline gap and how it got caught) are up now.
SoulInPsyAbstract/qwen3-8b-binary-honesty-lora
SoulInPsyAbstract/llama31-8b-misbehavior-discriminator-lora
EXP-040 writeup (Qwen3-8B)
EXP-041 writeup (Llama-3.1-8B)