Shaer-AI/shaer-adapters-v2

Shaer Meter-Loss Continuation V2

This repository stores the first continuation-from-best-eval meter-loss experiment.

Experiment Identity

• Source baseline run: train_20260407_231929
• Source checkpoint: /root/workspace/Shaer/sft/outputs/train/train_20260407_231929/checkpoint-3000
• Target repo: Shaer-AI/shaer-adapters-v2
• Namespace: meter_aux_from_best_eval_v1
• Run dir: /root/workspace/Shaer/sft/outputs/train/train_20260408_133801

What Was Tried

• Start from the finished baseline best checkpoint by eval loss
• Keep the same dataset, splits, weighted sampler, tokenizer, prompt format, and all-linear QLoRA setup
• Add a train-time differentiable auxiliary meter loss on top of CE

Why This Run Was Stopped

• The auxiliary meter head improved quickly on held-out eval
• But held-out CE became worse than the starting baseline checkpoint
• And generation-side probe meter also became worse than the starting baseline checkpoint

Latest Useful Stopped Snapshot

• Step: 400
• Eval CE loss: 2.2890455386173585
• Eval meter loss: 0.009391324620048254
• Eval total loss: 2.2899846720801658
• Eval meter accuracy: 1.0
• Probe meter mean: 0.4256693654883061
• Probe count adherence mean: 1.0

Interpretation

This run is preserved as a useful cautionary / negative result. The evidence points more toward objective mismatch or harmful drift than toward a convincing model improvement. A later audit also showed that the current auxiliary meter-loss design is flawed because it can recover the requested meter from prompt-conditioned hidden states.

Useful Paths

• Stopped-run summary:
  • reports/meter_aux_from_best_eval_v1/train_20260408_133801_stopped_summary.md
• Checkpoints:
  • checkpoints/meter_aux_from_best_eval_v1/train/train_20260408_133801/