Sasha Malahov
example-repro: spec-faithful Qwen3-Emb+LoRA+prototype-head training on synthetic data
68a82f9 | base_model: Qwen/Qwen3-Embedding-0.6B | |
| max_length: 512 | |
| lora: | |
| r: 16 | |
| alpha: 32 | |
| dropout: 0.05 | |
| target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj] | |
| head: | |
| scale_init: 20.0 | |
| loss: | |
| label_smoothing: 0.05 | |
| optimizer: | |
| lora_lr: 3.0e-4 | |
| lora_wd: 0.01 | |
| head_lr: 2.0e-3 | |
| head_wd: 0.0 | |
| warmup_frac: 0.10 | |
| final_lr_frac: 0.05 | |
| train: | |
| batch_size: 16 | |
| epochs: 12 | |
| patience: 3 | |
| dtype: bf16 # spec; if metrics look odd on this stack, try fp32 | |
| data: | |
| dir: data | |
| taxonomy: topic_taxonomy.json | |
| train: train.jsonl | |
| validation: validation.jsonl | |
| test: test.jsonl | |
| # hard topic pairs get x2 sampling weight (mirrors the spec's "hard-pair topics x2") | |
| hard_pairs: | |
| - [soccer, sports_general] | |
| - [swim_club, sports_general] | |
| - [art_club, crafts_club] | |
| - [music_band, choir] | |
| - [flu_report, allergies] | |
| - [bake_sale, fundraiser] | |
| out_root: runs | |