RetentionLabs/haan-stage1-pilot

Haan stage-1 pilot backup (2026-07-31)

450-step stage-1 pilot run (Nunchi timing + Zeroth-ko/LibriTTS-R content), trained on 4×A100-80GB with FSDP2, torchao 8-bit optimizer (eager-stanced), accumulation-boundary comm deferral. Code: github latentforge/Haan @ main (commit at backup time; every fix this run depends on is pushed).

Layout

• init_from_qwen3/ — the assembled warm-start (Qwen3-8B backbone + Moshi audio/depth grafts, bf16). Required by the trainer to BUILD the model before a resume loads weights over it (--init_ckpt).
• checkpoints/stage1/step_{300,400,450}/ — sharded training manifests: state_rank{0..3}.pt (fp32 model shards + quantized 8-bit Adam moments + per-rank RNG + scheduler/sampler state), provenance.json, train_args.json. Resume requires world_size=4 (any 4-GPU host).
• checkpoints/stage1/step_*_bf16/ — consolidated bf16 exports, from_pretrained-loadable (eval/inference; no optimizer state).
• train_stage1.log — full training log (per-step losses).
• samples/ — self-play listening wavs per checkpoint (same fixed seeds across checkpoints; also on wandb run eval-stage1, project haan).

Resume recipe

git clone git@github.com:latentforge/Haan && cd Haan && uv sync
# restore this repo's init_from_qwen3/ -> trainer_output/init_from_qwen3
#         and checkpoints/stage1/step_450 -> checkpoints/stage1/step_450
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
uv run torchrun --nproc_per_node=4 -m project_amnesty.utils.train \
  --stage stage1 --init_ckpt trainer_output/init_from_qwen3 \
  --resume checkpoints/stage1/step_450 \
  --steps 45000 --log_interval 1 --ckpt_interval 2000

Notes for the next round (from this run's eval curves, wandb eval-stage1):

• ko/ja text-retention ppl drifted upward once lr approached peak (ko 8.73→11.26, ja 11.91→14.39 by step 400) — gate this in the full run (candidates: lower lr, partial backbone freeze, text-retention anchor mix).
• FiLM identity pins held exactly (role0_identity_intact True throughout) after the pin-mask fix; gamma movement is one bf16 quantum so far.
• The training host had no GPU P2P (NCCL over SHM <1 GB/s): keep the accumulation-boundary comm deferral and sharded checkpointing; a host WITH P2P should cut the ~47 s/step roughly to ~15 s.