README.md

metadata

license: other
base_model: Qwen/Qwen3.5-4B
tags:
  - uncertainty-quantification
  - bayesian
  - grpo
  - lora

BayesClue-Latent checkpoints (private backup)

Latent-belief RL on the passive BayesClue detective game. Base: Qwen3.5-4B. Belief is read from forced-choice probe logits (never verbalized).

Contents

path	what	key metric
sft_final/adapter/	Stage-1 distributional-KL SFT adapter (LoRA r64). Soft-target match KL(p* ‖ softmax(logits[label_ids])) at the Answer: token. R1.	held-out belief KL 0.0066, top1 0.93
rl_lr1e5_step400/lora_adapter/	Stage-2 logit-probe GRPO LoRA delta (lr 1e-5, step 400, 346 modules). Reasoning-only response mask → probe gets zero gradient.	—
rl_lr1e5_step400/merged/	R2 = SFT⊕RL correctly merged into base (full fp model).	post-reasoning world_kl 0.0245 (= proper-scoring entropy floor), q_entropy→p_entropy, world_top1 0.71

Notes

• Merge the LoRA with credal_verl08/remerge_sft_qwen35.py (the model.layers.→model.language_model.layers. remap); the stock verl.model_merger writes a base copy.
• Reward R = α·(−CE(p*_H,q_H)) + (1−α)·mean_s(−CE(p*_R,q_R)), α=0.6. The −1.225 reward plateau IS the optimum −H(p*), not a truncation artifact.