Model card: full FT vs LoRA results, autocast caveat, repro command

README.md

@@ -4,17 +4,100 @@ base_model: Qwen/Qwen3-VL-4B-Instruct
 tags:
 - reward_model
 - rbm
-- preference_comparisons
+- robometer
+- mimicgen
 library_name: transformers
 ---
 
-# chomeed/robometer-4b-full-threading-d0
+# Robometer-4B Full Fine-tune — MimicGen threading_d0
 
-## Model Details
+Full-parameter fine-tune of [`robometer/Robometer-4B`](https://huggingface.co/robometer/Robometer-4B)
+(Qwen3-VL-4B-Instruct) on MimicGen `threading_d0` rollouts. Emits, per frame, a
+**progress** value in [0,1] and a **success probability**.
 
-- **Base Model**: Qwen/Qwen3-VL-4B-Instruct
-- **Model Type**: qwen2_5_vl
+This is the no-PEFT counterpart of
+[`chomeed/robometer-4b-lora-threading-d0`](https://huggingface.co/chomeed/robometer-4b-lora-threading-d0):
+identical data, schedule, and eval config — the only change is `model.use_peft=false`.
 
-## Citation
+- **4.03B trainable params (90.7%)** — language model + all three prediction heads; the vision
+  tower stays frozen (`train_vision_encoder: false`), same as the LoRA run
+- 1000 steps, batch 8, lr 2e-5, cosine, warmup 0.1, weight decay 0.01, grad-norm clip 10
+- bf16 full fine-tuning via Unsloth; **34 GB peak** on one B200 (the LoRA run peaked at 14.7 GB)
+- Uploaded checkpoint is **step 500**, the best of the run by mean(pearson, kendall_last)
+- Data: [`chomeed/mimicgen_threading_d0_train_rfm`](https://huggingface.co/datasets/chomeed/mimicgen_threading_d0_train_rfm)
+  (160 traj: 80 successful + 80 failure), evaluated on
+  [`..._test_rfm`](https://huggingface.co/datasets/chomeed/mimicgen_threading_d0_test_rfm) (40 traj, disjoint seeds)
 
-If you use this model, please cite:
+## Results (held-out, 40 trajectories with unseen seeds)
+
+Best checkpoint of each run, scored by the trainer's own eval:
+
+| metric | full FT (step 500) | LoRA (step 600) |
+|---|---|---|
+| reward-alignment Pearson | **0.9804** | 0.9753 |
+| policy-ranking Kendall (last) | **1.000** | **1.000** |
+| ranking acc, failure vs successful | **1.000** | **1.000** |
+| reward-alignment eval loss | **3.236** | 3.390 |
+| success AUPRC | 0.162 | 0.192 |
+
+Matched-step comparison at step 1000, so the two runs are compared at equal training budget:
+
+| metric @ step 1000 | full FT | LoRA |
+|---|---|---|
+| Pearson | **0.9758** | 0.9749 |
+| success-head AUROC (threshold-free) | **0.9944** | 0.9907 |
+| eval loss | **3.208** | 3.329 |
+
+Full fine-tuning wins consistently but by a small margin — Pearson is higher at every matched
+eval step, and eval loss is lower throughout. Kendall is saturated at 1.000 for both, so this
+test set does not separate them on trajectory ranking.
+
+**Success-head calibration:** both models emit very low absolute success probabilities
+(peak ≈0.002 on successful held-out trajectories). Ranking is excellent (AUROC 0.99) but the
+probabilities are not calibrated — threshold well below 0.01, or just use the progress signal.
+
+## Usage
+
+> **Important:** this is a full-weights checkpoint, like the upstream `robometer/Robometer-4B`.
+> It keeps fp32 `pixel_values` against bf16 vision blocks, which is fine under the Trainer's
+> autocast but raises `RuntimeError: expected scalar type BFloat16 but found Float` at plain
+> inference. **Wrap the forward pass in `torch.autocast`.** (LoRA checkpoints work either way.)
+
+```python
+import numpy as np, torch
+from robometer.data.dataset_types import ProgressSample, Trajectory
+from robometer.evals.eval_server import compute_batch_outputs
+from robometer.utils.save import load_model_from_hf
+from robometer.utils.setup_utils import setup_batch_collator
+
+device = torch.device("cuda")
+exp_config, tokenizer, processor, model = load_model_from_hf(model_path="chomeed/robometer-4b-full-threading-d0", device=device)
+model.eval()
+collator = setup_batch_collator(processor, tokenizer, exp_config, is_eval=True)
+
+frames = ...  # uint8 (T, H, W, C), RGB, third-person view
+traj = Trajectory(frames=frames, frames_shape=tuple(frames.shape), task="Pick up the needle and thread it through the hole in the tripod.",
+                  id="0", metadata={"subsequence_length": int(frames.shape[0])}, video_embeddings=None)
+batch = collator([ProgressSample(trajectory=traj, sample_type="progress")])
+inputs = {k: (v.to(device) if hasattr(v, "to") else v) for k, v in batch["progress_inputs"].items()}
+
+with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):   # <-- required
+    out = compute_batch_outputs(model, tokenizer, inputs, sample_type="progress",
+                                is_discrete_mode=True, num_bins=10)   # progress head is discrete, 10 bins
+
+progress = np.array(out["progress_pred"][0])                     # (T,) in [0,1]
+success  = np.array(out["outputs_success"]["success_probs"][0])  # (T,) in [0,1]
+```
+
+Serving: `python robometer/evals/eval_server.py model_path=chomeed/robometer-4b-full-threading-d0`
+
+## Reproducing
+
+```bash
+TASK=mimicgen_threading_d0 GPU=0 STEPS=1000 bash run_full_task.sh
+```
+
+Requires `robometer/data/dataset_success_cutoff.txt` to contain
+`mimicgen_threading_d0_train_rfm,1.0` and `mimicgen_threading_d0_test_rfm,1.0` (simulation
+trajectories have exact endpoints), and the RFM datasets preprocessed via
+`robometer/configs/preprocess_threading_d0.yaml`.