Add step-10000 model card, config, normalization, and provenance

README.md

@@ -0,0 +1,181 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: robotics
+library_name: molmobot
+base_model: allenai/MolmoBot-DROID
+datasets:
+- ccwatson/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot
+tags:
+- molmobot
+- robotics
+- manipulation
+- franka
+- droid
+- pick-and-place
+- spatial-reasoning
+- left-right
+- molmospaces
+- 10000-steps
+model-index:
+- name: MolmoBot-DROID Directional Pick/Place SimpleDesk Step 10000
+  results:
+  - task:
+      type: robotics
+      name: Franka pick-and-place relative left/right
+    dataset:
+      type: ccwatson/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench
+      name: MolmoSpaces SimpleDesk Diverse Targets Tested40
+      split: 40 benchmark episodes
+    metrics:
+    - type: success_rate
+      name: Strict success rate
+      value: 0.625
+    - type: lax_success_rate
+      name: Lax success rate
+      value: 0.65
+---
+
+# MolmoBot-DROID Directional Pick/Place SimpleDesk — Step 10,000
+
+This repository contains the **10,000-step intermediate checkpoint** from the
+`lr_custom_16k` MolmoBot finetuning run. It was finetuned from
+[`allenai/MolmoBot-DROID`](https://huggingface.co/allenai/MolmoBot-DROID) for
+Franka/FR3 tabletop pick-and-place commands that require placing one object to
+the **left or right of another object**.
+
+The root `config.yaml` and `model.pt` form the unsharded MolmoBot checkpoint
+expected by the MolmoBot server. This is not the final 16,000-step checkpoint;
+the step is deliberately included in both the repository name and this model
+card.
+
+## Checkpoint identity
+
+| Field | Value |
+| --- | --- |
+| Checkpoint step | **10,000** |
+| Training run | `lr_custom_16k` |
+| Checkpoint format | MolmoBot unsharded (`config.yaml` + `model.pt`) |
+| Base model | `allenai/MolmoBot-DROID` |
+| Base model revision | `cbe6ec358958d07ddfb20d3aa54e560e9e1b18c9` |
+| Training W&B run | [`ccwatson/molmobot_finetune/ouuzdjlg`](https://wandb.ai/ccwatson/molmobot_finetune/runs/ouuzdjlg) |
+| Training Slurm job | `7107997` (`lr-custom-ie-fe-16k-b16-32`) |
+| Checkpoint created | 2026-07-17 12:11 UTC |
+| `model.pt` size | 19,992,166,548 bytes |
+| `config.yaml` SHA-256 | `7eeb51134b93fc17955f3bd850240074d6b9faf3dcf527177861940c7c098400` |
+
+The complete resolved model/trainer configuration is in `config.yaml`. The
+submission-time environment is preserved as `metadata/training_launch.env`, and
+machine-readable provenance is in `metadata/provenance.json`.
+
+## Training data
+
+Training used
+[`ccwatson/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot`](https://huggingface.co/datasets/ccwatson/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot)
+at dataset revision `5446191605f7eb3020983d83f707153d9719ecfb`.
+
+The dataset contains 200 successful scripted trajectories derived from 40
+valid benchmark episodes, with five attempts per episode:
+
+| Split | Source episodes | Trajectories | HDF5 files | Videos |
+| --- | ---: | ---: | ---: | ---: |
+| Train | 36 | 180 | 180 | 360 |
+| Validation | 4 | 20 | 20 | 40 |
+
+The two cameras are `droid_shoulder_light_randomization` and
+`wrist_camera_zed_mini`. Prompts were **not randomized** during training.
+Normalization was computed from the train split using action quantiles and
+state min/max statistics for the eight-dimensional Franka action and state
+vectors. The exact statistics are included as `normalization_stats.yaml`
+(SHA-256 `972a88e10f02976ce5a4d1c90a2070a62e4daa08a5fe1cecb4ca00e8216eb15d`).
+
+## Main training settings
+
+| Setting | Value |
+| --- | --- |
+| Intended run length | 16,000 steps |
+| This checkpoint | 10,000 steps |
+| Device / global batch size | 16 / 32 |
+| Sequence length | 928 |
+| Observation frames | 2 |
+| Observation step delta | 8 |
+| Maximum images | 4 |
+| Maximum frames | 1 |
+| Single-frame preprocessing | false |
+| Action preset | `franka_joint` |
+| Camera preset | `franka_droid` |
+| Action horizon in data | 16 × 8 |
+| Weighted sampling | enabled |
+| Prompt randomization | disabled |
+| Image augmentation | enabled |
+| LLM finetuning | enabled |
+| Embedding finetuning mode | `ae` |
+| Flow timestamps | 8 |
+| Connector learning rate | `5e-6` |
+| Vision encoder learning rate | `5e-6` |
+| LLM learning rate | `1e-5` |
+| Action expert learning rate | `1e-4` |
+| Validation interval / max examples | 200 / 200 |
+| Save and external-eval interval | 2,000 steps |
+| Trainer and optimizer state | reset from base model |
+| Hardware | 1× NVIDIA B200, 28 CPUs, 224 GiB host memory |
+
+## Closed-loop evaluation
+
+This checkpoint was evaluated on all 40 episodes of
+`FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench` (20
+left and 20 right tasks) through the VLA evaluation harness and MolmoSpaces.
+The evaluation used natural-language prompts, Filament rendering, 10 parallel
+shards, a 500-step task horizon, and both DROID cameras.
+
+Server action settings were:
+
+- execution horizon: 8
+- state mode: `cross_attn`
+- maximum joint delta: 0.2
+
+| Metric | Result |
+| --- | ---: |
+| Strict success | **25 / 40 (62.5%)** |
+| Lax success | 26 / 40 (65.0%) |
+| Strict oracle-done rate | 25 / 40 (62.5%) |
+| Lax oracle-done rate | 27 / 40 (67.5%) |
+| Average episode length | 274.675 steps |
+| Average successful episode length | 140.08 steps |
+| Min / max episode length | 32 / 499 steps |
+| Failed shards | 0 |
+| Missing / duplicate / unexpected episodes | 0 / 0 / 0 |
+
+The eval completed successfully (`eval_returncode=0`) in 2,016.92 seconds.
+See the [evaluation W&B run](https://wandb.ai/ccwatson/mlspaces-json-eval/runs/external-eval-lr_custom_16k-step10000),
+`metadata/external_eval_request.json`, and
+`metadata/external_eval_result.json` for the complete recorded configuration
+and result.
+
+## Repository files
+
+- `model.pt`: unsharded MolmoBot weights.
+- `config.yaml`: complete resolved model and trainer configuration.
+- `normalization_stats.yaml`: exact state/action normalization statistics.
+- `metadata/training_launch.env`: exact experiment launch configuration.
+- `metadata/external_eval_request.json`: checkpoint identity and requested eval settings.
+- `metadata/external_eval_result.json`: complete eval result and coverage accounting.
+- `metadata/provenance.json`: source revisions, hashes, run identity, and artifact inventory.
+
+## Intended use and limitations
+
+This checkpoint is intended for research evaluation of the matching
+Franka/FR3 DROID-camera setup and directional SimpleDesk tasks. It should be
+served with the MolmoBot inference stack and the included normalization stats.
+It has only been measured on the 40-episode simulated benchmark described
+above. The 62.5% result is a single benchmark evaluation, not a claim of broad
+robotic capability or real-world safety. Validate action scaling, cameras,
+workspace limits, collision handling, and emergency-stop behavior before any
+physical-robot use.
+
+## License
+
+This derivative checkpoint follows the Apache-2.0 license declared by the base
+`allenai/MolmoBot-DROID` model. Users remain responsible for the terms of the
+base model, training dataset, simulator assets, and other dependencies.

config.yaml

@@ -0,0 +1,705 @@
+run_name: lr_custom_16k
+model:
+  model_name: molmobot
+  data_formatter:
+    prompt_templates: uber_model_v2
+    message_format: qwen3
+    system_prompt: demo_or_style_v2
+    always_start_with_space: false
+    default_inference_len: 65
+    select_answer: best
+    debug: false
+    image_last: false
+    format_message_list: null
+    p_one_message: 0.0
+    eval_system_prompt_mapping: null
+    p_choice_content_in_mc: 1.0
+    template_video_mc_questions: true
+    pointing_format: html-v2
+    points_decimal_places: 1
+    use_seperate_non_pointing_qa_style: false
+    timestamp_mode: 50-percent-seconds
+    output_timestamp_mode: seconds
+    seconds_decimal_places: 1
+    p_multi_point_all_image: 0.5
+    use_seperate_count_without_pointing_style: false
+    sample_random_initial_point: true
+  llm:
+    d_model: 2560
+    n_heads: 32
+    n_kv_heads: 8
+    head_dim: 128
+    qkv_bias: false
+    clip_qkv: null
+    n_layers: 36
+    mlp_ratio: 4
+    mlp_hidden_size: 19456
+    activation_type: swiglu
+    block_type: sequential
+    rope: true
+    rope_full_precision: true
+    rope_theta: 5000000.0
+    rope_type: default
+    rope_factor: null
+    rope_high_freq_factor: null
+    rope_low_freq_factor: null
+    rope_original_max_position_embeddings: null
+    rope_attention_factor: null
+    rope_beta_fast: null
+    rope_beta_slow: null
+    rope_mscale: null
+    rope_mscale_all_dim: null
+    rope_truncate: null
+    attention_type: sdpa
+    full_attention_layers: null
+    sliding_attention_rope_scaling: false
+    float32_attention: true
+    attention_dropout: 0.0
+    attention_layer_norm: true
+    attention_layer_norm_type: qwen3
+    residual_dropout: 0.1
+    response_residual_dropout: 0.0
+    layer_norm_type: rms
+    layer_norm_with_affine: true
+    layer_norm_eps: 1.0e-06
+    attention_layer_norm_with_affine: true
+    max_sequence_length: 8192
+    max_position_embeddings: null
+    include_bias: false
+    bias_for_layer_norm: null
+    norm_after: false
+    moe_num_experts: 8
+    moe_top_k: 2
+    moe_mlp_impl: sparse
+    moe_log_expert_assignment: false
+    moe_shared_expert: false
+    moe_lbl_in_fp32: false
+    moe_interleave: false
+    moe_loss_weight: 0.1
+    moe_zloss_weight: null
+    moe_dropless: true
+    moe_capacity_factor: 1.25
+    embedding_dropout: 0.0
+    scale_logits: false
+    vocab_size: 151936
+    additional_vocab_size: 128
+    weight_tying: true
+    embedding_size: 151936
+    use_position_ids: true
+    tokenizer:
+      identifier: Qwen/Qwen3-4B-Instruct-2507
+      tokenizer_dir: null
+    init_path: /weka/oe-training-default/mm-olmo/pretrained_llms/qwen3-4b-instruct.pt
+    init_incremental: null
+    new_embedding_init_range: 0.02
+    initializer_range: 0.02
+    normalize_input_embeds: false
+    activation_checkpoint: whole_layer
+    compile: blocks
+    fix_pad_tokenizer: false
+    init_std: 0.02
+    init_fn: normal
+    init_cutoff_factor: null
+  vision_backbone:
+    vit:
+      image_model_type: siglip
+      image_default_input_size:
+      - 378
+      - 378
+      image_patch_size: 14
+      image_pos_patch_size: 14
+      image_emb_dim: 1152
+      image_num_heads: 16
+      image_num_key_value_heads: 16
+      image_num_layers: 27
+      image_head_dim: 72
+      image_mlp_dim: 4304
+      image_mlp_activations: gelu_pytorch_tanh
+      image_dropout_rate: 0.0
+      image_num_pos: 729
+      image_norm_eps: 1.0e-06
+      attention_dropout: 0.0
+      residual_dropout: 0.0
+      initializer_range: 0.02
+      float32_attention: true
+      attention_type: sdpa
+      sdpa_backend: all
+      activation_checkpointing: true
+      init_path: /weka/oe-training-default/mm-olmo/pretrained_image_encoders/siglip2-so400m-14-384.pt
+      resize_mode: siglip
+      pad_value: 0.0
+      normalize: siglip
+    image_pooling_2d: attention_meanq
+    pooling_attention_mask: true
+    image_projector: mlp
+    image_padding_embed: null
+    vit_layers:
+    - -3
+    - -9
+    skip_unused_layers: true
+    use_deepstack: false
+    share_connector: false
+    image_feature_dropout: 0.0
+    connector_activation_checkpointing: true
+    compile_vit: blocks
+    pool_size_embeds: null
+    compile_connector: null
+    normalize_on_gpu: true
+    use_image_augmentation: true
+    use_resize_bottleneck: false
+  mm_preprocessor:
+    max_answer_len: null
+    last_message_loss_only: false
+    max_text_tokens: null
+    loss_token_weighting: root_subsegments_root_tokens
+    max_frames: 1
+    frame_sample_mode: uniform_last_frame
+    candidate_sampling_fps:
+    - 0.25
+    - 0.5
+    - 1.0
+    - 2.0
+    - 4.0
+    - 6.0
+    - 8.0
+    - 16.0
+    cache_videos: true
+    loading_method: torchcodec_exact
+    max_fps:
+    - 2.0
+    time_sampling: true
+    time_mode: per-frame-compact
+    subtitle_mode: frame_1
+    max_crops: 1
+    overlap_margins:
+    - 4.0
+    - 4.0
+    use_col_tokens: false
+    periodic_high_res_frame: null
+    high_low_train_mode: local_rnd
+    high_res_frame_sample_options: null
+    periodic_sample_rate_training:
+      4:
+      - 0.9
+      - 0.03
+      - 0.03
+      - 0.04
+      3:
+      - 0.6
+      - 0.2
+      - 0.2
+    skip_low_res_in_high_low: false
+    pooling_w: 3
+    pooling_h: 3
+    high_res_pooling_w: null
+    high_res_pooling_h: null
+    query_based_resolution_selection: false
+    max_queries_for_resolution_selection: 8
+    use_frame_special_tokens: true
+    frame_sel_clip_identifier: google/siglip2-so400m-patch14-384
+    image_padding_mask: false
+    max_subtitle_tokens: null
+    image:
+      crop_mode: resize
+      use_col_tokens: true
+      max_crops: 8
+      high_res_max_crops: 24
+      p_high_res: 0.0
+      pooling_w: 2
+      pooling_h: 2
+      overlap_margins:
+      - 4
+      - 4
+      max_images: 4
+      max_multi_image_crops: 1
+      multi_image_pooling_w: 2
+      multi_image_pooling_h: 2
+      use_single_crop_col_tokens: false
+      use_single_crop_start_token: true
+      single_frame: false
+    topk: null
+    prune_from_frame: 0
+  bi_directional_attn: image_tokens
+  shared_low_high_embedding: true
+  debug: null
+  cp_enabled: false
+  apply_cp_to_vision_backbone: false
+  action_dim: 8
+  action_horizon: 16
+  n_action_steps: 8
+  n_obs_steps: 2
+  obs_step_delta: 8
+  action_expert:
+    max_horizon: 32
+    action_dim: 8
+    hidden_size: 768
+    num_layers: 36
+    num_heads: 8
+    mlp_ratio: 4.0
+    timestep_embed_dim: 256
+    dropout: 0.0
+    attn_dropout: 0.0
+    context_layer_norm: true
+  action_expert_layer_mode: per_layer
+  flow_matching_num_steps: 10
+  flow_matching_cutoff: 0.999
+  flow_matching_beta_alpha: 1.0
+  flow_matching_beta_beta: 1.5
+  num_flow_timestamps: 8
+  same_noise_per_time: false
+  states_mode: cross_attn
+  robot_preprocessor:
+    stats_by_repo:
+      synthmanip:
+        observation.state:
+          min:
+          - -2.2766549587249756
+          - -1.7608288526535034
+          - -1.7191452980041504
+          - -2.8627865314483643
+          - -0.9076813459396362
+          - 0.8232163190841675
+          - -2.4616048336029053
+          - -0.0008297820459119976
+          max:
+          - 2.080810546875
+          - 0.6704062223434448
+          - 1.9463512897491455
+          - -0.6767909526824951
+          - 0.8670341372489929
+          - 2.381640672683716
+          - 0.4024277329444885
+          - 0.8239965438842773
+        action:
+          q01:
+          - -1.3642053604125977
+          - -1.5828913450241089
+          - -0.7952697277069092
+          - -2.813814878463745
+          - -0.8428052663803101
+          - 1.053465485572815
+          - -2.1978600025177
+          - 0.0
+          q99:
+          - 0.7378915548324585
+          - 0.468648225069046
+          - 1.7082799673080444
+          - -0.9352492094039917
+          - 0.7082295417785645
+          - 2.1873221397399902
+          - 0.3892104923725128
+          - 255.0
+    default_repo_id: synthmanip
+    action_key: action
+    state_keys:
+    - observation.state
+    action_norm_mode: quantiles
+    state_norm_mode: min_max
+  robot_postprocessor:
+    stats_by_repo:
+      synthmanip:
+        observation.state:
+          min:
+          - -2.2766549587249756
+          - -1.7608288526535034
+          - -1.7191452980041504
+          - -2.8627865314483643
+          - -0.9076813459396362
+          - 0.8232163190841675
+          - -2.4616048336029053
+          - -0.0008297820459119976
+          max:
+          - 2.080810546875
+          - 0.6704062223434448
+          - 1.9463512897491455
+          - -0.6767909526824951
+          - 0.8670341372489929
+          - 2.381640672683716
+          - 0.4024277329444885
+          - 0.8239965438842773
+        action:
+          q01:
+          - -1.3642053604125977
+          - -1.5828913450241089
+          - -0.7952697277069092
+          - -2.813814878463745
+          - -0.8428052663803101
+          - 1.053465485572815
+          - -2.1978600025177
+          - 0.0
+          q99:
+          - 0.7378915548324585
+          - 0.468648225069046
+          - 1.7082799673080444
+          - -0.9352492094039917
+          - 0.7082295417785645
+          - 2.1873221397399902
+          - 0.3892104923725128
+          - 255.0
+    default_repo_id: synthmanip
+    action_key: action
+    state_keys:
+    - observation.state
+    action_norm_mode: quantiles
+    state_norm_mode: min_max
+parallelism:
+  data_parallel_replicate_degree: 1
+  enable_compiled_autograd: false
+  data_parallel_shard_degree: -1
+  fsdp_reshard_after_forward: default
+  context_parallel_config:
+    degree: 1
+    attention_type: ulysses
+    load_balancer: ulysses
+    head_stride: 1
+  tensor_parallel_config:
+    degree: 1
+    enable_async: false
+  data_parallel_config:
+    name: fsdp
+    param_dtype: null
+    reduce_dtype: float32
+    num_replicas: null
+    shard_degree: null
+    wrapping_strategy: full
+    prefetch_factor: 0
+  context_parallel_rotate_method: allgather
+seed: 6198
+epoch: null
+dry_run: false
+ft_llm: true
+ft_vit: false
+ft_connector: false
+ft_embedding: ae
+optimizer:
+  name: adamw
+  learning_rate: 0.0001
+  weight_decay: 0.01
+  betas:
+  - 0.9
+  - 0.95
+  eps: 1.0e-05
+  connector_learning_rate: 5.0e-06
+  vit_learning_rate: 5.0e-06
+  llm_learning_rate: 1.0e-05
+  frame_selector_learning_rate: 0.0001
+  temporal_token_scorer_learning_rate: 0.0001
+  action_expert_learning_rate: 0.0001
+  connector_weight_decay: 0.0
+  vit_weight_decay: 0.0
+  llm_weight_decay: 0.0
+  frame_selector_weight_decay: 0.01
+  temporal_token_scorer_weight_decay: 0.01
+  action_expert_weight_decay: 0.0
+  connector_betas:
+  - 0.9
+  - 0.95
+  vit_betas:
+  - 0.9
+  - 0.95
+  llm_betas:
+  - 0.9
+  - 0.95
+  frame_selector_betas:
+  - 0.9
+  - 0.95
+  temporal_token_scorer_betas:
+  - 0.9
+  - 0.95
+  action_expert_betas:
+  - 0.9
+  - 0.95
+  connector_eps: 1.0e-06
+  vit_eps: 1.0e-06
+  llm_eps: 1.0e-06
+  frame_selector_eps: 1.0e-06
+  temporal_token_scorer_eps: 1.0e-06
+  action_expert_eps: 1.0e-06
+  metrics_log_interval: -1
+scheduler:
+  name: multimodal
+  units: steps
+  t_warmup: 100
+  t_max: null
+  alpha_f: 0.1
+  connector_t_warmup: 200
+  vit_t_warmup: 200
+  llm_t_warmup: 2000
+  frame_selector_t_warmup: 200
+  temporal_token_scorer_t_warmup: 200
+  action_expert_t_warmup: 200
+  grad_clip_warmup_steps: null
+  grad_clip_warmup_factor: null
+  warmup_min_lr: 0.0
+data:
+  dataset: null
+  mixture:
+    synthmanip/task_0: 1.0
+  root_size_mixture: null
+  kwargs_mixture: null
+  split: train
+  seed: 50189
+  pad: to_max
+  sequence_length: 928
+  max_text_seq_len: null
+  shuffle: true
+  start_index: 0
+  packing: null
+  enable_variable_sized_token_pooling: true
+  num_workers: 4
+  drop_last: true
+  pin_memory: true
+  prefetch_factor: 4
+  persistent_workers: false
+  timeout: 300
+action_data: null
+action_loader_rate: null
+action_batch_interval: 1
+restore_dataloader: true
+fast_forward_batches: null
+evaluators:
+- label: synthmanip_val
+  data:
+    dataset: synthmanip/task_0
+    mixture: null
+    root_size_mixture: null
+    kwargs_mixture: null
+    split: val
+    seed: 691203
+    pad: to_max
+    sequence_length: 928
+    max_text_seq_len: null
+    shuffle: false
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    num_workers: 2
+    drop_last: false
+    pin_memory: true
+    prefetch_factor: 4
+    persistent_workers: false
+    timeout: 300
+  device_batch_size: 16
+  subset_num_batches: null
+  max_examples: 200
+  console_log_interval: 10
+  response_logits_only: true
+  reduce_loss_metrics_manually: false
+eval_interval: 200
+inf_evaluators: []
+inf_eval_interval: 200
+eval_on_last_step: true
+eval_on_load: false
+eval_on: []
+save_folder: /ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k
+checkpointer_config:
+  save_thread_count: null
+  load_thread_count: null
+  pre_download: false
+  work_dir: null
+  throttle_uploads: false
+canceled_check_interval: 50
+save_interval: 2000
+save_at: null
+save_final_optim: false
+save_num_checkpoints_to_keep: 1
+checkpoint_retention_frequency: null
+save_final_unsharded_checkpoint: true
+save_interval_ephemeral: null
+save_overwrite: true
+load_path: null
+reset_optimizer_state: true
+reset_trainer_state: true
+initial_model_checkpoint: /vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot/ckpts/MolmoBot-DROID
+allow_resume: true
+max_duration: 16000
+global_train_batch_size: 32
+device_train_microbatch_size: 16
+max_grad_norm: 1.0
+multi_component_grad_norm: true
+batch_divisor: global_batch
+max_grad_norm_ratio: null
+precision: amp_bf16
+wandb:
+  project: molmobot_finetune
+  entity: ccwatson
+  group: null
+  name: lr_custom_16k
+  tags:
+  - watching
+  log_artifacts: false
+  rank_zero_only: true
+  log_interval: 20
+  allow_resume: true
+  finish_on_sigterm: true
+beaker_log_interval: 50
+speed_monitor:
+  window_size: 20
+  gpu_flops_available: null
+console_log_interval: 20
+enable_timing_logs: false
+gen1_gc_interval: 1
+compile:
+  mode: default
+  fullgraph: false
+  dynamic: false
+  backend: inductor
+activation_checkpointing: true
+fsdp:
+  fsdp2: true
+  precision: pure
+  use_orig_params: true
+  wrapping_strategy: null
+  sharding_strategy: FULL_SHARD
+  hybrid_sharding_num_model_replicas: null
+softmax_auxiliary_loss: false
+softmax_auxiliary_loss_scale: 0.0001
+response_logits_only: true
+saliency_score_loss_wt: null
+frame_score_loss_wt: null
+frame_score_loss_type: mse
+frame_score_loss_target: 0.7
+time_limit: null
+extra_steps_after_cancel: 0
+python_profiling: false
+torch_profiling: false
+stop_at: 16000
+stop_after: null
+fused_loss: false
+compile_loss: true
+runtime_data:
+  args: launch_scripts/train_molmobot.py /vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot/ckpts/MolmoBot-DROID
+    --data_paths /vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot/scratch/datasets/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot
+    --stats_path=/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/MolmoBot/MolmoBot/scratch/datasets/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_norm_stats.yaml
+    --action_preset franka_joint --camera_preset franka_droid --seq_len=928 --n_obs_steps=2
+    --obs_step_delta=8 --model.mm_preprocessor.image.max_images=4 --model.mm_preprocessor.image.single_frame=False
+    --model.mm_preprocessor.max_frames=1 --ft_embedding=ae --model.num_flow_timestamps=8
+    --ft_llm=True --max_duration=16000 --device_batch_size=16 --global_batch_size=32
+    --num_workers=4 --prefetch_factor=4 --val_interval=200 --val_max_examples=200
+    --save_folder=/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k
+    --save_interval=2000 --save_num_checkpoints_to_keep=1 --checkpoint_retention_frequency=null
+    --save_final_unsharded_checkpoint=True --save_final_optim=False --reset_trainer_state=True
+    --reset_optimizer_state=True --optimizer.name=adamw --optimizer.connector_learning_rate=5e-6
+    --optimizer.vit_learning_rate=5e-6 --optimizer.llm_learning_rate=1e-5 --optimizer.action_expert_learning_rate=1e-4
+    --optimizer.connector_weight_decay=0.0 --optimizer.vit_weight_decay=0.0 --optimizer.llm_weight_decay=0.0
+    --optimizer.action_expert_weight_decay=0.0 --optimizer.connector_betas=[0.9,0.95]
+    --optimizer.vit_betas=[0.9,0.95] --optimizer.llm_betas=[0.9,0.95] --optimizer.action_expert_betas=[0.9,0.95]
+    --optimizer.connector_eps=1e-6 --optimizer.vit_eps=1e-6 --optimizer.llm_eps=1e-6
+    --optimizer.action_expert_eps=1e-6 --scheduler.connector_t_warmup=200 --scheduler.vit_t_warmup=200
+    --scheduler.llm_t_warmup=2000 --scheduler.frame_selector_t_warmup=200 --scheduler.temporal_token_scorer_t_warmup=200
+    --scheduler.action_expert_t_warmup=200 --scheduler.alpha_f=0.1 --scheduler.warmup_min_lr=0.0
+    --max_grad_norm=1 --exp_name=lr_custom_16k --weighted_sampling --img_aug --external_closed_loop_eval
+    --external_eval.enabled=True --external_eval.interval=2000 --external_eval.num_checkpoints_to_keep=2
+    --external_eval.result_poll_interval_steps=50 --external_eval.final_result_wait_seconds=7200
+    --external_eval.metric_for_best=success_rate --external_eval.metric_mode=max --external_eval.benchmark_dir=/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+    --external_eval.eval_config_cls=scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig
+    --external_eval.prompt_variant=natural_language --external_eval.num_workers=1
+    --external_eval.parallel_shards=10 --external_eval.camera_names=droid_shoulder_light_randomization
+    wrist_camera_zed_mini --external_eval.eval_renderer=filament --external_eval.eval_extra_args=--task_horizon_steps
+    500 --external_eval.model_server_extra_args=--args.execute_horizon 8 --args.states_mode
+    cross_attn --args.max_joint_delta 0.2 --external_eval.model_server_python= --external_eval.model_server_pythonpath=
+    --external_eval.eval_wandb_enabled=1 --external_eval.eval_wandb_project=mlspaces-json-eval
+    --external_eval.eval_wandb_group=lr-custom-ie-16k-b16-32 --external_eval.eval_wandb_tags=betty,vla-harness,molmospaces,intermediate-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-intermediate-eval,filament,10x
+    --external_eval.coordinator_poll_interval_seconds=30 --external_eval.final.enabled=True
+    --external_eval.final.required=1 --external_eval.final.wait_seconds=7200 --external_eval.final.upload_logs=1
+    --external_eval.final.eval_config_cls=scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig
+    --external_eval.final.prompt_variant=natural_language --external_eval.final.num_workers=1
+    --external_eval.final.parallel_shards=10 --external_eval.final.camera_names=droid_shoulder_light_randomization
+    wrist_camera_zed_mini --external_eval.final.eval_renderer=filament --external_eval.final.eval_extra_args=--task_horizon_steps
+    500 --external_eval.final.model_server_extra_args=--args.execute_horizon 8 --args.states_mode
+    cross_attn --args.max_joint_delta 0.2 --external_eval.final.model_server_python=
+    --external_eval.final.model_server_pythonpath= --external_eval.final.eval_wandb_enabled=1
+    --external_eval.final.eval_wandb_project=mlspaces-json-eval --external_eval.final.eval_wandb_group=lr-custom-fe-16k-b16-32
+    --external_eval.final.eval_wandb_tags=betty,vla-harness,molmospaces,final-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-final-eval,filament,10x
+    --external_eval.final.coordinator_poll_interval_seconds=30 --external_eval.final.server_port_base=38000
+    --external_eval.final.server_port_span=20000 --external_eval.final.repo_root=/vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot
+    --external_eval.final.molmobot_root=/vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot
+    --external_eval.final.molmospaces_root=/vast/projects/dineshj/lab/ccwatson/Documents/molmospaces
+    --external_eval.final.vla_harness_root=/vast/projects/dineshj/lab/ccwatson/Documents/vla-evaluation-harness
+    --external_eval.final.benchmark_dir=/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+    --external_eval.final.benchmarks=[{"benchmark_dir":"/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark","name":"FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench","eval_extra_args":"--task_horizon_steps
+    500"}]
+  hostname: dgx001
+  date: 07/17/2026, 01:01
+  world_size: 1
+  resuming_from: null
+  beaker_experiment_id: null
+  beaker_experiment_url: null
+  wandb_id: ouuzdjlg
+  wandb_url: https://wandb.ai/ccwatson/molmobot_finetune/runs/ouuzdjlg
+external_eval:
+  enabled: true
+  interval: 2000
+  request_dir: null
+  result_dir: null
+  run_root: null
+  log_root: null
+  result_poll_interval_steps: 50
+  final_result_wait_seconds: 7200
+  metric_for_best: success_rate
+  metric_mode: max
+  num_checkpoints_to_keep: 2
+  wait_for_results: false
+  benchmark_dir: /vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+  eval_config_cls: scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig
+  prompt_variant: natural_language
+  num_workers: 1
+  parallel_shards: 10
+  camera_names: droid_shoulder_light_randomization wrist_camera_zed_mini
+  eval_renderer: filament
+  eval_extra_args: --task_horizon_steps 500
+  model_server_extra_args: --args.execute_horizon 8 --args.states_mode cross_attn
+    --args.max_joint_delta 0.2
+  model_server_python: null
+  model_server_pythonpath: null
+  eval_wandb_enabled: true
+  eval_wandb_project: mlspaces-json-eval
+  eval_wandb_group: lr-custom-ie-16k-b16-32
+  eval_wandb_tags: betty,vla-harness,molmospaces,intermediate-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-intermediate-eval,filament,10x
+  coordinator_poll_interval_seconds: 30
+  final:
+    enabled: true
+    required: true
+    wait_seconds: 7200
+    upload_logs: true
+    request_dir: null
+    result_dir: null
+    run_root: null
+    log_root: null
+    benchmarks:
+    - benchmark_dir: /vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+      name: FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench
+      eval_config_cls: null
+      prompt_variant: null
+      num_workers: null
+      parallel_shards: null
+      camera_names: null
+      eval_renderer: null
+      eval_extra_args: --task_horizon_steps 500
+      model_server_extra_args: null
+    benchmark_dir: /vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+    eval_config_cls: scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig
+    prompt_variant: natural_language
+    num_workers: 1
+    parallel_shards: 10
+    camera_names: droid_shoulder_light_randomization wrist_camera_zed_mini
+    eval_renderer: filament
+    eval_extra_args: --task_horizon_steps 500
+    model_server_extra_args: --args.execute_horizon 8 --args.states_mode cross_attn
+      --args.max_joint_delta 0.2
+    eval_wandb_enabled: true
+    eval_wandb_project: mlspaces-json-eval
+    eval_wandb_group: lr-custom-fe-16k-b16-32
+    eval_wandb_tags: betty,vla-harness,molmospaces,final-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-final-eval,filament,10x
+    coordinator_poll_interval_seconds: 30
+    repo_root: /vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot
+    molmobot_root: /vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot
+    molmospaces_root: /vast/projects/dineshj/lab/ccwatson/Documents/molmospaces
+    vla_harness_root: /vast/projects/dineshj/lab/ccwatson/Documents/vla-evaluation-harness
+    molmospaces_python: null
+    model_server_script: null
+    model_server_python: null
+    model_server_pythonpath: null
+    server_port: null
+    server_port_base: 38000
+    server_port_span: 20000

metadata/external_eval_request.json

@@ -0,0 +1,92 @@
+{
+  "checkpoint_dir": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/step10000-unsharded",
+  "created_at": "2026-07-17T12:11:48.152584+00:00",
+  "external_eval": {
+    "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+    "camera_names": "droid_shoulder_light_randomization wrist_camera_zed_mini",
+    "coordinator_poll_interval_seconds": 30,
+    "enabled": true,
+    "eval_config_cls": "scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig",
+    "eval_extra_args": "--task_horizon_steps 500",
+    "eval_renderer": "filament",
+    "eval_wandb_enabled": true,
+    "eval_wandb_group": "lr-custom-ie-16k-b16-32",
+    "eval_wandb_project": "mlspaces-json-eval",
+    "eval_wandb_tags": "betty,vla-harness,molmospaces,intermediate-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-intermediate-eval,filament,10x",
+    "final": {
+      "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+      "benchmarks": [
+        {
+          "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+          "camera_names": null,
+          "eval_config_cls": null,
+          "eval_extra_args": "--task_horizon_steps 500",
+          "eval_renderer": null,
+          "model_server_extra_args": null,
+          "name": "FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench",
+          "num_workers": null,
+          "parallel_shards": null,
+          "prompt_variant": null
+        }
+      ],
+      "camera_names": "droid_shoulder_light_randomization wrist_camera_zed_mini",
+      "coordinator_poll_interval_seconds": 30,
+      "enabled": true,
+      "eval_config_cls": "scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig",
+      "eval_extra_args": "--task_horizon_steps 500",
+      "eval_renderer": "filament",
+      "eval_wandb_enabled": true,
+      "eval_wandb_group": "lr-custom-fe-16k-b16-32",
+      "eval_wandb_project": "mlspaces-json-eval",
+      "eval_wandb_tags": "betty,vla-harness,molmospaces,final-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-final-eval,filament,10x",
+      "log_root": null,
+      "model_server_extra_args": "--args.execute_horizon 8 --args.states_mode cross_attn --args.max_joint_delta 0.2",
+      "model_server_python": null,
+      "model_server_pythonpath": null,
+      "model_server_script": null,
+      "molmobot_root": "/vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot",
+      "molmospaces_python": null,
+      "molmospaces_root": "/vast/projects/dineshj/lab/ccwatson/Documents/molmospaces",
+      "num_workers": 1,
+      "parallel_shards": 10,
+      "prompt_variant": "natural_language",
+      "repo_root": "/vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot",
+      "request_dir": null,
+      "required": true,
+      "result_dir": null,
+      "run_root": null,
+      "server_port": null,
+      "server_port_base": 38000,
+      "server_port_span": 20000,
+      "upload_logs": true,
+      "vla_harness_root": "/vast/projects/dineshj/lab/ccwatson/Documents/vla-evaluation-harness",
+      "wait_seconds": 7200
+    },
+    "final_result_wait_seconds": 7200,
+    "interval": 2000,
+    "log_root": null,
+    "metric_for_best": "success_rate",
+    "metric_mode": "max",
+    "model_server_extra_args": "--args.execute_horizon 8 --args.states_mode cross_attn --args.max_joint_delta 0.2",
+    "model_server_python": null,
+    "model_server_pythonpath": null,
+    "num_checkpoints_to_keep": 2,
+    "num_workers": 1,
+    "parallel_shards": 10,
+    "prompt_variant": "natural_language",
+    "request_dir": null,
+    "result_dir": null,
+    "result_poll_interval_steps": 50,
+    "run_root": null,
+    "wait_for_results": false
+  },
+  "run_name": "lr_custom_16k",
+  "save_folder": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k",
+  "schema_version": 1,
+  "sharded_checkpoint_dir": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/step10000",
+  "step": 10000,
+  "training_wandb": {
+    "id": "ouuzdjlg",
+    "url": "https://wandb.ai/ccwatson/molmobot_finetune/runs/ouuzdjlg"
+  }
+}

metadata/external_eval_result.json

@@ -0,0 +1,168 @@
+{
+  "aggregate_results_path": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/external_eval/runs/step10000/FrankaJointPosVlaHarnessEvalConfig/external-eval-lr_custom_16k-step10000/aggregate_results.json",
+  "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+  "benchmark_name": null,
+  "checkpoint_dir": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/step10000-unsharded",
+  "checkpoint_format": "molmobot_unsharded",
+  "checkpoint_path": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/step10000-unsharded",
+  "duration_seconds": 2016.921890544938,
+  "episode_count": 40,
+  "eval_kind": "intermediate",
+  "eval_returncode": 0,
+  "final_eval_index": null,
+  "final_eval_label": null,
+  "final_eval_name": null,
+  "final_eval_wandb_prefix": null,
+  "finished_at": "2026-07-17T12:45:30.410588+00:00",
+  "logs": {
+    "eval_driver": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/external_eval/logs/step10000/eval_driver.log",
+    "model_server": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/external_eval/logs/step10000/model_server.log"
+  },
+  "metrics": {
+    "avg_episode_length": 274.675,
+    "avg_successful_episode_length": 140.08,
+    "completed_episode_count": 40,
+    "coverage_adjusted_success_rate": 0.625,
+    "duplicate_episode_count": 0,
+    "episode_count": 40,
+    "expected_episode_count": 40,
+    "failed_shard_count": 0,
+    "failure_count": 15,
+    "house_success_rates": {
+      "house_1088": 0.625
+    },
+    "lax_failure_count": 14,
+    "lax_house_success_rates": {
+      "house_1088": 0.65
+    },
+    "lax_oracle_done_count": 27,
+    "lax_oracle_done_rate": 0.675,
+    "lax_oracle_done_total_episodes": 40,
+    "lax_success_count": 26,
+    "lax_success_rate": 0.65,
+    "lax_total_episodes": 40,
+    "max_episode_length": 499,
+    "min_episode_length": 32,
+    "missing_episode_count": 0,
+    "num_houses": 1,
+    "oracle_done_count": 25,
+    "oracle_done_rate": 0.625,
+    "success_count": 25,
+    "success_rate": 0.625,
+    "total_episodes": 40,
+    "unexpected_result_count": 0
+  },
+  "output_root": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/external_eval/runs/step10000",
+  "request": {
+    "checkpoint_dir": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/step10000-unsharded",
+    "created_at": "2026-07-17T12:11:48.152584+00:00",
+    "external_eval": {
+      "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+      "camera_names": "droid_shoulder_light_randomization wrist_camera_zed_mini",
+      "coordinator_poll_interval_seconds": 30,
+      "enabled": true,
+      "eval_config_cls": "scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig",
+      "eval_extra_args": "--task_horizon_steps 500",
+      "eval_renderer": "filament",
+      "eval_wandb_enabled": true,
+      "eval_wandb_group": "lr-custom-ie-16k-b16-32",
+      "eval_wandb_project": "mlspaces-json-eval",
+      "eval_wandb_tags": "betty,vla-harness,molmospaces,intermediate-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-intermediate-eval,filament,10x",
+      "final": {
+        "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+        "benchmarks": [
+          {
+            "benchmark_dir": "/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark",
+            "camera_names": null,
+            "eval_config_cls": null,
+            "eval_extra_args": "--task_horizon_steps 500",
+            "eval_renderer": null,
+            "model_server_extra_args": null,
+            "name": "FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench",
+            "num_workers": null,
+            "parallel_shards": null,
+            "prompt_variant": null
+          }
+        ],
+        "camera_names": "droid_shoulder_light_randomization wrist_camera_zed_mini",
+        "coordinator_poll_interval_seconds": 30,
+        "enabled": true,
+        "eval_config_cls": "scripts.eval.vla_eval_harness_adapter:FrankaJointPosVlaHarnessEvalConfig",
+        "eval_extra_args": "--task_horizon_steps 500",
+        "eval_renderer": "filament",
+        "eval_wandb_enabled": true,
+        "eval_wandb_group": "lr-custom-fe-16k-b16-32",
+        "eval_wandb_project": "mlspaces-json-eval",
+        "eval_wandb_tags": "betty,vla-harness,molmospaces,final-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-final-eval,filament,10x",
+        "log_root": null,
+        "model_server_extra_args": "--args.execute_horizon 8 --args.states_mode cross_attn --args.max_joint_delta 0.2",
+        "model_server_python": null,
+        "model_server_pythonpath": null,
+        "model_server_script": null,
+        "molmobot_root": "/vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot/MolmoBot",
+        "molmospaces_python": null,
+        "molmospaces_root": "/vast/projects/dineshj/lab/ccwatson/Documents/molmospaces",
+        "num_workers": 1,
+        "parallel_shards": 10,
+        "prompt_variant": "natural_language",
+        "repo_root": "/vast/projects/dineshj/lab/ccwatson/Documents/MolmoBot",
+        "request_dir": null,
+        "required": true,
+        "result_dir": null,
+        "run_root": null,
+        "server_port": null,
+        "server_port_base": 38000,
+        "server_port_span": 20000,
+        "upload_logs": true,
+        "vla_harness_root": "/vast/projects/dineshj/lab/ccwatson/Documents/vla-evaluation-harness",
+        "wait_seconds": 7200
+      },
+      "final_result_wait_seconds": 7200,
+      "interval": 2000,
+      "log_root": null,
+      "metric_for_best": "success_rate",
+      "metric_mode": "max",
+      "model_server_extra_args": "--args.execute_horizon 8 --args.states_mode cross_attn --args.max_joint_delta 0.2",
+      "model_server_python": null,
+      "model_server_pythonpath": null,
+      "num_checkpoints_to_keep": 2,
+      "num_workers": 1,
+      "parallel_shards": 10,
+      "prompt_variant": "natural_language",
+      "request_dir": null,
+      "result_dir": null,
+      "result_poll_interval_steps": 50,
+      "run_root": null,
+      "wait_for_results": false
+    },
+    "run_name": "lr_custom_16k",
+    "save_folder": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k",
+    "schema_version": 1,
+    "sharded_checkpoint_dir": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/step10000",
+    "step": 10000,
+    "training_wandb": {
+      "id": "ouuzdjlg",
+      "url": "https://wandb.ai/ccwatson/molmobot_finetune/runs/ouuzdjlg"
+    }
+  },
+  "resolved_output_dir": "/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs/lr_custom_16k/external_eval/runs/step10000/FrankaJointPosVlaHarnessEvalConfig/external-eval-lr_custom_16k-step10000",
+  "schema_version": 1,
+  "server": {
+    "host": "127.0.0.1",
+    "port": 29253,
+    "port_base": 18000,
+    "port_span": 20000
+  },
+  "started_at": "2026-07-17T12:11:53.488701+00:00",
+  "status": "succeeded",
+  "step": 10000,
+  "summary_source": "aggregate_results_json",
+  "wandb": {
+    "enabled": true,
+    "group": "lr-custom-ie-16k-b16-32",
+    "name": "lr_custom_16k external eval step 10000",
+    "project": "mlspaces-json-eval",
+    "run_id": "external-eval-lr_custom_16k-step10000",
+    "tags": "betty,vla-harness,molmospaces,intermediate-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-intermediate-eval,filament,10x"
+  }
+}

metadata/provenance.json

@@ -0,0 +1,150 @@
+{
+  "schema_version": 1,
+  "model_repo": "ccwatson/MolmoBot-DROID-Directional-PickPlace-SimpleDesk-Step-10000",
+  "checkpoint": {
+    "run_name": "lr_custom_16k",
+    "step": 10000,
+    "intended_final_step": 16000,
+    "format": "molmobot_unsharded",
+    "files": {
+      "model.pt": {
+        "size_bytes": 19992166548,
+        "sha256": null,
+        "hash_note": "The authoritative content hash is recorded by Hugging Face after upload."
+      },
+      "config.yaml": {
+        "size_bytes": 25063,
+        "sha256": "7eeb51134b93fc17955f3bd850240074d6b9faf3dcf527177861940c7c098400"
+      },
+      "normalization_stats.yaml": {
+        "size_bytes": 1083,
+        "sha256": "972a88e10f02976ce5a4d1c90a2070a62e4daa08a5fe1cecb4ca00e8216eb15d"
+      }
+    },
+    "created_at_utc": "2026-07-17T12:11:48.152584+00:00"
+  },
+  "base_model": {
+    "repo_id": "allenai/MolmoBot-DROID",
+    "revision": "cbe6ec358958d07ddfb20d3aa54e560e9e1b18c9",
+    "license": "apache-2.0"
+  },
+  "dataset": {
+    "repo_id": "ccwatson/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot",
+    "revision": "5446191605f7eb3020983d83f707153d9719ecfb",
+    "source_valid_episode_count": 40,
+    "train_episode_count": 36,
+    "train_trajectory_count": 180,
+    "val_episode_count": 4,
+    "val_trajectory_count": 20,
+    "attempts_per_episode": 5,
+    "cameras": [
+      "droid_shoulder_light_randomization",
+      "wrist_camera_zed_mini"
+    ],
+    "normalization": {
+      "computed_from_split": "train",
+      "action_norm_mode": "quantiles",
+      "state_norm_mode": "min_max",
+      "action_dim": 8,
+      "state_dim": 8
+    }
+  },
+  "training": {
+    "wandb_entity": "ccwatson",
+    "wandb_project": "molmobot_finetune",
+    "wandb_run_id": "ouuzdjlg",
+    "wandb_url": "https://wandb.ai/ccwatson/molmobot_finetune/runs/ouuzdjlg",
+    "slurm_job_id": "7107997",
+    "slurm_job_name": "lr-custom-ie-fe-16k-b16-32",
+    "slurm_partition": "dgx-b200",
+    "slurm_start_time": "2026-07-17T00:59:31-04:00",
+    "resources": {
+      "nodes": 1,
+      "gpus": 1,
+      "gpu_type": "NVIDIA B200",
+      "cpus": 28,
+      "host_memory_gib": 224
+    },
+    "device_batch_size": 16,
+    "global_batch_size": 32,
+    "sequence_length": 928,
+    "observation_steps": 2,
+    "observation_step_delta": 8,
+    "max_images": 4,
+    "max_frames": 1,
+    "single_frame": false,
+    "weighted_sampling": true,
+    "randomize_prompts": false,
+    "image_augmentation": true,
+    "finetune_llm": true,
+    "finetune_embedding": "ae",
+    "num_flow_timestamps": 8,
+    "action_preset": "franka_joint",
+    "camera_preset": "franka_droid",
+    "learning_rates": {
+      "connector": 5e-6,
+      "vit": 5e-6,
+      "llm": 1e-5,
+      "action_expert": 1e-4
+    },
+    "validation_interval_steps": 200,
+    "validation_max_examples": 200,
+    "checkpoint_interval_steps": 2000,
+    "external_eval_interval_steps": 2000,
+    "reset_trainer_state": true,
+    "reset_optimizer_state": true
+  },
+  "evaluation": {
+    "kind": "intermediate",
+    "status": "succeeded",
+    "benchmark": "FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench",
+    "episode_count": 40,
+    "left_episode_count": 20,
+    "right_episode_count": 20,
+    "prompt_variant": "natural_language",
+    "renderer": "filament",
+    "parallel_shards": 10,
+    "task_horizon_steps": 500,
+    "execute_horizon": 8,
+    "states_mode": "cross_attn",
+    "max_joint_delta": 0.2,
+    "strict_success_count": 25,
+    "strict_success_rate": 0.625,
+    "lax_success_count": 26,
+    "lax_success_rate": 0.65,
+    "oracle_done_count": 25,
+    "oracle_done_rate": 0.625,
+    "lax_oracle_done_count": 27,
+    "lax_oracle_done_rate": 0.675,
+    "average_episode_length": 274.675,
+    "average_successful_episode_length": 140.08,
+    "min_episode_length": 32,
+    "max_episode_length": 499,
+    "failed_shard_count": 0,
+    "missing_episode_count": 0,
+    "duplicate_episode_count": 0,
+    "unexpected_result_count": 0,
+    "eval_returncode": 0,
+    "duration_seconds": 2016.921890544938,
+    "started_at": "2026-07-17T12:11:53.488701+00:00",
+    "finished_at": "2026-07-17T12:45:30.410588+00:00",
+    "wandb_project": "mlspaces-json-eval",
+    "wandb_group": "lr-custom-ie-16k-b16-32",
+    "wandb_run_id": "external-eval-lr_custom_16k-step10000",
+    "wandb_url": "https://wandb.ai/ccwatson/mlspaces-json-eval/runs/external-eval-lr_custom_16k-step10000"
+  },
+  "source_revisions": {
+    "molmo_robotics_stack": "0363218a1b019c35a8155c9cfbb847a9cba98a6c",
+    "molmobot": "c68c5ad159c48f9fc45c6ef3de4d6ff39242f97b",
+    "molmospaces": "ba251a63d1e5dd9ee4b0200f897cf6e3c446d320",
+    "vla_evaluation_harness": "ee4949157c0aa7dbb5207a5800224f251207886a",
+    "molmobot_worktree_was_dirty": true,
+    "note": "The exact launch env and resolved config are included because the MolmoBot worktree contained uncommitted orchestration/config changes."
+  },
+  "source_file_sha256": {
+    "train_molmobot.py": "4682dee5b6f8a2fb85c839f88108e2b2ce40a102077d1647c06d55a248983d64",
+    "slurm_train_molmobot_betty.sh": "c497cd4722352f4d1698ed578bd54a5a82342e3ab69d9307c8f5d131cd40f5ff",
+    "external_closed_loop_eval_coordinator.py": "64e2299ac78f50b0efa199a7a85368980d61731c421cd1716699e45dbed7a966",
+    "vla_eval_harness_adapter.py": "8407d8bc842584458de838c15beb563f5662f3699bb0be7ac54d6eaef4c904e9"
+  }
+}

metadata/training_launch.env

@@ -0,0 +1,102 @@
+# 16k MolmoBot-DROID finetune on the directional left/right SimpleDesk pick/place
+# dataset (200 demos, tested40 target set). Sharded intermediate eval and one
+# matching final eval on FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench.
+# No prompt randomization. FT_LLM=True. Eval uses 10 parent-managed MolmoSpaces shards.
+
+RUN_NAME=lr_custom_16k
+DATASET_ID=ccwatson/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot
+DATASET_LOCAL_NAME=directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot
+BASE_MODEL_ID=allenai/MolmoBot-DROID
+
+SAVE_ROOT=/ceph/projects/dineshj/lab/ccwatson/MolmoBot_runs
+WANDB_DIR=${SAVE_ROOT}/${RUN_NAME}/wandb
+WANDB_CACHE_DIR=${SAVE_ROOT}/${RUN_NAME}/.cache/wandb
+WANDB_ARTIFACT_DIR=${SAVE_ROOT}/${RUN_NAME}/wandb/artifacts
+WANDB_CONFIG_DIR=${SAVE_ROOT}/${RUN_NAME}/wandb/config
+STATS_PATH=/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/MolmoBot/MolmoBot/scratch/datasets/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_molmobot/directional_pick_place_simpledesk_diverse_targets_tested40_200demos_20260716_norm_stats.yaml
+RANDOMIZE_PROMPTS=0
+
+RUN_STEPS=16000
+DEVICE_BATCH_SIZE=16
+GLOBAL_BATCH_SIZE=32
+SAVE_INTERVAL=2000
+SAVE_NUM_CHECKPOINTS_TO_KEEP=1
+CHECKPOINT_RETENTION_FREQUENCY=null
+SAVE_FINAL_UNSHARDED_CHECKPOINT=True
+SAVE_FINAL_OPTIM=False
+RESET_TRAINER_STATE=True
+RESET_OPTIMIZER_STATE=True
+HF_UPLOAD=0
+
+ACTION_PRESET=franka_joint
+CAMERA_PRESET=franka_droid
+SEQ_LEN=928
+N_OBS_STEPS=2
+OBS_STEP_DELTA=8
+MAX_IMAGES=4
+MAX_FRAMES=1
+SINGLE_FRAME=False
+WEIGHTED_SAMPLING=1
+FT_EMBEDDING=ae
+NUM_FLOW_TIMESTAMPS=8
+FT_LLM=True
+IMG_AUG=1
+CONNECTOR_LR=5e-6
+VIT_LR=5e-6
+LLM_LR=1e-5
+ACTION_EXPERT_LR=1e-4
+NO_VAL=0
+VAL_INTERVAL=200
+VAL_MAX_EXAMPLES=200
+NUM_WORKERS=4
+PREFETCH_FACTOR=4
+WANDB_PROJECT=molmobot_finetune
+WANDB_ENTITY=ccwatson
+WANDB_MODE=online
+
+# Intermediate eval on the left/right SimpleDesk relative-next-to benchmark.
+EXTERNAL_EVAL_ENABLED=1
+EXTERNAL_EVAL_INTERVAL=2000
+EXTERNAL_EVAL_NUM_CHECKPOINTS_TO_KEEP=2
+EXTERNAL_EVAL_RESULT_POLL_INTERVAL_STEPS=50
+EXTERNAL_EVAL_FINAL_RESULT_WAIT_SECONDS=7200
+EXTERNAL_EVAL_METRIC_FOR_BEST=success_rate
+EXTERNAL_EVAL_METRIC_MODE=max
+
+EXTERNAL_EVAL_BENCHMARK_DIR=/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+EXTERNAL_EVAL_EXTRA_ARGS="--task_horizon_steps 500"
+EXTERNAL_EVAL_NUM_WORKERS=1
+EXTERNAL_EVAL_PARALLEL_SHARDS=10
+EXTERNAL_EVAL_RENDERER=filament
+EXTERNAL_EVAL_CAMERA_NAMES="droid_shoulder_light_randomization wrist_camera_zed_mini"
+
+EXTERNAL_EVAL_WANDB_ENABLED=1
+EXTERNAL_EVAL_WANDB_PROJECT=mlspaces-json-eval
+EXTERNAL_EVAL_WANDB_GROUP=lr-custom-ie-16k-b16-32
+EXTERNAL_EVAL_WANDB_TAGS=betty,vla-harness,molmospaces,intermediate-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-intermediate-eval,filament,10x
+
+# Final eval on the same left/right SimpleDesk benchmark.
+EXTERNAL_EVAL_FINAL_ENABLED=1
+EXTERNAL_EVAL_FINAL_REQUIRED=1
+EXTERNAL_EVAL_FINAL_WAIT_SECONDS=7200
+EXTERNAL_EVAL_FINAL_UPLOAD_LOGS=1
+EXTERNAL_EVAL_FINAL_NUM_WORKERS=1
+EXTERNAL_EVAL_FINAL_PARALLEL_SHARDS=10
+EXTERNAL_EVAL_FINAL_RENDERER=filament
+EXTERNAL_EVAL_FINAL_CAMERA_NAMES="droid_shoulder_light_randomization wrist_camera_zed_mini"
+EXTERNAL_EVAL_FINAL_EVAL_WANDB_ENABLED=1
+EXTERNAL_EVAL_FINAL_EVAL_WANDB_PROJECT=mlspaces-json-eval
+EXTERNAL_EVAL_FINAL_EVAL_WANDB_GROUP=lr-custom-fe-16k-b16-32
+EXTERNAL_EVAL_FINAL_EVAL_WANDB_TAGS=betty,vla-harness,molmospaces,final-eval,molmobot-droid,left-right-pick-place,simple-desk,relative-next-to,sharded-final-eval,filament,10x
+
+EXTERNAL_EVAL_FINAL_BENCHMARK_1_NAME=FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench
+EXTERNAL_EVAL_FINAL_BENCHMARK_1_DIR=/vast/projects/dineshj/lab/ccwatson/Documents/molmo-robotics-stack/components/molmospaces/benchmarks/custom/molmospaces-bench-v2/procthor-objaverse/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench/FrankaPickAndPlaceRelativeNextToSimpleDeskDiverseTargetsTested40Bench_20260716_json_benchmark
+EXTERNAL_EVAL_FINAL_BENCHMARK_1_EXTRA_ARGS="--task_horizon_steps 500"
+
+SLURM_JOB_NAME=lr-custom-ie-fe-16k-b16-32
+SLURM_PARTITION=dgx-b200
+SLURM_QOS=normal
+SLURM_GRES=gpu:1
+SLURM_CPUS=28
+SLURM_MEM=224G
+SLURM_TIME=20:00:00

normalization_stats.yaml

@@ -0,0 +1,46 @@
+stats_by_repo:
+  synthmanip:
+    observation.state:
+      min:
+      - -2.2766549587249756
+      - -1.7608288526535034
+      - -1.7191452980041504
+      - -2.8627865314483643
+      - -0.9076813459396362
+      - 0.8232163190841675
+      - -2.4616048336029053
+      - -0.0008297820459119976
+      max:
+      - 2.080810546875
+      - 0.6704062223434448
+      - 1.9463512897491455
+      - -0.6767909526824951
+      - 0.8670341372489929
+      - 2.381640672683716
+      - 0.4024277329444885
+      - 0.8239965438842773
+    action:
+      q01:
+      - -1.3642053604125977
+      - -1.5828913450241089
+      - -0.7952697277069092
+      - -2.813814878463745
+      - -0.8428052663803101
+      - 1.053465485572815
+      - -2.1978600025177
+      - 0.0
+      q99:
+      - 0.7378915548324585
+      - 0.468648225069046
+      - 1.7082799673080444
+      - -0.9352492094039917
+      - 0.7082295417785645
+      - 2.1873221397399902
+      - 0.3892104923725128
+      - 255.0
+default_repo_id: synthmanip
+action_key: action
+state_keys:
+- observation.state
+action_norm_mode: quantiles
+state_norm_mode: min_max