Upload folder using huggingface_hub

README.md

@@ -1,3 +1,190 @@
----
-license: apache-2.0
----
+<div align="center">
+
+#  | *FALCON* | From Spatial to Actions: <br>Grounding Vision-Language-Action Model in Spatial Foundation Priors (ICLR 2026)
+
+<a href="https://arxiv.org/abs/2510.17439" target="_blank">
+    <img alt="arXiv" src="https://img.shields.io/badge/arXiv-FALCON-red?logo=arxiv" height="25" />
+</a>
+<a href="https://falcon-vla.github.io/" target="_blank">
+    <img alt="Website" src="https://img.shields.io/badge/🌎_Website-falcon.io-blue.svg" height="25" />
+</a>
+<a href="https://github.com/FALCON-VLA" target="_blank">
+    <img alt="GitHub Code: FALCON" src="https://img.shields.io/badge/Code-FALCON-181717?logo=github&logoColor=white" height="25" />
+</a>
+<a href="https://huggingface.co/papers/2510.17439" target="_blank">
+    <img alt="HF Paper: FALCON" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Paper-FALCON-ffc107?color=ffc107&logoColor=white" height="25" />
+</a>
+<!-- <a href="https://huggingface.co/datasets/robovlms/bytedance_robot_benchmark_20" target="_blank">
+    <img alt="HF Dataset: BDRBench-20" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Dataset-BDRBench20-ffc107?color=ffc107&logoColor=white" height="25" />
+</a> -->
+<br>
+<a href="https://www.python.org/" target="_blank">
+    <img alt="Python 3.8" src="https://img.shields.io/badge/Python-%3E=3.8-blue" height="25" />
+</a>
+<a href="https://pytorch.org/" target="_blank">
+    <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-%3E=2.1-orange" height="25" />
+</a>
+
+</div>
+
+<div align="center">
+    <br>
+<div style="text-align: center;">
+    <a href="https://scholar.google.com/citations?user=8nrJ1vsAAAAJ&hl=en"  target="_blank">Zhengshen Zhang</a> &emsp;
+    <a href="https://scholar.google.com/citations?user=4dokjDoAAAAJ&hl=zh-CN"  target="_blank">Hao Li</a> &emsp;
+    <a href="https://scholar.google.com/citations?user=6XyNVowAAAAJ&hl=en"  target="_blank">Yalun Dai</a> &emsp;
+    <a href="https://scholar.google.com/citations?user=ozatRA0AAAAJ&hl=zh-CN"  target="_blank">Zhengbang Zhu</a> &emsp;
+    <a href="https://scholar.google.com/citations?user=VhToj4wAAAAJ&hl=zh-CN"  target="_blank">Lei Zhou</a> &emsp;
+    <br>
+    <a href="https://sg.linkedin.com/in/liu-chenchen"  target="_blank">Chenchen Liu</a> &emsp;
+    <a href=""  target="_blank">Dong Wang</a> &emsp;
+    <a href="https://scholar.google.com/citations?user=mfH9UFIAAAAJ&hl=en"  target="_blank">Francis E. H. Tay</a> &emsp;
+    <a href="https://ch3cook-fdu.github.io/"  target="_blank">Sijin Chen</a> &emsp;
+    <br>
+    <a href="https://liuziwei7.github.io/"  target="_blank">Ziwei Liu</a> &emsp;
+    <a href="https://scholar.google.com/citations?user=i8wNtSgAAAAJ&hl=en"  target="_blank">Yuxiao Liu</a><sup>*</sup><sup>&dagger;</sup> &emsp;
+    <a href="https://scholar.google.com/citations?user=laOWyTQAAAAJ&hl=zh-CN"  target="_blank">Xinghang Li</a><sup>*</sup> &emsp;
+    <a href="https://panzhous.github.io/"  target="_blank">Pan Zhou</a><sup>*</sup> &emsp;
+    <br>
+    <p style="text-align: center; margin-bottom: 0;">
+        <span class="author-note"><sup>*</sup>Corresponding Author</span>&emsp;
+        <span class="author-note"><sup>&dagger;</sup>Project Lead</span>
+    </p>
+<br>
+<p style="text-align: center;">
+    ByteDance Seed <br> 
+    National University of Singapore &emsp; Nanyang Technological University <br>
+    Tsinghua University &emsp; Singapore Management University</p>
+</div>
+</div>
+
+
+## 🚀 Introduction
+Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders, leaving a spatial reasoning gap that limits generalization and adaptability. In this work, we introduce **FALCON (From Spatial to Action)**, a novel paradigm that injects rich 3D spatial tokens into the action head of a VLA model, enabling robust spatial understanding and SOTA performance across diverse manipulation tasks without disrupting vision-language alignment. See our paper at [here](https://arxiv.org/abs/2510.17439).
+
+## 🤗 Model Zoo
+We provide the following model weights and their config files in our paper:
+
+<table>
+  <tr>
+    <th>Model Name</th>
+    <th>VLA Model</th>
+    <th>Embodied Spatial Model</th>
+    <th>Note</th>
+  </tr>
+  <tr>
+    <td>FALCON-FC-CALVIN-ABC</td>
+    <td><a href="">falcon-esm-fc-calvin-abc-pt</a></td>
+    <td><a href="">esm-1b</a></td>
+    <td>finetune on calvin-abc with RGB inputs to ESM, Tab. 4 and 5.</td>
+  </tr>
+  <tr>
+    <td>FALCON-FC-CALVIN-ABC-WDepth</td>
+    <td><a href="">falcon-esm-fc-calvin-abc-wdepth-pt</a></td>
+    <td><a href="">esm-1b</a></td>
+    <td>finetune on calvin-abc with RGB-D inputs to ESM, Tab. 5.</td>
+  </tr>
+  <tr>
+    <td>FALCON-3DPC-FC-CALVIN-ABC</td>
+    <td><a href="">falcon-3dpc-fc-calvin-abc-pt</a></td>
+    <td><a href="https://github.com/YanjieZe/Improved-3D-Diffusion-Policy">improved DP3 encoder</a></td>
+    <td>finetune on calvin-abc with point cloud inputs to idp3 encoder, Tab. 5-Kosmos-VLA <i>(w/ rgb-d)</i>.</td>
+  </tr>
+  <tr>
+    <td>FALCON-LSTM-CALVIN-ABC</td>
+    <td><a href="">falcon-lstm-calvin-abc-pt</a></td>
+    <td><a href="">esm-1b</a></td>
+    <td>finetune on calvin-abc with RGB inputs to ESM, Tab. 1.</td>
+  </tr>
+  <tr>
+    <td>FALCON-LSTM-CALVIN-ABCD</td>
+    <td><a href="">falcon-lstm-calvin-abcd-pt</a></td>
+    <td><a href="">esm-1b</a></td>
+    <td>finetune on calvin-abcd with RGB inputs to ESM, Tab. 1.</td>
+  </tr>
+  <tr>
+    <td>FALCON-FC-SimplerEnv-Bridge</td>
+    <td><a href="">falcon-fc-simpler-bridge-pt</a></td>
+    <td><a href="">esm-1b</a></td>
+    <td>pretrained on oxe then finetune on bridge dataset with RGB inputs to ESM, Tab. 2.</td>
+  </tr>
+  <tr>
+    <td>FALCON-FC-SimplerEnv-Fractal</td>
+    <td><a href="">falcon-fc-simpler-fractal-pt</a></td>
+    <td><a href="">esm-1b</a></td>
+    <td>pretrained on oxe then finetune on fractal dataset with RGB inputs to ESM, Tab. 3.</td>
+  </tr>
+</table>
+
+## 📦 Usage
+FALCON can be used to predict action based on the vision and language input. FALCON supports several VLA structures, multi-view input, and multi-sensory input (RGB, RGB-D, point cloud). Taking `FALCON-FC-CALVIN-ABC` as an example:
+
+```python
+import torch
+import json, functools
+from PIL import Image
+from copy import deepcopy
+from falcon.train.base_trainer import BaseTrainer
+from falcon.data.data_utils import preprocess_image
+from falcon.data.data_utils import get_text_function
+from falcon.model.policy_head.esm_utils.vggt.utils.load_fn import load_and_preprocess_images_square_new
+
+configs = josn.load(open('configs/falcon-esm-fc-calvin-abc.json', 'r'))
+pretrained_path = 'checkpoints/falcon-esm-fc-calvin-abc-pt'
+configs['model_load_path'] = pretrained_path
+
+model = BaseTrainer.from_checkpoint(configs)
+
+image_fn = functools.partial(
+    preprocess_image,
+    image_processor=model.model.image_processor,
+    model_type=configs["model"],
+)
+text_fn = get_text_function(model.model.tokenizer, configs["model"])
+prompt = "Task: pull the handle to open the drawer"
+text_tensor, attention_mask = text_fn([prompt])
+
+for step in range(MAX_STEPS):
+    image: Image.Image = get_from_side_camera(...)
+    # get inputs for esm
+    image_vggt = deepcopy(image)
+    image = image_fn([image]).unsqueeze(0)
+
+    esm_target_size = 224
+    image_vggt_x, _ = load_and_preprocess_images_square_new([image_vggt], target_size=esm_target_size)
+    image_vggt_x = image_vggt_x.unsqueeze(0)
+    
+    input_dict["rgb"] = image
+    input_dict["text"] = text_tensor
+    input_dict['text_mask'] = attention_mask
+    input_dict["rgb_vggt"] = image_vggt_x
+
+    ### if wrist camera is available
+    wrist_image: Image.Image = get_from_wrist_camera(...)
+    wrist_image = image_fn([wrist_image]).unsqueeze(0)
+    input_dict["hand_rgb"] = wrist_image
+
+    with torch.no_grad():
+      action = model.inference_step(input_dict)["action"]
+    print(action)
+```  
+
+## 🤗 FAQs
+If you encounter any issues, feel free to open an issue or reach out through discussions. We appreciate your feedback and contributions! 🚀
+
+## 🖊️ Citation
+If you find this project useful in your research, please consider cite:
+```BibTeX
+@article{zhang2025spatial,
+  title={From spatial to actions: Grounding vision-language-action model in spatial foundation priors},
+  author={Zhang, Zhengshen and Li, Hao and Dai, Yalun and Zhu, Zhengbang and Zhou, Lei and Liu, Chenchen and Wang, Dong and Tay, Francis EH and Chen, Sijin and Liu, Ziwei and others},
+  journal={arXiv preprint arXiv:2510.17439},
+  year={2025}
+}
+```
+
+## 🪪 License 
+All FALCON checkpoints, as well as our [codebase](https://github.com/FALCON-VLA) are released under the Apache-2.0 License.
+
+## ❤️ Acknowledgement
+FALCON is built with reference to the code of the following projects: [RoboVLMs](https://github.com/Robot-VLAs/RoboVLMs/tree/main?tab=readme-ov-file), [Microsoft Kosmos-2](https://github.com/microsoft/unilm/tree/master/kosmos-2), [VGGT](https://github.com/facebookresearch/vggt), and [ManiUniCon](https://github.com/Universal-Control/ManiUniCon). Thanks for their awesome work!

esm/esm.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:336d305e2ad09a37cf16e9c820ec6bd99cee6490eee558715b76dd8b3c09477a
+size 4633579216

falcon-3dpc-fc-calvin-abc/ckpts/falcon_3dpc_fc_calvin_abc.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6884c55d4d99797bc11fee026b96b70125b95c4a57542fae20afe682f8424dd6
+size 6726858178

falcon-3dpc-fc-calvin-abc/logs/logs.json

@@ -0,0 +1,172 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "fc_calvin_finetune_vlm_pcd_injector_lr5e-5_linear-twin-v1_pre-fusion_fp32_pcd-norm",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 1,
+    "fwd_pred_next_n": 10,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 4,
+    "num_workers": 32,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 5e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0.25,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": true,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "/mnt/bn/zhengshen-lq/runs/cache/kosmos",
+    "model_load_path": "/mnt/bn/zhengshen-lq/runs/checkpoints/kosmos/fc_calvin_finetune_pcd_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_fp32_pcd-norm/2025-06-17/19-06/epoch=3-step=32932.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "32",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_pcd_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "FCDecoder_3D",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_pcd_injector": true,
+        "use_hand_pcd": false,
+        "pcd_n_points": 2048
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "32",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 10,
+        "val_check_interval": null,
+        "check_val_every_n_epoch": 1,
+        "max_steps": -1,
+        "accumulate_grad_batches": 1
+    },
+    "train_dataset": {
+        "type": "DiskCalvinDataset3D",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/training",
+        "pcd_data_dir": "/mnt/bn/zhengshen-lq/dataset/calvin_data/full_traj_pcds/packaged_ABC_D/training",
+        "shift_first": false,
+        "model_name": "kosmos",
+        "rgb_pad": 10,
+        "gripper_pad": 4,
+        "pcd_augment": false,
+        "jitter_sigma": 0.005,
+        "jitter_clip": 0.02,
+        "dropout_ratio": 0.1
+    },
+    "val_dataset": {
+        "type": "DiskCalvinDataset3D",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/validation",
+        "pcd_data_dir": "/mnt/bn/zhengshen-lq/dataset/calvin_data/full_traj_pcds/packaged_ABC_D/validation",
+        "model_name": "kosmos"
+    },
+    "norm_action": true,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/calvin_finetune/pcd_inject_test/vlm_unfreeze_head_freeze/fc/finetune_kosmos_cont-fc-post-pcd-norm-aug_part-ft_vlm_pcd-injector_wd-0_use-hand_ws-1_act-10.json",
+    "config": "configs/calvin_finetune/pcd_inject_test/vlm_unfreeze_head_freeze/fc/finetune_kosmos_cont-fc-post-pcd-norm-aug_part-ft_vlm_pcd-injector_wd-0_use-hand_ws-1_act-10.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "/mnt/bn/zhengshen-lq/runs/logs/kosmos/fc_calvin_finetune_vlm_pcd_injector_lr5e-5_linear-twin-v1_pre-fusion_fp32_pcd-norm/2025-06-19/01-47",
+    "output_dir": "/mnt/bn/zhengshen-lq/runs/checkpoints/kosmos/fc_calvin_finetune_vlm_pcd_injector_lr5e-5_linear-twin-v1_pre-fusion_fp32_pcd-norm/2025-06-19/01-47",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "01-47",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}

falcon-esm-fc-calvin-abc-wdepth/ckpts/falcon_esm_fc_calvin_abc_wdepth.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f026a7bd00a154e7db061398753737003f97e89aa2c111b87070e9afd3d4160
+size 15997622110

falcon-esm-fc-calvin-abc-wdepth/logs/logs.json

@@ -0,0 +1,168 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "fc-esm-gt-depth_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_fp32",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 1,
+    "fwd_pred_next_n": 10,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 4,
+    "num_workers": 32,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 5e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0.25,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": true,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "/mnt/bn/zhengshen-lq2/runs/cache/kosmos",
+    "model_load_path": "/mnt/bn/zhengshen-lq2/runs/checkpoints/kosmos/fc-esm-gt-depth_calvin_finetune_spatial_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_fp32/2025-09-12/01-00/epoch=1-step=16466.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "32",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_spatial_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "FCDecoder_ESM",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_spatial_injector": true,
+        "esm_use_hand_rgb": false,
+        "camera_gt_pt": 0.0,
+        "depth_gt_pt": 1.0,
+        "esm_ckpt_path": "/mnt/bn/zhengshen-lq2/falcon_pub/esm/esm.safetensors"
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "32",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 10,
+        "val_check_interval": null,
+        "check_val_every_n_epoch": 1,
+        "max_steps": -1,
+        "accumulate_grad_batches": 1
+    },
+    "train_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/training",
+        "shift_first": false,
+        "model_name": "kosmos",
+        "rgb_pad": 10,
+        "gripper_pad": 4
+    },
+    "val_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/validation",
+        "model_name": "kosmos"
+    },
+    "norm_action": true,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/calvin_finetune/vlm_unfreeze_head_freeze/fc/finetune_kosmos_cont-fc-post_esm-use-gt_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-1_act-10.json",
+    "config": "configs/calvin_finetune/vlm_unfreeze_head_freeze/fc/finetune_kosmos_cont-fc-post_esm-use-gt_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-1_act-10.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "/mnt/bn/zhengshen-lq2/runs/logs/kosmos/fc-esm-gt-depth_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_fp32/2025-09-12/20-32",
+    "output_dir": "/mnt/bn/zhengshen-lq2/runs/checkpoints/kosmos/fc-esm-gt-depth_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_fp32/2025-09-12/20-32",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "20-32",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}

falcon-esm-fc-calvin-abc/ckpts/falcon_esm_fc_calvin_abc.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c99ff25a706224f9047ab0eb649b5c883af46f6a6ff2e12507ee8ec6bd4911b4
+size 15997622302

falcon-esm-fc-calvin-abc/logs/logs.json

@@ -0,0 +1,168 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "fc-esm-rgb_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_fp32",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 1,
+    "fwd_pred_next_n": 10,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 4,
+    "num_workers": 32,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 5e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0.25,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": true,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "/mnt/bn/zhengshen-lq2/runs/cache/kosmos",
+    "model_load_path": "/mnt/bn/zhengshen-lq2/runs/checkpoints/kosmos/fc-esm-rgb_calvin_finetune_spatial_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_fp32/2025-08-29/14-49/epoch=1-step=16466.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "32",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_spatial_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "FCDecoder_ESM",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_spatial_injector": true,
+        "esm_use_hand_rgb": false,
+        "camera_gt_pt": 0.0,
+        "depth_gt_pt": 0.0,
+        "esm_ckpt_path": "/mnt/bn/zhengshen-lq2/falcon_pub/esm/esm.safetensors"
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "32",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 10,
+        "val_check_interval": null,
+        "check_val_every_n_epoch": 1,
+        "max_steps": -1,
+        "accumulate_grad_batches": 1
+    },
+    "train_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/training",
+        "shift_first": false,
+        "model_name": "kosmos",
+        "rgb_pad": 10,
+        "gripper_pad": 4
+    },
+    "val_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/validation",
+        "model_name": "kosmos"
+    },
+    "norm_action": true,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/calvin_finetune/vlm_unfreeze_head_freeze/fc/finetune_kosmos_cont-fc-post_esm-rgb_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-1_act-10.json",
+    "config": "configs/calvin_finetune/vlm_unfreeze_head_freeze/fc/finetune_kosmos_cont-fc-post_esm-rgb_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-1_act-10.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "/mnt/bn/zhengshen-lq2/runs/logs/kosmos/fc-esm-rgb_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_fp32/2025-08-30/16-55",
+    "output_dir": "/mnt/bn/zhengshen-lq2/runs/checkpoints/kosmos/fc-esm-rgb_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_fp32/2025-08-30/16-55",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "16-55",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}

falcon-esm-fc-simpler-bridge/ckpts/falcon_esm_fc_simpler_bridge.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:350a1102fa52b158d6b70f4615b6f60e0c4eb612af33af13b0d25accb2b6cf05
+size 15997550686

falcon-esm-fc-simpler-bridge/logs/logs.json

@@ -0,0 +1,179 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "fc-esm-rgb_bridge-finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_fp32",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 1,
+    "fwd_pred_next_n": 5,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 4,
+    "num_workers": 16,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 2e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": false,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train/cache/kosmos",
+    "model_load_path": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train/checkpoints/kosmos/fc-esm-rgb_bridge-finetune_spatial_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_fp32/2025-08-26/21-16/epoch=0-step=50000.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "32",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_spatial_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "FCDecoder_ESM",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_spatial_injector": true,
+        "esm_use_hand_rgb": false,
+        "camera_gt_pt": 0.0,
+        "depth_gt_pt": 0.0,
+        "esm_ckpt_path": "/mnt/bn/zhengshen-lq2/falcon_pub/esm/esm.safetensors"
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "32",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 5,
+        "val_check_interval": 10000,
+        "check_val_every_n_epoch": null,
+        "max_steps": 200000,
+        "accumulate_grad_batches": 1,
+        "limit_val_batches": 100
+    },
+    "train_dataset": {
+        "type": "OpenVLADataset",
+        "data_root_dir": "/mnt/bn/robotics-data-lxh-lq/openvla/datasets/open-x-embodiment",
+        "model_name": "kosmos",
+        "image_aug": true,
+        "mode": "train",
+        "data_mix": "bridge",
+        "window_sample": "sliding",
+        "organize_type": "interleave",
+        "shuffle_buffer_size": 51200,
+        "train": true
+    },
+    "val_dataset": {
+        "type": "OpenVLADataset",
+        "data_root_dir": "/mnt/bn/robotics-data-lxh-lq/openvla/datasets/open-x-embodiment",
+        "model_name": "kosmos",
+        "mode": "train",
+        "data_mix": "bridge",
+        "window_sample": "sliding",
+        "organize_type": "interleave",
+        "shuffle_buffer_size": 10000,
+        "train": false
+    },
+    "norm_action": false,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/oxe_training/vlm_unfreeze_head_freeze/bridge/finetune_kosmos_cont-fc-post_esm-rgb_part-ft_vlm-spatial-injector_wd-0_ws-1_act-5_bridge_finetune.json",
+    "config": "configs/oxe_training/vlm_unfreeze_head_freeze/bridge/finetune_kosmos_cont-fc-post_esm-rgb_part-ft_vlm-spatial-injector_wd-0_ws-1_act-5_bridge_finetune.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train/logs/kosmos/fc-esm-rgb_bridge-finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_fp32/2025-08-30/23-37",
+    "output_dir": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train/checkpoints/kosmos/fc-esm-rgb_bridge-finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_fp32/2025-08-30/23-37",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "23-37",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}

falcon-esm-fc-simpler-gr/ckpts/falcon_esm_fc_simpler_gr.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:845745b1048ae90d3d09946dbea44d546e3bb512efeeaa2a08c27afaec1082f7
+size 15997553490

falcon-esm-fc-simpler-gr/logs/logs.json

@@ -0,0 +1,179 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "fc-esm-rgb_gr-finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_fp32",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 1,
+    "fwd_pred_next_n": 5,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 4,
+    "num_workers": 16,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 2e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": false,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train_gr/cache/kosmos",
+    "model_load_path": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train_gr/checkpoints/kosmos/fc-esm-rgb_gr-finetune_spatial_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_fp32/2025-08-26/23-42/epoch=0-step=20000.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "32",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_spatial_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "FCDecoder_ESM",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_spatial_injector": true,
+        "esm_use_hand_rgb": false,
+        "camera_gt_pt": 0.0,
+        "depth_gt_pt": 0.0,
+        "esm_ckpt_path": "/mnt/bn/zhengshen-lq2/falcon_pub/esm/esm.safetensors"
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "32",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 5,
+        "val_check_interval": 10000,
+        "check_val_every_n_epoch": null,
+        "max_steps": 200000,
+        "accumulate_grad_batches": 1,
+        "limit_val_batches": 100
+    },
+    "train_dataset": {
+        "type": "OpenVLADataset",
+        "data_root_dir": "/mnt/bn/robotics-data-lxh-lq/openvla/datasets/open-x-embodiment",
+        "model_name": "kosmos",
+        "image_aug": true,
+        "mode": "train",
+        "data_mix": "rt_1",
+        "window_sample": "sliding",
+        "organize_type": "interleave",
+        "shuffle_buffer_size": 51200,
+        "train": true
+    },
+    "val_dataset": {
+        "type": "OpenVLADataset",
+        "data_root_dir": "/mnt/bn/robotics-data-lxh-lq/openvla/datasets/open-x-embodiment",
+        "model_name": "kosmos",
+        "mode": "train",
+        "data_mix": "rt_1",
+        "window_sample": "sliding",
+        "organize_type": "interleave",
+        "shuffle_buffer_size": 10000,
+        "train": false
+    },
+    "norm_action": false,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/oxe_training/vlm_unfreeze_head_freeze/gr/finetune_kosmos_cont-fc-post_esm-rgb_part-ft_vlm-spatial-injector_wd-0_ws-1_act-5_gr_finetune.json",
+    "config": "configs/oxe_training/vlm_unfreeze_head_freeze/gr/finetune_kosmos_cont-fc-post_esm-rgb_part-ft_vlm-spatial-injector_wd-0_ws-1_act-5_gr_finetune.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train_gr/logs/kosmos/fc-esm-rgb_gr-finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_fp32/2025-08-28/23-08",
+    "output_dir": "/mnt/bn/zhengshen-lq2/runs/oxe_post_train_gr/checkpoints/kosmos/fc-esm-rgb_gr-finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_fp32/2025-08-28/23-08",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "23-08",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}

falcon-esm-lstm-calvin-abc/ckpts/falcon_esm_lstm_calvin_abc.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c5e9a8fbd7c956aba0cf35ed0721c8879895c3279a589045cacfab4d49c5276
+size 11476423326

falcon-esm-lstm-calvin-abc/logs/logs.json

@@ -0,0 +1,168 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "lstm-esm-rgb-ws16_calvin_finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_acc2_abc",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 16,
+    "fwd_pred_next_n": 10,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 2,
+    "num_workers": 16,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 2e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0.25,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": true,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "runs/cache/kosmos",
+    "model_load_path": "/mnt/hdfs/public/zhengshen/falcon_logs/lstm-esm-rgb-ws16_calvin_finetune_spatial_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_abc/ckpts/2025-08-15/20-18/epoch=4-step=30695.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "16",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_spatial_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "LSTMDecoder_ESM",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_spatial_injector": true,
+        "esm_use_hand_rgb": false,
+        "camera_gt_pt": 0.0,
+        "depth_gt_pt": 0.0,
+        "esm_ckpt_path": "/mnt/bn/zhengshen-lq2/falcon_pub/esm/esm.safetensors"
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "16",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 10,
+        "val_check_interval": null,
+        "check_val_every_n_epoch": 1,
+        "max_steps": -1,
+        "accumulate_grad_batches": 2
+    },
+    "train_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/training",
+        "shift_first": false,
+        "model_name": "kosmos",
+        "rgb_pad": 10,
+        "gripper_pad": 4
+    },
+    "val_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABC_D/validation",
+        "model_name": "kosmos"
+    },
+    "norm_action": true,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/calvin_finetune/vlm_unfreeze_head_freeze/lstm/finetune_kosmos_cont-lstm-post_esm-rgb_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-16_act-10_abc.json",
+    "config": "configs/calvin_finetune/vlm_unfreeze_head_freeze/lstm/finetune_kosmos_cont-lstm-post_esm-rgb_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-16_act-10_abc.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "runs/logs/kosmos/lstm-esm-rgb-ws16_calvin_finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_acc2_abc/2025-08-18/11-45",
+    "output_dir": "runs/checkpoints/kosmos/lstm-esm-rgb-ws16_calvin_finetune_vlm-spatial-injector_lr2e-5_linear-twin-v1_pre-fusion_acc2_abc/2025-08-18/11-45",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "11-45",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}

falcon-esm-lstm-calvin-abcd/ckpts/falcon_esm_lstm_calvin_abcd.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9825d7082177d8c6cff468464a45821b0515c89e9e1966e9d209ae4f0c1ccaf
+size 11476423326

falcon-esm-lstm-calvin-abcd/logs/logs.json

@@ -0,0 +1,168 @@
+{
+    "robovlm_name": "RoboKosMos",
+    "parent": null,
+    "task_name": "lstm-esm-rgb-ws16_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_acc2_abcd",
+    "model": "kosmos",
+    "model_url": "https://huggingface.co/microsoft/kosmos-2-patch14-224",
+    "seq_len": 1,
+    "image_size": 224,
+    "image_mean": [
+        0.48145466,
+        0.4578275,
+        0.40821073
+    ],
+    "image_std": [
+        0.26862954,
+        0.26130258,
+        0.27577711
+    ],
+    "window_size": 16,
+    "fwd_pred_next_n": 10,
+    "arm_gripper_loss_ratio": 0.01,
+    "cap_loss_ratio": 0.05,
+    "fwd_loss_ratio": 0,
+    "seed": 123,
+    "batch_size": 2,
+    "num_workers": 16,
+    "data_scale": 1,
+    "optimizer": "adam",
+    "learning_rate": 5e-05,
+    "min_lr_scale": 0.01,
+    "weight_decay": 0,
+    "warmup_epochs": 0.25,
+    "warmup_steps": 0,
+    "warmup_ratio": null,
+    "use_hand_rgb": true,
+    "use_time_causal_attn": false,
+    "use_mim_obs_loss": false,
+    "use_pixel_loss": true,
+    "use_obs_queries": true,
+    "use_vision_resampler": false,
+    "vision_masked_ratio": 0.9,
+    "use_tube_mask": false,
+    "cache_root": "runs/cache/kosmos",
+    "model_load_path": "/mnt/hdfs/public/zhengshen/falcon_logs/lstm-esm-rgb-ws16_calvin_finetune_spatial_injector_only_lr1e-4_zero-linear-twin-v1_pre-fusion_abcd/ckpts/2025-08-07/23-03/epoch=2-step=23745.ckpt",
+    "model_load_source": "deepspeed",
+    "resume": null,
+    "model_path": ".vlms/kosmos-2-patch14-224",
+    "model_config": ".vlms/kosmos-2-patch14-224/config.json",
+    "tcp_rel": false,
+    "use_clip_norm": false,
+    "use_state": false,
+    "train_setup": {
+        "precision": "16",
+        "predict_action": true,
+        "predict_forward": false,
+        "predict_forward_hand": false,
+        "predict_caption": false,
+        "train_vision": true,
+        "bits": -1,
+        "freeze_mm_mlp_adapter": false,
+        "freeze_backbone": false,
+        "freeze_resampler": false,
+        "tune_mm_mlp_adapter": false,
+        "mm_use_im_start_end": false,
+        "mm_use_im_patch_token": false,
+        "gradient_checkpointing": false,
+        "lora_enable": false,
+        "mm_projector_lr": 0.0001,
+        "lora_r": 64,
+        "lora_alpha": 16,
+        "lora_dropout": 0.05,
+        "lora_bias": "none",
+        "train_text_embedding": true,
+        "train_act_head": false,
+        "train_spatial_injector": true,
+        "train_decoder_layers": -1
+    },
+    "vision_resampler": {
+        "vis_dim": 1024,
+        "depth": 8,
+        "dim_head": 64,
+        "heads": 8,
+        "num_latents": 64
+    },
+    "act_encoder": null,
+    "act_head": {
+        "type": "LSTMDecoder_ESM",
+        "hidden_size": 1024,
+        "action_dim": 7,
+        "down_sample": "none",
+        "latent": 1,
+        "fwd_pred_next_n": 1,
+        "window_size": 1,
+        "action_space": "continuous",
+        "with_history": true,
+        "history_type": "post",
+        "use_spatial_injector": true,
+        "esm_use_hand_rgb": false,
+        "camera_gt_pt": 0.0,
+        "depth_gt_pt": 0.0,
+        "esm_ckpt_path": "/mnt/bn/zhengshen-lq2/falcon_pub/esm/esm.safetensors"
+    },
+    "fwd_head": null,
+    "tokenizer": {
+        "type": "AutoProcessor",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224",
+        "tokenizer_type": "kosmos",
+        "max_text_len": 256,
+        "additional_special_tokens": null
+    },
+    "vlm": {
+        "type": "AutoModelForVision2Seq",
+        "name": "kosmos",
+        "pretrained_model_name_or_path": ".vlms/kosmos-2-patch14-224"
+    },
+    "trainer": {
+        "accelerator": "gpu",
+        "strategy": "deepspeed_stage_2",
+        "precision": "16",
+        "logger": [
+            "wandb"
+        ],
+        "gradient_clip_val": 1.0,
+        "use_distributed_sampler": false,
+        "log_every_n_steps": 10,
+        "max_epochs": 10,
+        "val_check_interval": null,
+        "check_val_every_n_epoch": 1,
+        "max_steps": -1,
+        "accumulate_grad_batches": 2
+    },
+    "train_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABCD_D/training",
+        "shift_first": false,
+        "model_name": "kosmos",
+        "rgb_pad": 10,
+        "gripper_pad": 4
+    },
+    "val_dataset": {
+        "type": "DiskCalvinDataset_ESM",
+        "data_dir": "/mnt/bn/robotics/manipulation_data/calvin_data/task_ABCD_D/validation",
+        "model_name": "kosmos"
+    },
+    "norm_action": true,
+    "norm_min": -0.65,
+    "norm_max": 0.65,
+    "raw_config_path": "configs/calvin_finetune/vlm_unfreeze_head_freeze/lstm/finetune_kosmos_cont-lstm-post_esm-rgb_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-16_act-10_abcd.json",
+    "config": "configs/calvin_finetune/vlm_unfreeze_head_freeze/lstm/finetune_kosmos_cont-lstm-post_esm-rgb_part-ft_vlm_spatial-injector_wd-0_use-hand_ws-16_act-10_abcd.json",
+    "gpus": 8,
+    "num_nodes": 4,
+    "log_dir": "runs/logs/kosmos/lstm-esm-rgb-ws16_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_acc2_abcd/2025-08-11/22-59",
+    "output_dir": "runs/checkpoints/kosmos/lstm-esm-rgb-ws16_calvin_finetune_vlm-spatial-injector_lr5e-5_linear-twin-v1_pre-fusion_acc2_abcd/2025-08-11/22-59",
+    "data_dir": null,
+    "annotation_file": null,
+    "data_subfolder": null,
+    "task_num": null,
+    "exp_name": "22-59",
+    "use_multi_modal_emb": false,
+    "no_video_pretrained_model": false,
+    "finetune": false,
+    "llm": {
+        "type": null,
+        "n_embd": null,
+        "n_layer": null,
+        "n_head": null
+    }
+}