Add model card

README.md

@@ -0,0 +1,105 @@
+---
+license: apache-2.0
+---
+
+# N0-TWAM task-specific checkpoints (one checkpoint per task)
+
+Twenty task-specific post-trained N0-TWAM checkpoints: the 8 UniVTAC single-arm tasks and
+the 12 NeoSim tasks (4 single-arm + 8 dual-arm). Each checkpoint was post-trained on a single
+task. 
+
+The action space of each checkpoint, **absEE** (absolute end-effector) or **delta EE**
+(horizon delta), is listed per task below.
+
+## UniVTAC 8
+
+| Task (NeoSim env) | Arms | Action space | `TWAM_SERVE_TASK` | Prompt (send verbatim) |
+|---|---|---|---|---|
+| `insert_hole` | single | delta EE | `univtac_insert_hole_rot6d_current` | Insert the peg into the hole |
+| `insert_tube` | single | delta EE | `univtac_insert_tube_rot6d_current` | Insert the tube into the slot |
+| `grasp_classify` | single | delta EE | `univtac_grasp_classify_hdf5_current` | Grasp classify |
+| `pull_out_key` | single | delta EE | `univtac_pull_out_key_rot6d_current` | Extract a key from a lock |
+| `insert_HDMI` | single | delta EE | `univtac_insert_HDMI_rot6d_current` | Insert the HDMI connector into the slot |
+| `put_bottle_in_shelf` | single | delta EE | `univtac_put_bottle_in_shelf_rot6d_current` | Place a bottle onto a shelf |
+| `lift_can` | single | absEE | `univtac_lift_can_rot6d_current` | Lift the can |
+| `lift_bottle` | single | absEE | `univtac_lift_bottle_rot6d_current` | Lift the bottle |
+
+## NeoSim 12
+
+| Task (NeoSim env) | Arms | Action space | `TWAM_SERVE_TASK` | Prompt (send verbatim) |
+|---|---|---|---|---|
+| `grasp_chip` | single | absEE | `univtac_grasp_chip_hdf5_current` | Grasp chip |
+| `insert_USB` | single | delta EE | `univtac_insert_USB_hdf5_current` | Insert USB |
+| `phone_socket_replug` | single | absEE | `univtac_phone_socket_replug_hdf5_current` | Pull a phone connector out of its socket and plug it back in |
+| `pour_ball` | single | delta EE | `univtac_pour_ball_hdf5_current` | Grasp a cup and pour the balls inside it out onto a plate |
+| `dual_bowl_place_stack` | dual | delta EE | `univtac_dual_bowl_stack_hdf5_current` | Use dual arms to stack a bowl |
+| `dual_plate_place_stack` | dual | delta EE | `univtac_dual_plate_stack_hdf5_current` | Use dual arms to stack a plate |
+| `dual_bowl_unstack` | dual | delta EE | `univtac_dual_bowl_unstack_hdf5_current` | Use dual arms to unstack a bowl |
+| `dual_screw_sleeve` | dual | delta EE | `univtac_dual_screw_sleeve_hdf5_current` | Use dual arms to screw the sleeve |
+| `dual_cup_unstack` | dual | delta EE | `univtac_dual_cup_unstack_hdf5_current` | Use dual arms to unstack the cups |
+| `dual_cup_handover_place` | dual | delta EE | `univtac_dual_cup_handover_hdf5_current` | Use dual arms to hand over a cup |
+| `dual_cup_place_stack` | dual | delta EE | `univtac_dual_cup_stack_hdf5_current` | Use dual arms to stack a cup |
+| `dual_gear_holder` | dual | delta EE | `univtac_dual_gear_holder_hdf5_current` | Use dual arms to place the gear in the holder |
+
+## Serving and evaluation
+
+Everything needed besides this repo is public: the code
+(<https://github.com/neoteai/N0-TWAM>), the base model components
+(`NeoteAI/n0-twam-base`: `vae/`, `text_encoder/`, `tokenizer/`) and the simulator with its
+evaluation clients (<https://github.com/neoteai/NeoSim>, Isaac Sim 4.5.0 / Isaac Lab 2.1.1).
+
+**1. Build a serve bundle** for one task (N0-TWAM repo):
+
+```bash
+python script/make_serve_bundle.py \
+    --checkpoint /path/to/this-repo/checkpoints/univtac8/lift_can \
+    --base /path/to/n0-twam-base --bundle /path/to/bundles/lift_can
+```
+
+The script notes that there is no `train_meta.json`; that is expected for these checkpoints.
+
+**2. Set the inference settings** in
+`n0_twam/configs/twam_posttrain_server_cfg.py`:
+
+```python
+s.num_inference_steps = 15
+s.action_num_inference_steps = 10
+```
+
+**3. Start one server per task.** `TWAM_SERVE_TASK` and `TWAM_SERVE_ACTION_MODE`
+(`absee` | `delta`) come from the tables above:
+
+```bash
+TWAM_SERVE_POOL=/path/to/this-repo/pool \
+TWAM_SERVE_TASK=univtac_lift_can_rot6d_current \
+TWAM_SERVE_ACTION_MODE=absee \
+TWAM_SERVE_BUNDLE=/path/to/bundles/lift_can TWAM_SERVE_OUT=/path/to/serve-output \
+  python -m n0_twam.n0_twam_server --config-name multitask_server --port 29601
+```
+
+The config wires that task's own normalization stats, camera/tactile keys and action
+channels. Actions must be de-normalized with the task's own stats and in the task's own
+action space; mixing them up rescales actions by orders of magnitude.
+
+**4. Run the client** (NeoSim repo). The `demo` task config sends the marker-less `rgb`
+tactile image these checkpoints were trained on; the prompt is passed verbatim:
+
+```bash
+python eval/eval_twam_ee_cl.py lift_can demo --server_host <server-ip> --server_port 29601 \
+    --prompt "Lift the can"
+```
+
+Dual-arm tasks use `eval/eval_twam_ee_dual_cl.py` with the same arguments. Keep the client
+defaults (`UNIVTAC_TICKS_PER_SLOT=2`, all tactile keyframes, per-task step limits).
+
+## Layout
+
+```
+checkpoints/<suite>/<task>/transformer/   config.json + diffusion_pytorch_model.safetensors
+pool/                                     serve-time task pool for the `multitask_server` config
+  norm_stat_per_robot.json                  delta-EE stats, one entry per task (16)
+  norm_stat_absee_per_robot.json            absEE stats, one entry per task (4)
+  train/<TWAM_SERVE_TASK>/meta/             info.json + tasks.jsonl (minimal stubs, see Notes)
+norm/<task>/                              the same stats as one file per task
+```
+