Update README.md

README.md

@@ -7,4 +7,147 @@ language:
 - en
 ---
 
-Access our model by downloading the `lcamp_model.pt` checkpoint from the Files tab
+# L-CAMP: Language-Conditioned Axis and Motion Prediction for Articulated Object Manipulation
+
+[Ikechukwu Daniel Adebi](https://danieladebi.github.io), [Peter Stone](https://www.cs.utexas.edu/~pstone/), [Mitchell Pryor](https://www.me.utexas.edu/people/faculty-directory/pryor)
+
+The University of Texas at Austin
+
+## Download Model
+
+A pretrained checkpoint is available on HuggingFace:
+[danieladebi/lcamp-model](https://huggingface.co/danieladebi/lcamp-model).
+Download it into `checkpoints/` to skip training and go straight to
+evaluation.
+
+You can access our model by downloading the `lcamp_model.pt` checkpoint from the Files tab
+
+## Setup
+
+1. Create the conda/micromamba environment from `environment_lcamp.yml`:
+   ```bash
+   micromamba create -f environment_lcamp.yml   # or: conda env create -f environment_lcamp.yml
+   micromamba activate lcamp
+   ```
+2. Build the LCAMP dataset (only needed once per dataset version), see
+   `process_lcamp_dataset.py` / `process_lcamp_dataset.slurm`. This produces a
+   dataset directory containing `train_dataset.json`, `test_dataset.json`,
+   and `intrinsics.npy`, referenced below as `--data_root`/`--data_dir`.
+
+All commands below (`train.py`, `eval.py`) assume the `lcamp` environment is
+active.
+
+
+## Training
+
+Training is done with `train.py` via `torchrun` (DDP). The dataset must already
+be built (see `process_lcamp_dataset.py` / `process_lcamp_dataset.slurm`) as a
+directory containing `train_dataset.json`, `test_dataset.json`, and
+`intrinsics.npy`.
+
+Single-GPU / local run:
+```bash
+torchrun --nproc_per_node=1 train.py \
+    --save_path checkpoints/my_lcamp_model \
+    --data_root ./lcamp_dataset_limited_dataset \
+    --model_type resnet \
+    --epochs 50 \
+    --batch_size 64 \
+    --lr 1e-3 \
+    --min_lr 1e-5 \
+    --use_camera_frame \
+    --use_depth \
+    --use_text_instructions \
+    --emphasize_part_mask \
+    --lambda_axis 8 \
+    --lambda_anchor 2 \
+    --lambda_joint 1
+```
+
+Multi-GPU (cluster) example:
+```bash
+torchrun --nproc_per_node=8 train.py \
+    --save_path "$SCRATCH/lcamp_model_RGB_log_norm_DEPTH_NO_AXIS_SYMMETRY" \
+    --data_root "$SCRATCH/lcamp_dataset_off_target_depth_LANGUAGE_EXPLICIT_OBJ_CMDS" \
+    --model_type resnet \
+    --epochs 50 \
+    --warmup_epochs 5 \
+    --batch_size 256 \
+    --lr 1e-3 \
+    --min_lr 1e-5 \
+    --use_camera_frame \
+    --use_depth \
+    --use_text_instructions \
+    --emphasize_part_mask \
+    --randomize_background \
+    --lambda_axis 8 \
+    --lambda_anchor 4 \
+    --lambda_joint 1 \
+    --lambda_plucker 0 \
+    --log_dir "$SCRATCH/train_logs" \
+    --cluster_loc "stampede3"
+```
+See `train.slurm` / `train_h100.slurm` for full Slurm submission scripts (note:
+those scripts currently hold older `--lr`/`--lambda_anchor`/`--lambda_joint`
+values, update them if you want the Slurm jobs to match the above).
+
+Notable flags (`train.py --help` for the complete list):
+- `--model_type`: `resnet` (default LCAMP backbone) or `dino`.
+- `--use_text_instructions`: enables the language-conditioned model
+  (`LCAMPModelResnetLanguage`); required if you plan to run the `lcamp`
+  backend with instructions in `lcamp_service_node.py`.
+- `--use_depth`: adds depth as an input channel.
+- `--bbox_loc`: predict axis location as `(u,v,z)` decoded via the part
+  bounding box instead of raw XYZ, must match `--bbox_loc` in `eval.py` and
+  `use_bbox_loc` in `lcamp_service_node.py`/`lcamp.launch.py` for the same
+  checkpoint.
+- `--resume` / `--resume_from <path>`: resume from `checkpoint_best.pt` in
+  `--save_path`, or from an explicit checkpoint path.
+- `--save_freq`, `--eval_freq`: checkpoint/eval cadence in epochs.
+
+Checkpoints are written to `--save_path` as `checkpoint_best.pt`,
+`checkpoint_last.pt`, and periodic `checkpoint_{epoch}.pt` files.
+
+## Evaluation
+
+Evaluation is done with `eval.py` (single process, no DDP):
+```bash
+python eval.py \
+    --model_path checkpoints/my_lcamp_model/checkpoint_best.pt \
+    --data_dir ./lcamp_dataset_limited_dataset \
+    --split test \
+    --model_type resnet \
+    --use_camera_frame \
+    --use_depth \
+    --use_text_instructions \
+    --visualize
+```
+
+This must use the **same architecture flags used at training time**
+(`--model_type`, `--use_depth`, `--use_text_instructions`,
+`--exclude_object_mask`, `--emphasize_part_mask`/`--emphasize_object_mask`,
+`--bbox_loc`, etc.), mismatches will load state dict weights incorrectly or
+error, and `--use_camera_frame`/`--cluster_loc` must match how the dataset
+was built.
+
+Notable flags:
+- `--split`: `train` or `test`.
+- `--visualize`: saves per-sample prediction images.
+- `--random_preds`: baseline using random axis/location predictions.
+- `--free_motion_query_prob`: fraction of eval samples using a free-motion
+  (zero-vector) query instead of the GT axis, to measure reliance on visual
+  cues vs. the GT axis itself.
+
+Results (per-object/per-category success rates, axis/location error, joint
+type accuracy) are written to `results/<model_name>/`, where `<model_name>`
+is derived from `--model_path`.
+
+Once you have a checkpoint you're happy with, drop it into `checkpoints/`
+and point the `model_name` launch argument in `launch/lcamp.launch.py` at it
+(see the launch file for details on `use_bbox_loc` and other runtime flags).
+
+## Deploying on a Robot
+
+Running predictions on a real Boston Dynamics Spot (via the ROS2 service node
+in `scripts/lcamp_service_node.py`) is outside the scope of this README, see
+[ros2_setup/ROS2_SETUP.md](ros2_setup/ROS2_SETUP.md) for that setup.