Create README.md
Browse files# L-CAMP: Language-Conditioned Axis and Motion Prediction for Articulated Object Manipulation
[Ikechukwu Daniel Adebi](https://danieladebi.github.io), [Mitchell Pryor](https://www.me.utexas.edu/people/faculty-directory/pryor)
## Setup
1. Create the conda/micromamba environment from `environment_lcamp.yml`:
```bash
micromamba create -f environment_lcamp.yml # or: conda env create -f environment_lcamp.yml
micromamba activate lcamp
```
2. Build the LCAMP dataset (only needed once per dataset version), see
`process_lcamp_dataset.py` / `process_lcamp_dataset.slurm`. This produces a
dataset directory containing `train_dataset.json`, `test_dataset.json`,
and `intrinsics.npy`, referenced below as `--data_root`/`--data_dir`.
All commands below (`train.py`, `eval.py`) assume the `lcamp` environment is
active.
### Download Model
A pretrained checkpoint is available on HuggingFace:
[danieladebi/lcamp-model](https://huggingface.co/danieladebi/lcamp-model).
Download it into `checkpoints/` to skip training and go straight to
evaluation.
## Training
Training is done with `train.py` via `torchrun` (DDP). The dataset must already
be built (see `process_lcamp_dataset.py` / `process_lcamp_dataset.slurm`) as a
directory containing `train_dataset.json`, `test_dataset.json`, and
`intrinsics.npy`.
Single-GPU / local run:
```bash
torchrun --nproc_per_node=1 train.py \
--save_path checkpoints/my_lcamp_model \
--data_root ./lcamp_dataset_limited_dataset \
--model_type resnet \
--epochs 50 \
--batch_size 64 \
--lr 1e-3 \
--min_lr 1e-5 \
--use_camera_frame \
--use_depth \
--use_text_instructions \
--emphasize_part_mask \
--lambda_axis 8 \
--lambda_anchor 2 \
--lambda_joint 1
```
Multi-GPU (cluster) example:
```bash
torchrun --nproc_per_node=8 train.py \
--save_path "$SCRATCH/lcamp_model_RGB_log_norm_DEPTH_NO_AXIS_SYMMETRY" \
--data_root "$SCRATCH/lcamp_dataset_off_target_depth_LANGUAGE_EXPLICIT_OBJ_CMDS" \
--model_type resnet \
--epochs 50 \
--warmup_epochs 5 \
--batch_size 256 \
--lr 1e-3 \
--min_lr 1e-5 \
--use_camera_frame \
--use_depth \
--use_text_instructions \
--emphasize_part_mask \
--randomize_background \
--lambda_axis 8 \
--lambda_anchor 4 \
--lambda_joint 1 \
--lambda_plucker 0 \
--log_dir "$SCRATCH/train_logs" \
--cluster_loc "stampede3"
```
See `train.slurm` / `train_h100.slurm` for full Slurm submission scripts (note:
those scripts currently hold older `--lr`/`--lambda_anchor`/`--lambda_joint`
values, update them if you want the Slurm jobs to match the above).
Notable flags (`train.py --help` for the complete list):
- `--model_type`: `resnet` (default LCAMP backbone) or `dino`.
- `--use_text_instructions`: enables the language-conditioned model
(`LCAMPModelResnetLanguage`); required if you plan to run the `lcamp`
backend with instructions in `lcamp_service_node.py`.
- `--use_depth`: adds depth as an input channel.
- `--bbox_loc`: predict axis location as `(u,v,z)` decoded via the part
bounding box instead of raw XYZ, must match `--bbox_loc` in `eval.py` and
`use_bbox_loc` in `lcamp_service_node.py`/`lcamp.launch.py` for the same
checkpoint.
- `--resume` / `--resume_from <path>`: resume from `checkpoint_best.pt` in
`--save_path`, or from an explicit checkpoint path.
- `--save_freq`, `--eval_freq`: checkpoint/eval cadence in epochs.
Checkpoints are written to `--save_path` as `checkpoint_best.pt`,
`checkpoint_last.pt`, and periodic `checkpoint_{epoch}.pt` files.
## Evaluation
Evaluation is done with `eval.py` (single process, no DDP):
```bash
python eval.py \
--model_path checkpoints/my_lcamp_model/checkpoint_best.pt \
--data_dir ./lcamp_dataset_limited_dataset \
--split test \
--model_type resnet \
--use_camera_frame \
--use_depth \
--use_text_instructions \
--visualize
```
This must use the **same architecture flags used at training time**
(`--model_type`, `--use_depth`, `--use_text_instructions`,
`--exclude_object_mask`, `--emphasize_part_mask`/`--emphasize_object_mask`,
`--bbox_loc`, etc.), mismatches will load state dict weights incorrectly or
error, and `--use_camera_frame`/`--cluster_loc` must match how the dataset
was built.
Notable flags:
- `--split`: `train` or `test`.
- `--visualize`: saves per-sample prediction images.
- `--random_preds`: baseline using random axis/location predictions.
- `--free_motion_query_prob`: fraction of eval samples using a free-motion
(zero-vector) query instead of the GT axis, to measure reliance on visual
cues vs. the GT axis itself.
Results (per-object/per-category success rates, axis/location error, joint
type accuracy) are written to `results/<model_name>/`, where `<model_name>`
is derived from `--model_path`.
Once you have a checkpoint you're happy with, drop it into `checkpoints/`
and point the `model_name` launch argument in `launch/lcamp.launch.py` at it
(see the launch file for details on `use_bbox_loc` and other runtime flags).
## Deploying on a Robot
Running predictions on a real Boston Dynamics Spot (via the ROS2 service node
in `scripts/lcamp_service_node.py`) is outside the scope of this README, see
[ros2_setup/ROS2_SETUP.md](ros2_setup/ROS2_SETUP.md) for that setup.