Update README.md
Browse files
README.md
CHANGED
|
@@ -6,6 +6,14 @@ tags:
|
|
| 6 |
language:
|
| 7 |
- en
|
| 8 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
# L-CAMP: Language-Conditioned Axis and Motion Prediction for Articulated Object Manipulation
|
| 11 |
|
|
@@ -13,18 +21,17 @@ language:
|
|
| 13 |
|
| 14 |
The University of Texas at Austin
|
| 15 |
|
| 16 |
-
|
|
|
|
|
|
|
| 17 |
|
| 18 |
-
|
| 19 |
-
[danieladebi/lcamp-model](https://huggingface.co/danieladebi/lcamp-model).
|
| 20 |
-
Download it into `checkpoints/` to skip training and go straight to
|
| 21 |
-
evaluation.
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
## Setup
|
| 26 |
|
| 27 |
-
1.
|
| 28 |
```bash
|
| 29 |
micromamba create -f environment_lcamp.yml # or: conda env create -f environment_lcamp.yml
|
| 30 |
micromamba activate lcamp
|
|
@@ -37,13 +44,9 @@ You can access our model by downloading the `lcamp_model.pt` checkpoint from the
|
|
| 37 |
All commands below (`train.py`, `eval.py`) assume the `lcamp` environment is
|
| 38 |
active.
|
| 39 |
|
| 40 |
-
|
| 41 |
## Training
|
| 42 |
|
| 43 |
-
|
| 44 |
-
be built (see `process_lcamp_dataset.py` / `process_lcamp_dataset.slurm`) as a
|
| 45 |
-
directory containing `train_dataset.json`, `test_dataset.json`, and
|
| 46 |
-
`intrinsics.npy`.
|
| 47 |
|
| 48 |
Single-GPU / local run:
|
| 49 |
```bash
|
|
@@ -87,9 +90,7 @@ torchrun --nproc_per_node=8 train.py \
|
|
| 87 |
--log_dir "$SCRATCH/train_logs" \
|
| 88 |
--cluster_loc "stampede3"
|
| 89 |
```
|
| 90 |
-
See `train.slurm` / `train_h100.slurm` for full Slurm submission scripts (note:
|
| 91 |
-
those scripts currently hold older `--lr`/`--lambda_anchor`/`--lambda_joint`
|
| 92 |
-
values, update them if you want the Slurm jobs to match the above).
|
| 93 |
|
| 94 |
Notable flags (`train.py --help` for the complete list):
|
| 95 |
- `--model_type`: `resnet` (default LCAMP backbone) or `dino`.
|
|
@@ -110,10 +111,10 @@ Checkpoints are written to `--save_path` as `checkpoint_best.pt`,
|
|
| 110 |
|
| 111 |
## Evaluation
|
| 112 |
|
| 113 |
-
Evaluation is done with `eval.py` (single process, no DDP):
|
| 114 |
```bash
|
| 115 |
python eval.py \
|
| 116 |
-
--model_path checkpoints/
|
| 117 |
--data_dir ./lcamp_dataset_limited_dataset \
|
| 118 |
--split test \
|
| 119 |
--model_type resnet \
|
|
@@ -142,12 +143,13 @@ Results (per-object/per-category success rates, axis/location error, joint
|
|
| 142 |
type accuracy) are written to `results/<model_name>/`, where `<model_name>`
|
| 143 |
is derived from `--model_path`.
|
| 144 |
|
| 145 |
-
Once you
|
| 146 |
-
|
| 147 |
-
|
| 148 |
|
| 149 |
## Deploying on a Robot
|
| 150 |
|
| 151 |
Running predictions on a real Boston Dynamics Spot (via the ROS2 service node
|
| 152 |
-
in `scripts/lcamp_service_node.py`) is outside the scope of this
|
| 153 |
-
[ros2_setup/ROS2_SETUP.md](ros2_setup/ROS2_SETUP.md) for that setup.
|
|
|
|
|
|
| 6 |
language:
|
| 7 |
- en
|
| 8 |
---
|
| 9 |
+
---
|
| 10 |
+
tags:
|
| 11 |
+
- computer-vision
|
| 12 |
+
- robotics
|
| 13 |
+
- articulated-objects
|
| 14 |
+
language:
|
| 15 |
+
- en
|
| 16 |
+
---
|
| 17 |
|
| 18 |
# L-CAMP: Language-Conditioned Axis and Motion Prediction for Articulated Object Manipulation
|
| 19 |
|
|
|
|
| 21 |
|
| 22 |
The University of Texas at Austin
|
| 23 |
|
| 24 |
+
Code: [UTNuclearRobotics/l_camp](https://github.com/UTNuclearRobotics/l_camp)
|
| 25 |
+
|
| 26 |
+
## About this checkpoint
|
| 27 |
|
| 28 |
+
This repository hosts a pretrained L-CAMP checkpoint, `lcamp_model.pt`, available from the Files tab above. It predicts the screw axis and motion of an articulated object part from an RGB(-D) image conditioned on a language instruction.
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
To use it, download `lcamp_model.pt` from the Files tab and place it in a `checkpoints/` directory inside a clone of the [GitHub repo](https://github.com/UTNuclearRobotics/l_camp), then follow the setup and evaluation steps below.
|
| 31 |
|
| 32 |
## Setup
|
| 33 |
|
| 34 |
+
1. Clone the [GitHub repo](https://github.com/UTNuclearRobotics/l_camp) and create the conda/micromamba environment from `environment_lcamp.yml`:
|
| 35 |
```bash
|
| 36 |
micromamba create -f environment_lcamp.yml # or: conda env create -f environment_lcamp.yml
|
| 37 |
micromamba activate lcamp
|
|
|
|
| 44 |
All commands below (`train.py`, `eval.py`) assume the `lcamp` environment is
|
| 45 |
active.
|
| 46 |
|
|
|
|
| 47 |
## Training
|
| 48 |
|
| 49 |
+
If you'd rather train your own checkpoint instead of using this one, training is done with `train.py` via `torchrun` (DDP). The dataset must already be built (see `process_lcamp_dataset.py` / `process_lcamp_dataset.slurm`) as a directory containing `train_dataset.json`, `test_dataset.json`, and `intrinsics.npy`.
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
Single-GPU / local run:
|
| 52 |
```bash
|
|
|
|
| 90 |
--log_dir "$SCRATCH/train_logs" \
|
| 91 |
--cluster_loc "stampede3"
|
| 92 |
```
|
| 93 |
+
See `train.slurm` / `train_h100.slurm` in the GitHub repo for full Slurm submission scripts (note: those scripts currently hold older `--lr`/`--lambda_anchor`/`--lambda_joint` values, update them if you want the Slurm jobs to match the above).
|
|
|
|
|
|
|
| 94 |
|
| 95 |
Notable flags (`train.py --help` for the complete list):
|
| 96 |
- `--model_type`: `resnet` (default LCAMP backbone) or `dino`.
|
|
|
|
| 111 |
|
| 112 |
## Evaluation
|
| 113 |
|
| 114 |
+
Evaluation is done with `eval.py` (single process, no DDP). To evaluate this checkpoint:
|
| 115 |
```bash
|
| 116 |
python eval.py \
|
| 117 |
+
--model_path checkpoints/lcamp_model.pt \
|
| 118 |
--data_dir ./lcamp_dataset_limited_dataset \
|
| 119 |
--split test \
|
| 120 |
--model_type resnet \
|
|
|
|
| 143 |
type accuracy) are written to `results/<model_name>/`, where `<model_name>`
|
| 144 |
is derived from `--model_path`.
|
| 145 |
|
| 146 |
+
Once you're happy with a checkpoint (this one or your own), point the
|
| 147 |
+
`model_name` launch argument in `launch/lcamp.launch.py` at it (see the
|
| 148 |
+
launch file for details on `use_bbox_loc` and other runtime flags).
|
| 149 |
|
| 150 |
## Deploying on a Robot
|
| 151 |
|
| 152 |
Running predictions on a real Boston Dynamics Spot (via the ROS2 service node
|
| 153 |
+
in `scripts/lcamp_service_node.py`) is outside the scope of this model card,
|
| 154 |
+
see [ros2_setup/ROS2_SETUP.md](https://github.com/UTNuclearRobotics/l_camp/blob/main/ros2_setup/ROS2_SETUP.md) in the GitHub repo for that setup.
|
| 155 |
+
|