File size: 1,034 Bytes
7187e60 428485d 7187e60 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
license: apache-2.0
---
## Requirements
bitsandbytes==0.45.3,
peft==0.14.0,
transformers==4.49.0
## Example Usage
```python
from auto_eval.success_detector import detectors
# creating the classifier instance
vlm_config={
"model_id": os.path.expanduser(
"~/checkpoints/auto-eval-paligemma/drawer-checkpoint-600"
),
"device": "cuda:0",
"quantize": True,
},
success_detector = detectors[FLAGS.config.success_detector_type](
save_data=False,
vlm_config,
)
# calling the classifier
success = success_detector(
"is the drawer open? answer yes or no",
obs,
answer="yes",
)
```
For the Drawer task, query the classifier with question
> "is the drawer open? answer yes or no"
Then the `answer` field should be "yes" or "no" depending on what task you are evaluating.
For the Sink task, query the classifier with question
> "is the eggplant in the sink or in the basket? answer sink or basket or invalid"
Then the `answer` field should be either "sink" or "basket" if the task succeeded. |