README.md

---
license: apache-2.0
---

## Requirements
bitsandbytes==0.45.3,
peft==0.14.0,
transformers==4.49.0

## Example Usage
```python
from auto_eval.success_detector import detectors

# creating the classifier instance
vlm_config={
    "model_id": os.path.expanduser(
        "~/checkpoints/auto-eval-paligemma/drawer-checkpoint-600"
    ),
    "device": "cuda:0",
    "quantize": True,
},
success_detector = detectors[FLAGS.config.success_detector_type](
    save_data=False,
    vlm_config,
)

# calling the classifier
success = success_detector(
    "is the drawer open? answer yes or no",
    obs,
    answer="yes",
)
```

For the Drawer task, query the classifier with question
> "is the drawer open? answer yes or no"

Then the `answer` field should be "yes" or "no" depending on what task you are evaluating.

For the Sink task, query the classifier with question
> "is the eggplant in the sink or in the basket? answer sink or basket or invalid"

Then the `answer` field should be either "sink" or "basket" if the task succeeded.