README.md

metadata

license: apache-2.0

Requirements

bitsandbytes==0.45.3, peft==0.14.0, transformers==4.49.0

Example Usage

from auto_eval.success_detector import detectors

# creating the classifier instance
vlm_config={
    "model_id": os.path.expanduser(
        "~/checkpoints/auto-eval-paligemma/drawer-checkpoint-600"
    ),
    "device": "cuda:0",
    "quantize": True,
},
success_detector = detectors[FLAGS.config.success_detector_type](
    save_data=False,
    vlm_config,
)

# calling the classifier
success = success_detector(
    "is the drawer open? answer yes or no",
    obs,
    answer="yes",
)

For the Drawer task, query the classifier with question

"is the drawer open? answer yes or no"

Then the answer field should be "yes" or "no" depending on what task you are evaluating.

For the Sink task, query the classifier with question

"is the eggplant in the sink or in the basket? answer sink or basket or invalid"

Then the answer field should be either "sink" or "basket" if the task succeeded.