---
language:
- zh
- en
tags:
- sweepgpm
- sweepmm
- chatglm
- multimodal
- sweeping-robot
- lora
- blip2
license: mit
---
# SweepGPM

SweepGPM is a multimodal dialogue model for sweeping robots in home scenarios, fine-tuned from [VisualGLM-6B](https://github.com/THUDM/VisualGLM-6B). The language model is based on [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B) (6.2B parameters, frozen), and the image encoder uses [CLIP ViT-L/14](https://github.com/openai/CLIP) (frozen). The Q-Former, fully connected projection layer, and LoRA adapters (rank=4, last 2 layers only) are trained to adapt the model to the domain knowledge of sweeping robots.


## Performance

| Downstream Task | Metric | SweepGPM |
|----------------|--------|----------|
| Room Type Classification | Mean Accuracy | **84.3%** |
| Obstacle Detection | mAP@0.5 | **86.1%** |
| Lost Item Search | Mean Recall | **80.2%** |

## Usage

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bazaar-research/sweepgpm", trust_remote_code=True)
model = AutoModel.from_pretrained("bazaar-research/sweepgpm", trust_remote_code=True).half().cuda()

image_path = "your_image.jpg"
response, history = model.chat(tokenizer, image_path, "Give the room type in the image.", history=[])
print(response)

response, history = model.chat(tokenizer, image_path, "Provide fine-grained bounding boxes for all objects in the image.", history=history)
print(response)
```

## Dependencies

```bash
pip install SwissArmyTransformer>=0.3.6 torch>=2.0.1 torchvision transformers>=4.31.0 cpm_kernels peft>=0.4.0
```