File size: 1,610 Bytes
7b3209b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
language:
- zh
- en
tags:
- sweepgpm
- sweepmm
- chatglm
- multimodal
- sweeping-robot
- lora
- blip2
license: mit
---
# SweepGPM
SweepGPM is a multimodal dialogue model for sweeping robots in home scenarios, fine-tuned from [VisualGLM-6B](https://github.com/THUDM/VisualGLM-6B). The language model is based on [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B) (6.2B parameters, frozen), and the image encoder uses [CLIP ViT-L/14](https://github.com/openai/CLIP) (frozen). The Q-Former, fully connected projection layer, and LoRA adapters (rank=4, last 2 layers only) are trained to adapt the model to the domain knowledge of sweeping robots.
## Performance
| Downstream Task | Metric | SweepGPM |
|----------------|--------|----------|
| Room Type Classification | Mean Accuracy | **84.3%** |
| Obstacle Detection | mAP@0.5 | **86.1%** |
| Lost Item Search | Mean Recall | **80.2%** |
## Usage
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("bazaar-research/sweepgpm", trust_remote_code=True)
model = AutoModel.from_pretrained("bazaar-research/sweepgpm", trust_remote_code=True).half().cuda()
image_path = "your_image.jpg"
response, history = model.chat(tokenizer, image_path, "Give the room type in the image.", history=[])
print(response)
response, history = model.chat(tokenizer, image_path, "Provide fine-grained bounding boxes for all objects in the image.", history=history)
print(response)
```
## Dependencies
```bash
pip install SwissArmyTransformer>=0.3.6 torch>=2.0.1 torchvision transformers>=4.31.0 cpm_kernels peft>=0.4.0
```
|