File size: 2,872 Bytes
4a28d4d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | # MetaX-tech
The usage of lmdeploy on a MetaX-tech device is almost the same as its usage on CUDA with PytorchEngine in lmdeploy.
Please read the original [Get Started](../get_started.md) guide before reading this tutorial.
Here is the [supported model list](../../supported_models/supported_models.md#PyTorchEngine-on-Other-Platforms).
> \[!IMPORTANT\]
> We have uploaded a docker image to aliyun.
> Please try to pull the image by following command:
>
> `docker pull crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/maca:latest`
## Offline batch inference
### LLM inference
Set `device_type="maca"` in the `PytorchEngineConfig`:
```python
from lmdeploy import pipeline
from lmdeploy import PytorchEngineConfig
pipe = pipeline("internlm/internlm2_5-7b-chat",
backend_config=PytorchEngineConfig(tp=1, device_type="maca"))
question = ["Shanghai is", "Please introduce China", "How are you?"]
response = pipe(question)
print(response)
```
### VLM inference
Set `device_type="maca"` in the `PytorchEngineConfig`:
```python
from lmdeploy import pipeline, PytorchEngineConfig
from lmdeploy.vl import load_image
pipe = pipeline('OpenGVLab/InternVL2-2B',
backend_config=PytorchEngineConfig(tp=1, device_type='maca'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
response = pipe(('describe this image', image))
print(response)
```
## Online serving
### Serve a LLM model
Add `--device maca` in the serve command.
```bash
lmdeploy serve api_server --backend pytorch --device maca internlm/internlm2_5-7b-chat
```
Run the following commands to launch docker container for lmdeploy LLM serving:
```bash
docker run -it --net=host crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/maca:latest \
bash -i -c "lmdeploy serve api_server --backend pytorch --device maca internlm/internlm2_5-7b-chat"
```
### Serve a VLM model
Add `--device maca` in the serve command
```bash
lmdeploy serve api_server --backend pytorch --device maca OpenGVLab/InternVL2-2B
```
Run the following commands to launch docker container for lmdeploy VLM serving:
```bash
docker run -it --net=host crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/maca:latest \
bash -i -c "lmdeploy serve api_server --backend pytorch --device maca OpenGVLab/InternVL2-2B"
```
## Inference with Command line Interface
Add `--device maca` in the serve command.
```bash
lmdeploy chat internlm/internlm2_5-7b-chat --backend pytorch --device maca
```
Run the following commands to launch lmdeploy chatting after starting container:
```bash
docker run -it crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/maca:latest \
bash -i -c "lmdeploy chat --backend pytorch --device maca internlm/internlm2_5-7b-chat"
```
|