File size: 1,970 Bytes
c33a7ce |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
## Whisper
Whisper models are useful for converting audio files to text. The sample application [audio-to-text](../audio-to-text/README.md)
describes how to run an inference application. This document describes how to build a service for a Whisper model.
### Build model service
To build a Whisper model service container image from this directory,
```bash
podman build -t whisper:image .
```
or
```bash
make -f Makefile build
```
### Download Whisper model
You can to download the model from HuggingFace. There are various Whisper models available which vary in size and can be found
[here](https://huggingface.co/ggerganov/whisper.cpp). We will be using the `small` model which is about 466 MB.
- **small**
- Download URL: [https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin](https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.bin)
- **base.en**
- Download URL: [https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin](https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin)
```bash
cd ../../models
curl -sLO <Download URL>
cd ../model_servers/whispercpp
```
### Deploy Model Service
Deploy the LLM and volume mount the model of choice.
Here, we are mounting the `ggml-small.bin` model as downloaded from above.
```bash
# Note: the :Z may need to be omitted from the model volume mount if not running on Linux
podman run --rm -it \
-p 8001:8001 \
-v /local/path/to/locallm/models/ggml-small.bin:/models/ggml-small.bin:Z,ro \
-e HOST=0.0.0.0 \
-e MODEL_PATH=/models/ggml-small.bin \
-e PORT=8001 \
whisper:image
```
or using the make command:
`make -f Makefile run`
By default, a sample `jfk.wav` file is included in the whisper image. This can be used to test with.
The environment variable `AUDIO_FILE`, can be passed with your own audio file to override the default `/app/jfk.wav` file within the whisper image.
|