AlexWortega commited on
Commit
6d2b16a
·
verified ·
1 Parent(s): 6fc7c1c

Upload vllm_plugin/README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. vllm_plugin/README.md +43 -0
vllm_plugin/README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # vLLM Plugin for Borealis
2
+
3
+ vLLM plugin to enable inference with Borealis Audio-Language Model.
4
+
5
+ ## Installation
6
+
7
+ ```bash
8
+ pip install -e .
9
+ ```
10
+
11
+ ## Usage
12
+
13
+ After installation, the Borealis model will be automatically registered with vLLM.
14
+
15
+ ```python
16
+ from vllm import LLM, SamplingParams
17
+
18
+ # Load model
19
+ llm = LLM(
20
+ model="Vikhrmodels/Borealis-5b-it",
21
+ trust_remote_code=True,
22
+ )
23
+
24
+ # Inference with audio
25
+ # Note: Audio preprocessing should be done separately
26
+ sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
27
+ outputs = llm.generate(prompts, sampling_params)
28
+ ```
29
+
30
+ ## Architecture
31
+
32
+ Borealis combines:
33
+ - **Whisper Large V3** encoder for audio processing
34
+ - **Qwen3-4B** LLM for text generation
35
+ - **Adapter** to project audio embeddings to LLM space
36
+
37
+ ## Model
38
+
39
+ - HuggingFace: [Vikhrmodels/Borealis-5b-it](https://huggingface.co/Vikhrmodels/Borealis-5b-it)
40
+
41
+ ## Note
42
+
43
+ This plugin is experimental. For production use, consider using the standard transformers implementation.