Instructions to use NbAiLab/borealis2-26b-a4b-preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NbAiLab/borealis2-26b-a4b-preview with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="NbAiLab/borealis2-26b-a4b-preview")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("NbAiLab/borealis2-26b-a4b-preview")
model = AutoModelForMultimodalLM.from_pretrained("NbAiLab/borealis2-26b-a4b-preview", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NbAiLab/borealis2-26b-a4b-preview with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NbAiLab/borealis2-26b-a4b-preview"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NbAiLab/borealis2-26b-a4b-preview",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/NbAiLab/borealis2-26b-a4b-preview

SGLang

How to use NbAiLab/borealis2-26b-a4b-preview with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NbAiLab/borealis2-26b-a4b-preview" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NbAiLab/borealis2-26b-a4b-preview",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NbAiLab/borealis2-26b-a4b-preview" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NbAiLab/borealis2-26b-a4b-preview",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use NbAiLab/borealis2-26b-a4b-preview with Docker Model Runner:
```
docker model run hf.co/NbAiLab/borealis2-26b-a4b-preview
```

versae commited on 5 days ago

Commit

678c233

verified ·

1 Parent(s): 1ffd490

Update README.md

Browse files

Files changed (1) hide show

README.md +15 -0

README.md CHANGED Viewed

@@ -120,6 +120,21 @@ This is a preview experiment and has not been completely safety-aligned. The mod
 - **NbAiLab/borealis2-26b-a4b-preview** (safetensors).
 ## Acknowledgements
 Thanks to the Gemma team at Google and to everyone contributing feedback on this preview.

 - **NbAiLab/borealis2-26b-a4b-preview** (safetensors).
+## License
+The license of this model is an adaptation of the Apache 2.0 license with additional use-based restrictions. In particular, users of the model are required to refrain from intentionally using the model to recreate data the model has been trained on. The license also requires users not to use the model or its output to provide end-user services whose primary purpose is to give access to licensed press publications in the training data.
+For more information, see the [LICENSE](./LICENSE) and the [License FAQ](./LICENSE_FAQ.pdf).
 ## Acknowledgements
 Thanks to the Gemma team at Google and to everyone contributing feedback on this preview.
+## Disclaimer
+These models may have bias and/or any other undesirable distortions. When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence. In no event shall the owner of the models (The National Library of Norway) be liable for any results arising from the use made by third parties of these models.
+## Contact
+For feedback, technical concerns, or collaboration inquiries, please contact <a rel="noopener nofollow" href="mailto:ailab@nb.no">ailab@nb.no</a>.