Instructions to use NousResearch/Obsidian-3B-V0.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use NousResearch/Obsidian-3B-V0.5 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="NousResearch/Obsidian-3B-V0.5")# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("NousResearch/Obsidian-3B-V0.5", dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use NousResearch/Obsidian-3B-V0.5 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "NousResearch/Obsidian-3B-V0.5" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "NousResearch/Obsidian-3B-V0.5", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/NousResearch/Obsidian-3B-V0.5
- SGLang
How to use NousResearch/Obsidian-3B-V0.5 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "NousResearch/Obsidian-3B-V0.5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "NousResearch/Obsidian-3B-V0.5", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "NousResearch/Obsidian-3B-V0.5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "NousResearch/Obsidian-3B-V0.5", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use NousResearch/Obsidian-3B-V0.5 with Docker Model Runner:
docker model run hf.co/NousResearch/Obsidian-3B-V0.5
any idea how to make it describe what's physically there and omit the artistic critique?
Using this image:
and this llama.cpp code:
./server -m models/obsidian-f16.gguf --mmproj models/mmproj-obsidian-f16.gguf --host 0.0.0.0 -ngl 42
and this prompt:
<|im_start|>user
What does this image contain? Describe each item, including the color in the description. Only describe physical objects present in the image. Do not make any other comments.\n[img-1]
###
<|im_start|>assistant''',
it replies:
The image features a wooden table with three Legos blocks on it. Two of the blocks are red, and one is blue. They are placed in a way that makes them look like they are standing up against a white wall. This arrangement creates an interesting visual effect that adds depth to the scene. The Legos are positioned in such a way that they appear to be looking at the camera, capturing attention with their vibrant colors and unique design.
This is convincing and all, but I don't really want it to offer a judgement of the blocks. Has anyone had any success making it obey?
