Instructions to use jeiku/Rosa_v2_3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jeiku/Rosa_v2_3B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="jeiku/Rosa_v2_3B", trust_remote_code=True)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("jeiku/Rosa_v2_3B", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use jeiku/Rosa_v2_3B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jeiku/Rosa_v2_3B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jeiku/Rosa_v2_3B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/jeiku/Rosa_v2_3B
- SGLang
How to use jeiku/Rosa_v2_3B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "jeiku/Rosa_v2_3B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jeiku/Rosa_v2_3B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "jeiku/Rosa_v2_3B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jeiku/Rosa_v2_3B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use jeiku/Rosa_v2_3B with Docker Model Runner:
docker model run hf.co/jeiku/Rosa_v2_3B
Whats the system prompt that you use to chat with Rosa?
Hi, Im interested. Would you mind to share the prompts to start chatting with Rosa?
Unfortunately, these 3B models do not adhere to system prompts very well. For the longest time I did not even enable instruct formatting in SillyTavern and only used a custom character card. Lately, I have switched to 7B models as part of my local server, but even with those, I feel uncomfortable sharing my system prompts in this space. I no longer have Rosa's character card and I regenerate Luna often using this: https://chargen.kubes-lab.com/
@jeiku understood! Thanks for the reply :pray:
If you are willing to share with me any system prompt you can find me on Discord davidgortega
Thanks!!