Instructions to use aixonlab/RocRacoon-3b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aixonlab/RocRacoon-3b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aixonlab/RocRacoon-3b", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aixonlab/RocRacoon-3b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("aixonlab/RocRacoon-3b", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use aixonlab/RocRacoon-3b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aixonlab/RocRacoon-3b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aixonlab/RocRacoon-3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/aixonlab/RocRacoon-3b

SGLang

How to use aixonlab/RocRacoon-3b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aixonlab/RocRacoon-3b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aixonlab/RocRacoon-3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aixonlab/RocRacoon-3b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aixonlab/RocRacoon-3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use aixonlab/RocRacoon-3b with Docker Model Runner:
```
docker model run hf.co/aixonlab/RocRacoon-3b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

RocRacoon-3b 🦝

RocRacoon-3b is a versatile language model designed to excel in creative writing, storytelling, and multi-turn conversations. Built on the Phi-3-mini-128k-instruct model, it has been fine-tuned to enhance its contextual understanding and generate more engaging and coherent responses.

Model Details 📊

Developed by: Aixon Lab
Model type: Causal Language Model
Language(s): English (primarily), may support other languages
License: MIT
Repository: https://huggingface.co/aixonlab/RocRacoon-3b

Quantization

GGUF: https://huggingface.co/mradermacher/RocRacoon-3b-GGUF

Model Architecture 🏗️

Base model: microsoft/Phi-3-mini-128k-instruct
Parameter count: ~3 billion
Architecture specifics: Transformer-based language model

Intended Use 🎯

RocRacoon-3b is designed for a wide range of natural language processing tasks, with a particular focus on article writing and topic based multi-turn conversations. It can be used for text generation, dialogue systems, and content creation.

Ethical Considerations 🤔

As a derivative of the Phi-3-mini model, RocRacoon-3b may inherit some biases and limitations. Users should be aware of potential biases in generated content and use the model responsibly, especially in sensitive contexts.

Performance and Evaluation

Comprehensive performance metrics for RocRacoon-3b are currently being compiled. Initial tests show improvements in coherence and creativity compared to the base model. Users are encouraged to contribute their findings and benchmarks.

Limitations and Biases

While efforts have been made to mitigate biases, the model may still exhibit some biases present in its training data. Users should critically evaluate the model's outputs and use them in conjunction with human judgment, particularly for sensitive applications.

Additional Information

For more details on the base Phi-3-mini-128k-instruct model, please refer to its model card and documentation.

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("aixonlab/RocRacoon-3b")
tokenizer = AutoTokenizer.from_pretrained("aixonlab/RocRacoon-3b")

prompt = "Write a short story about a clever raccoon"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

generated_ids = model.generate(input_ids, max_length=200)
generated_text = tokenizer.decode(generated_ids, skip_special_tokens=True)
print(generated_text)

Downloads last month: 4

Safetensors

Model size

4B params

Tensor type

F16

Model tree for aixonlab/RocRacoon-3b

Base model

microsoft/Phi-3-mini-128k-instruct

Finetuned

(44)

this model

Quantizations

1 model