Instructions to use Felladrin/Minueza-32M-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Felladrin/Minueza-32M-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Felladrin/Minueza-32M-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Felladrin/Minueza-32M-Chat")
model = AutoModelForCausalLM.from_pretrained("Felladrin/Minueza-32M-Chat", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Felladrin/Minueza-32M-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Felladrin/Minueza-32M-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Felladrin/Minueza-32M-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Felladrin/Minueza-32M-Chat

SGLang

How to use Felladrin/Minueza-32M-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Felladrin/Minueza-32M-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Felladrin/Minueza-32M-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Felladrin/Minueza-32M-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Felladrin/Minueza-32M-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Felladrin/Minueza-32M-Chat with Docker Model Runner:
```
docker model run hf.co/Felladrin/Minueza-32M-Chat
```

Hallucination

by cnmoro - opened Feb 26, 2024

Discussion

cnmoro

Feb 26, 2024

Hey!
I have also been messing around for a while with smaller models - I have also tried finetuning some models that you have posted, for some specific tasks.
Have you found a real application for them yet?
I've noticed that any model that has less than 1B params tends to hallucinate a lot.
Just curious :)

aloobun

Feb 26, 2024

@cnmoro small models are fun to play with for expanding storylines, completing drafts and/or roleplay. Qwen 0.5B being a small model is a beast at function calling and had a pass rate of 77% on function calling eval dataset.

Felladrin

Owner Feb 26, 2024

Hey, @cnmoro ! I'm also following your projects!

As @aloobun mentioned, those small models are great for storytelling/RP. I also find them good at zero-shot Q&A over specific topics.

But during chats I also see all of them hallucinating a lot. However, I did put some of those in production, with a different purpose:

In MiniSearch, when not running on a WebGPU-supported web browser, it uses these small models via Transformers.js.
Currently, it's running these models: onnx-Pythia-31M-Chat-v1, onnx-Smol-Llama-101M-Chat-v1, onnx-Llama-160M-Chat-v1. [Reference]

I opted to use those because, when using Transformers.js v2, the inference speed on browsers slows down significantly on models larger than that. [More info here].

It's also worth noting that smartphones' web browsers can handle 30M ONNX models without requiring quantization. This is important because when quantization is applied to small models, it can lead to significant differences in the output. For example, an unquantized 30M model can perform better than a quantized 100M model.

And although it is possible to convert them to GGUF and run llama.cpp through WASM (LLM.js & llama-cpp-wasm), the ONNX runtime remains the fastest option for inference on mobile devices.

Felladrin

Owner Mar 1, 2024

I believe that the combination of different styles from the dataset mix in this model has increased the chance of hallucination. To verify it, I've fine-tuned Minueza-32M-Base with a single, large dataset, and the results were less prone to hallucinations. This new model, Felladrin/Minueza-32M-UltraChat, can be tested in Models Playground.

By the way, I forgot to mention earlier that there is another way to further reduce hallucinations: Contrastive Search. This strategy is great for question-and-answer or instruction-response scenarios. However, it may have a negative impact on multi-turn chats.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment