Instructions to use iFaz/llama32_3B_en_emo_v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use iFaz/llama32_3B_en_emo_v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="iFaz/llama32_3B_en_emo_v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("iFaz/llama32_3B_en_emo_v2")
model = AutoModelForCausalLM.from_pretrained("iFaz/llama32_3B_en_emo_v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use iFaz/llama32_3B_en_emo_v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "iFaz/llama32_3B_en_emo_v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "iFaz/llama32_3B_en_emo_v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/iFaz/llama32_3B_en_emo_v2

SGLang

How to use iFaz/llama32_3B_en_emo_v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "iFaz/llama32_3B_en_emo_v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "iFaz/llama32_3B_en_emo_v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "iFaz/llama32_3B_en_emo_v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "iFaz/llama32_3B_en_emo_v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio

How to use iFaz/llama32_3B_en_emo_v2 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for iFaz/llama32_3B_en_emo_v2 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for iFaz/llama32_3B_en_emo_v2 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for iFaz/llama32_3B_en_emo_v2 to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="iFaz/llama32_3B_en_emo_v2",
    max_seq_length=2048,
)

Docker Model Runner
How to use iFaz/llama32_3B_en_emo_v2 with Docker Model Runner:
```
docker model run hf.co/iFaz/llama32_3B_en_emo_v2
```

Uploaded model

Developed by: iFaz
License: apache-2.0
Finetuned from model : unsloth/Llama-3.2-3B-Instruct-bnb-4bit

Model Card: `unsloth/Llama-3.2-3B-Instruct-bnb-4bit`

Overview

This is a fine-tuned version of the unsloth/Llama-3.2-3B-Instruct-bnb-4bit model, optimized for instruction-following tasks. The model leverages the efficiency of 4-bit quantization, making it lightweight and resource-efficient while maintaining high-quality outputs. It is particularly suited for text generation tasks in English, with applications ranging from conversational AI to natural language understanding tasks.

Key Features

Base Model: unsloth/Llama-3.2-3B
Quantization: Utilizes 4-bit precision, enabling deployment on resource-constrained systems while maintaining performance.
Language: English-focused, with robust generalization capabilities across diverse text-generation tasks.
Fine-Tuning: Enhanced for instruction-following tasks to generate coherent and contextually relevant responses.
Versatile Applications: Ideal for text generation, summarization, dialogue systems, and other natural language processing (NLP) tasks.

Model Details

Developer: iFaz
License: Apache 2.0 (permitting commercial and research use)
Tags:
- Text generation inference
- Transformers
- Unsloth
- LLaMA
- TRL (Transformers Reinforcement Learning)

Usage

This model is designed for use in text-generation pipelines and can be easily integrated with the Hugging Face Transformers library. Its optimized architecture allows for inference on low-resource hardware, making it an excellent choice for applications that require efficient and scalable NLP solutions.

Example Code:

from transformers import AutoTokenizer, AutoModelForCausalLM
# Load the model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("iFaz/llama32_3B_en_emo_v1")
model = AutoModelForCausalLM.from_pretrained("iFaz/llama32_3B_en_emo_v1")
# Generate text
input_text = "Explain the benefits of AI in education."
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance

The fine-tuned model demonstrates strong performance on instruction-based tasks, providing detailed and contextually accurate responses. The 4-bit quantization enhances its speed and reduces memory consumption, enabling usage on devices with limited computational resources.

Applications

Conversational AI: Develop chatbots and virtual assistants with coherent, context-aware dialogue generation.
Text Summarization: Extract concise summaries from lengthy texts for improved readability.
Creative Writing: Assist in generating stories, articles, or creative content.
Education: Enhance e-learning platforms with interactive and adaptive learning tools.

Limitations and Considerations

Language Limitation: Currently optimized for English. Performance on other languages may be suboptimal.
Domain-Specific Knowledge: While the model performs well on general tasks, it may require additional fine-tuning for domain-specific applications.

About the Developer

This model was developed and fine-tuned by iFaz, leveraging the capabilities of the unsloth/Llama-3.2-3B architecture to create an efficient and high-performance NLP tool.

Acknowledgments

The model builds upon the unsloth/Llama-3.2-3B framework and incorporates advancements in quantization techniques. Special thanks to the Hugging Face community for providing tools and resources to support NLP development.

License

The model is distributed under the Apache 2.0 License, allowing for both research and commercial use. For more details, refer to the license documentation.

Downloads last month: 5

Safetensors

Model size

3B params

Tensor type

F32

F16

Model tree for iFaz/llama32_3B_en_emo_v2

Base model

meta-llama/Llama-3.2-3B-Instruct

Quantized

unsloth/Llama-3.2-3B-Instruct-bnb-4bit