Instructions to use ReBatch/Llama-3-8B-dutch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ReBatch/Llama-3-8B-dutch with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ReBatch/Llama-3-8B-dutch")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ReBatch/Llama-3-8B-dutch")
model = AutoModelForCausalLM.from_pretrained("ReBatch/Llama-3-8B-dutch")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ReBatch/Llama-3-8B-dutch with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ReBatch/Llama-3-8B-dutch"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ReBatch/Llama-3-8B-dutch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ReBatch/Llama-3-8B-dutch

SGLang

How to use ReBatch/Llama-3-8B-dutch with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ReBatch/Llama-3-8B-dutch" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ReBatch/Llama-3-8B-dutch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ReBatch/Llama-3-8B-dutch" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ReBatch/Llama-3-8B-dutch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ReBatch/Llama-3-8B-dutch with Docker Model Runner:
```
docker model run hf.co/ReBatch/Llama-3-8B-dutch
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Llama 3 8B - Dutch

A conversational model for Dutch, based on Llama 3 8B

Try chatting with the model!

This model is a QLORA and ORPO fine-tuned version of meta-llama/Meta-Llama-3-8B on the synthetic feedback dataset BramVanroy/ultra_feedback_dutch

Model description

This model is a Dutch chat model, originally developed from Llama 3 8B and further refined through a feedback dataset with ORPO and trained on BramVanroy/ultra_feedback_dutch

Intended uses & limitations

Although the model has been aligned with gpt-4-turbo output, which has strong content filters, the model could still generate wrong, misleading, and potentially even offensive content. Use at your own risk.

Training procedure

The model was trained in bfloat16 with QLORA with flash attention 2 on one GPU - H100 80GB SXM5 for around 24 hours on RunPod.

Evaluation Results

The model was evaluated using scandeval

The model showed mixed results across different benchmarks; it exhibited slight improvements on some while experiencing a decrease in scores on others. This occurred despite being trained on only 200,000 samples for a single epoch. We are curious to see whether its performance could be enhanced by training with more data or additional epochs.

Model	conll_nl	dutch_social	scala_nl	squad_nl	wiki_lingua_nl	mmlu_nl	hellaswag_nl
meta-llama/Meta-Llama-3-8B-Instruct	68.72	14.67	32.91	45.36	67.62	36.18	33.91
ReBatch/Llama-3-8B-dutch	58.85	11.14	15.58	59.96	64.51	36.27	28.34
meta-llama/Meta-Llama-3-8B	62.26	10.45	30.3	62.99	65.17	36.38	28.33

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 8e-06
train_batch_size: 2
eval_batch_size: 2
num_devices: 1
gradient_accumulation_steps: 4
optimizer: paged_adamw_8bit
lr_scheduler_type: linear
warmup_steps: 10
num_epochs: 1.0
r: 16
lora_alpha: 32
lora_dropout: 0.05

Downloads last month: 239

Safetensors

Model size

8B params

Tensor type

F16

Model tree for ReBatch/Llama-3-8B-dutch

Base model

meta-llama/Meta-Llama-3-8B

Finetuned

(595)

this model

Quantizations

3 models

ReBatch
/

Llama-3-8B-dutch