Instructions to use DiscoResearch/DiscoLM_German_7b_v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DiscoResearch/DiscoLM_German_7b_v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DiscoResearch/DiscoLM_German_7b_v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DiscoResearch/DiscoLM_German_7b_v1")
model = AutoModelForCausalLM.from_pretrained("DiscoResearch/DiscoLM_German_7b_v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DiscoResearch/DiscoLM_German_7b_v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DiscoResearch/DiscoLM_German_7b_v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DiscoResearch/DiscoLM_German_7b_v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DiscoResearch/DiscoLM_German_7b_v1

SGLang

How to use DiscoResearch/DiscoLM_German_7b_v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DiscoResearch/DiscoLM_German_7b_v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DiscoResearch/DiscoLM_German_7b_v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DiscoResearch/DiscoLM_German_7b_v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DiscoResearch/DiscoLM_German_7b_v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use DiscoResearch/DiscoLM_German_7b_v1 with Docker Model Runner:
```
docker model run hf.co/DiscoResearch/DiscoLM_German_7b_v1
```

Endless Spaces

by vacanickel - opened Jan 20, 2024

Discussion

vacanickel

Jan 20, 2024

Tried the gguf version with ollama, used the right template, it does generate endless empty spaces and lines after an answer.

Blizado

Jan 20, 2024

•

edited Jan 21, 2024

Same here, SillyTavern shows many ' \n' at the end of the generation up to the full context size but filtering it out before it shows it in the WebUI. But this slows down the response time.

Update: it didn't happen with the original files here, but on the gguf, so the issue is there not here.

genericgod

Jan 22, 2024

Same. Converted it to gguf myself but same problem.

jphme

Disco Research org Jan 23, 2024

this should be fixed now with this change, sorry for the oversight: https://huggingface.co/DiscoResearch/DiscoLM_German_7b_v1/commit/560f972f9f735fc9289584b3aa8d75d0e539c44e

Will ping TheBloke to reup quants as soon as we´ve confirmed everything is working now. Thanks everybody for reporting this issue!

jjherb

Oct 2, 2024

•

edited Oct 2, 2024

I have exactly this problem, if I use this model with LangChain and do not set explicitly the eos_token_id:

# Loading Mistral 7b model 
llm = HuggingFacePipeline.from_model_id(
    model_id='DiscoResearch/DiscoLM_German_7b_v1',
    task='text-generation',
    model_kwargs={
        'temperature': .3,
        'max_length': 1024,
        'quantization_config': quantization_config,
        'low_cpu_mem_usage': True,
        
    },
    pipeline_kwargs={
        "max_new_tokens": 2000, 
        "eos_token_id": 32000 # needed to avoid "endless spaces"!
    },
    device_map="auto",
    device=None,
    
)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment