Instructions to use fblgit/LUNA-SOLARkrautLM-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fblgit/LUNA-SOLARkrautLM-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fblgit/LUNA-SOLARkrautLM-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("fblgit/LUNA-SOLARkrautLM-Instruct")
model = AutoModelForCausalLM.from_pretrained("fblgit/LUNA-SOLARkrautLM-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Local Apps Settings

vLLM

How to use fblgit/LUNA-SOLARkrautLM-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fblgit/LUNA-SOLARkrautLM-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fblgit/LUNA-SOLARkrautLM-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/fblgit/LUNA-SOLARkrautLM-Instruct

SGLang

How to use fblgit/LUNA-SOLARkrautLM-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fblgit/LUNA-SOLARkrautLM-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fblgit/LUNA-SOLARkrautLM-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fblgit/LUNA-SOLARkrautLM-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fblgit/LUNA-SOLARkrautLM-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use fblgit/LUNA-SOLARkrautLM-Instruct with Docker Model Runner:
```
docker model run hf.co/fblgit/LUNA-SOLARkrautLM-Instruct
```

Update README.md

by DavidGF - opened Dec 22, 2023

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

-11

Files changed (1) hide show

README.md +8 -11

README.md CHANGED Viewed

@@ -73,15 +73,13 @@ Our results, with `result < 0.1, %:` being well below 0.9, indicate that our dat
 ### Prompt Template:
 ```
-### User:
-Hallo, wie geht es dir?
-### Assistant:
-Hallo! Es freut mich, dass du mit mir kommunizierst. Ich bin hier, um zu helfen und deine Anfragen zu erfüllen. Du fragst, wie ich mich fühle. Als künstliche Intelligenz habe ich keine eigentlichen Emotionen im Sinne eines Menschen, aber ich funktioniere optimal und bin bereit, Dienste anzubieten.
-Wie geht es dir momentan? Können wir zusammen etwas interessantes oder hilfreiches erledigen?
 ```
-*Prompt Example on Temp 0.5
 ```
 ### User:
@@ -92,7 +90,6 @@ Hi there! I am an AI language model, so I don't have personal feelings or emotio
 How may I assist you today?
 ```
-*Prompt Example on Temp 0.5
 ## Evaluation
 ```
@@ -265,11 +262,11 @@ Additionally, it is essential to understand that the licensing of these models d
 If you are interested in customized LLMs for business applications, please get in contact with us via our website or contact us at [Dr. Daryoush Vaziri](mailto:vaziri@vago-solutions.de). We are also grateful for your feedback and suggestions.
 ## Collaborations
-We are also keenly seeking support and investment for our startup, VAGO solutions, where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us.
-Juanako.AI is also seeking support and investment for our startup, we also are open for collaborating with other labs to make awesome models like this one.
 ## Acknowledgement
-Big Hug to VAGO Solutions, we merely used our transformers library on their code and dataset, nothing else. This won't be possible without them, thanks!
 Many thanks to [argilla](https://huggingface.co/datasets/argilla) and [Huggingface](https://huggingface.co) for providing such valuable datasets to the Open-Source community. And of course a big thanks to [upstage](https://huggingface.co/upstage) for providing the open source community with their latest technology!

 ### Prompt Template:
 ```
+<|im_start|>system
+Du bist LUNA-SOLARkrautLM, ein großes Sprachmodell, das höflich und kompetent antwortet.<|im_end|>
+<|im_start|>user
+Wie geht es dir?<|im_end|>
+<|im_start|>assistant
 ```
 ```
 ### User:
 How may I assist you today?
 ```
 ## Evaluation
 ```
 If you are interested in customized LLMs for business applications, please get in contact with us via our website or contact us at [Dr. Daryoush Vaziri](mailto:vaziri@vago-solutions.de). We are also grateful for your feedback and suggestions.
 ## Collaborations
+We are also keenly seeking support and investment for our startup, [VAGO Solutions](https://huggingface.co/VAGOsolutions), where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us.
+[Juanako.AI](https://huggingface.co/fblgitis) also seeking support and investment for our startup, we also are open for collaborating with other labs to make awesome models like this one.
 ## Acknowledgement
+Big Hug to [VAGO Solutions](https://huggingface.co/VAGOsolutions), we merely used our transformers library on their code and dataset, nothing else. This won't be possible without them, thanks!
 Many thanks to [argilla](https://huggingface.co/datasets/argilla) and [Huggingface](https://huggingface.co) for providing such valuable datasets to the Open-Source community. And of course a big thanks to [upstage](https://huggingface.co/upstage) for providing the open source community with their latest technology!