Instructions to use fblgit/LUNA-SOLARkrautLM-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fblgit/LUNA-SOLARkrautLM-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fblgit/LUNA-SOLARkrautLM-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("fblgit/LUNA-SOLARkrautLM-Instruct")
model = AutoModelForCausalLM.from_pretrained("fblgit/LUNA-SOLARkrautLM-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use fblgit/LUNA-SOLARkrautLM-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fblgit/LUNA-SOLARkrautLM-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fblgit/LUNA-SOLARkrautLM-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/fblgit/LUNA-SOLARkrautLM-Instruct

SGLang

How to use fblgit/LUNA-SOLARkrautLM-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fblgit/LUNA-SOLARkrautLM-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fblgit/LUNA-SOLARkrautLM-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fblgit/LUNA-SOLARkrautLM-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fblgit/LUNA-SOLARkrautLM-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use fblgit/LUNA-SOLARkrautLM-Instruct with Docker Model Runner:
```
docker model run hf.co/fblgit/LUNA-SOLARkrautLM-Instruct
```

fblgit commited on Dec 22, 2023

Commit

bc405bc

1 Parent(s): 6b69b7d

Update README.md

Browse files

Files changed (1) hide show

README.md +74 -0

README.md CHANGED Viewed

@@ -138,6 +138,80 @@ hf (pretrained=fblgit/LUNA-SOLARkrautLM-Instruct), gen_kwargs: (), limit: None,
 |           |       |none  |     5|ter   |63.9997|±  |0.4591|
 |           |       |none  |     5|chrf  |51.1399|±  |0.3978|
 |xnli_de    |Yaml   |none  |     5|acc   | 0.4703|±  |0.0100|
 ```
 ## Disclaimer

 |           |       |none  |     5|ter   |63.9997|±  |0.4591|
 |           |       |none  |     5|chrf  |51.1399|±  |0.3978|
 |xnli_de    |Yaml   |none  |     5|acc   | 0.4703|±  |0.0100|
+hf (pretrained=fblgit/LUNA-SOLARkrautLM-Instruct,dtype=float16), gen_kwargs: (), limit: None, num_fewshot: 5, batch_size: auto (16)
+|                 Tasks                 |Version|Filter|n-shot|Metric|Value |   |Stderr|
+|---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
+|mmlu                                   |N/A    |none  |     0|acc   |0.6461|±  |0.1215|
+| - humanities                          |N/A    |none  |     5|acc   |0.5960|±  |0.1200|
+|  - formal_logic                       |Yaml   |none  |     5|acc   |0.4683|±  |0.0446|
+|  - high_school_european_history       |Yaml   |none  |     5|acc   |0.8121|±  |0.0305|
+|  - high_school_us_history             |Yaml   |none  |     5|acc   |0.8480|±  |0.0252|
+|  - high_school_world_history          |Yaml   |none  |     5|acc   |0.8312|±  |0.0244|
+|  - international_law                  |Yaml   |none  |     5|acc   |0.7851|±  |0.0375|
+|  - jurisprudence                      |Yaml   |none  |     5|acc   |0.7685|±  |0.0408|
+|  - logical_fallacies                  |Yaml   |none  |     5|acc   |0.7423|±  |0.0344|
+|  - moral_disputes                     |Yaml   |none  |     5|acc   |0.7283|±  |0.0239|
+|  - moral_scenarios                    |Yaml   |none  |     5|acc   |0.3899|±  |0.0163|
+|  - philosophy                         |Yaml   |none  |     5|acc   |0.7074|±  |0.0258|
+|  - prehistory                         |Yaml   |none  |     5|acc   |0.7716|±  |0.0234|
+|  - professional_law                   |Yaml   |none  |     5|acc   |0.4824|±  |0.0128|
+|  - world_religions                    |Yaml   |none  |     5|acc   |0.7661|±  |0.0325|
+| - other                               |N/A    |none  |     5|acc   |0.7097|±  |0.0900|
+|  - business_ethics                    |Yaml   |none  |     5|acc   |0.7700|±  |0.0423|
+|  - clinical_knowledge                 |Yaml   |none  |     5|acc   |0.6792|±  |0.0287|
+|  - college_medicine                   |Yaml   |none  |     5|acc   |0.6647|±  |0.0360|
+|  - global_facts                       |Yaml   |none  |     5|acc   |0.3600|±  |0.0482|
+|  - human_aging                        |Yaml   |none  |     5|acc   |0.6861|±  |0.0311|
+|  - management                         |Yaml   |none  |     5|acc   |0.8350|±  |0.0368|
+|  - marketing                          |Yaml   |none  |     5|acc   |0.8504|±  |0.0234|
+|  - medical_genetics                   |Yaml   |none  |     5|acc   |0.6700|±  |0.0473|
+|  - miscellaneous                      |Yaml   |none  |     5|acc   |0.7893|±  |0.0146|
+|  - nutrition                          |Yaml   |none  |     5|acc   |0.7549|±  |0.0246|
+|  - professional_accounting            |Yaml   |none  |     5|acc   |0.5213|±  |0.0298|
+|  - professional_medicine              |Yaml   |none  |     5|acc   |0.7353|±  |0.0268|
+|  - virology                           |Yaml   |none  |     5|acc   |0.5783|±  |0.0384|
+| - social_sciences                     |N/A    |none  |     5|acc   |0.7501|±  |0.0684|
+|  - econometrics                       |Yaml   |none  |     5|acc   |0.5175|±  |0.0470|
+|  - high_school_geography              |Yaml   |none  |     5|acc   |0.8485|±  |0.0255|
+|  - high_school_government_and_politics|Yaml   |none  |     5|acc   |0.8912|±  |0.0225|
+|  - high_school_macroeconomics         |Yaml   |none  |     5|acc   |0.6615|±  |0.0240|
+|  - high_school_microeconomics         |Yaml   |none  |     5|acc   |0.7311|±  |0.0288|
+|  - high_school_psychology             |Yaml   |none  |     5|acc   |0.8385|±  |0.0158|
+|  - human_sexuality                    |Yaml   |none  |     5|acc   |0.7023|±  |0.0401|
+|  - professional_psychology            |Yaml   |none  |     5|acc   |0.6683|±  |0.0190|
+|  - public_relations                   |Yaml   |none  |     5|acc   |0.6909|±  |0.0443|
+|  - security_studies                   |Yaml   |none  |     5|acc   |0.7633|±  |0.0272|
+|  - sociology                          |Yaml   |none  |     5|acc   |0.8358|±  |0.0262|
+|  - us_foreign_policy                  |Yaml   |none  |     5|acc   |0.8800|±  |0.0327|
+| - stem                                |N/A    |none  |     5|acc   |0.5569|±  |0.1360|
+|  - abstract_algebra                   |Yaml   |none  |     5|acc   |0.3800|±  |0.0488|
+|  - anatomy                            |Yaml   |none  |     5|acc   |0.6148|±  |0.0420|
+|  - astronomy                          |Yaml   |none  |     5|acc   |0.7237|±  |0.0364|
+|  - college_biology                    |Yaml   |none  |     5|acc   |0.7708|±  |0.0351|
+|  - college_chemistry                  |Yaml   |none  |     5|acc   |0.4600|±  |0.0501|
+|  - college_computer_science           |Yaml   |none  |     5|acc   |0.5400|±  |0.0501|
+|  - college_mathematics                |Yaml   |none  |     5|acc   |0.2700|±  |0.0446|
+|  - college_physics                    |Yaml   |none  |     5|acc   |0.3333|±  |0.0469|
+|  - computer_security                  |Yaml   |none  |     5|acc   |0.7300|±  |0.0446|
+|  - conceptual_physics                 |Yaml   |none  |     5|acc   |0.6213|±  |0.0317|
+|  - electrical_engineering             |Yaml   |none  |     5|acc   |0.6276|±  |0.0403|
+|  - elementary_mathematics             |Yaml   |none  |     5|acc   |0.4788|±  |0.0257|
+|  - high_school_biology                |Yaml   |none  |     5|acc   |0.8065|±  |0.0225|
+|  - high_school_chemistry              |Yaml   |none  |     5|acc   |0.5123|±  |0.0352|
+|  - high_school_computer_science       |Yaml   |none  |     5|acc   |0.7000|±  |0.0461|
+|  - high_school_mathematics            |Yaml   |none  |     5|acc   |0.3889|±  |0.0297|
+|  - high_school_physics                |Yaml   |none  |     5|acc   |0.3576|±  |0.0391|
+|  - high_school_statistics             |Yaml   |none  |     5|acc   |0.5926|±  |0.0335|
+|  - machine_learning                   |Yaml   |none  |     5|acc   |0.4554|±  |0.0473|
+|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
+|------------------|-------|------|-----:|------|-----:|---|-----:|
+|mmlu              |N/A    |none  |     0|acc   |0.6461|±  |0.1215|
+| - humanities     |N/A    |none  |     5|acc   |0.5960|±  |0.1200|
+| - other          |N/A    |none  |     5|acc   |0.7097|±  |0.0900|
+| - social_sciences|N/A    |none  |     5|acc   |0.7501|±  |0.0684|
+| - stem           |N/A    |none  |     5|acc   |0.5569|±  |0.1360|
 ```
 ## Disclaimer