Instructions to use OmnicromsBrain/NeuralStar_AlphaWriter_4x7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OmnicromsBrain/NeuralStar_AlphaWriter_4x7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OmnicromsBrain/NeuralStar_AlphaWriter_4x7b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OmnicromsBrain/NeuralStar_AlphaWriter_4x7b")
model = AutoModelForCausalLM.from_pretrained("OmnicromsBrain/NeuralStar_AlphaWriter_4x7b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OmnicromsBrain/NeuralStar_AlphaWriter_4x7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OmnicromsBrain/NeuralStar_AlphaWriter_4x7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OmnicromsBrain/NeuralStar_AlphaWriter_4x7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OmnicromsBrain/NeuralStar_AlphaWriter_4x7b

SGLang

How to use OmnicromsBrain/NeuralStar_AlphaWriter_4x7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OmnicromsBrain/NeuralStar_AlphaWriter_4x7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OmnicromsBrain/NeuralStar_AlphaWriter_4x7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OmnicromsBrain/NeuralStar_AlphaWriter_4x7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OmnicromsBrain/NeuralStar_AlphaWriter_4x7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use OmnicromsBrain/NeuralStar_AlphaWriter_4x7b with Docker Model Runner:
```
docker model run hf.co/OmnicromsBrain/NeuralStar_AlphaWriter_4x7b
```

NeuralStar_AlphaWriter_4x7b

Commit History

Update README.md

67136d1
verified

OmnicromsBrain commited on Jun 17, 2024

Update README.md

a76b177
verified

OmnicromsBrain commited on Jun 17, 2024

Adding Evaluation Results (#3)

7a7b92f
verified

OmnicromsBrain

leaderboard-pr-bot commited on Jun 7, 2024

Update README.md

7319bbf
verified

OmnicromsBrain commited on Apr 20, 2024

Update README.md

38da3a8
verified

OmnicromsBrain commited on Apr 20, 2024

Update README.md

50764e2
verified

OmnicromsBrain commited on Apr 20, 2024

Update README.md

30f48bb
verified

OmnicromsBrain commited on Apr 16, 2024

Update README.md

cadde44
verified

OmnicromsBrain commited on Apr 16, 2024

Update README.md

e59d4f7
verified

OmnicromsBrain commited on Apr 16, 2024

Update README.md

1a30d8f
verified

OmnicromsBrain commited on Apr 16, 2024

Upload folder using huggingface_hub

1d92242
verified

OmnicromsBrain commited on Apr 15, 2024

initial commit

b0c6d1b
verified

OmnicromsBrain commited on Apr 15, 2024

Commit History

Update README.md 67136d1 verified

Update README.md a76b177 verified

Adding Evaluation Results (#3) 7a7b92f verified

Update README.md 7319bbf verified

Update README.md 38da3a8 verified

Update README.md 50764e2 verified

Update README.md 30f48bb verified

Update README.md cadde44 verified

Update README.md e59d4f7 verified

Update README.md 1a30d8f verified

Upload folder using huggingface_hub 1d92242 verified

initial commit b0c6d1b verified

Update README.md

67136d1
verified

Update README.md

a76b177
verified

Adding Evaluation Results (#3)

7a7b92f
verified

Update README.md

7319bbf
verified

Update README.md

38da3a8
verified

Update README.md

50764e2
verified

Update README.md

30f48bb
verified

Update README.md

cadde44
verified

Update README.md

e59d4f7
verified

Update README.md

1a30d8f
verified

Upload folder using huggingface_hub

1d92242
verified

initial commit

b0c6d1b
verified