Instructions to use P0x0/Epos-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use P0x0/Epos-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="P0x0/Epos-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("P0x0/Epos-8b")
model = AutoModelForCausalLM.from_pretrained("P0x0/Epos-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use P0x0/Epos-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "P0x0/Epos-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "P0x0/Epos-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/P0x0/Epos-8b

SGLang

How to use P0x0/Epos-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "P0x0/Epos-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "P0x0/Epos-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "P0x0/Epos-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "P0x0/Epos-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use P0x0/Epos-8b with Docker Model Runner:
```
docker model run hf.co/P0x0/Epos-8b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Epos-8B

Epos-8B is a fine-tuned version of the base model Llama-3.1-8B from Meta, optimized for storytelling, dialogue generation, and creative writing. The model specializes in generating rich narratives, immersive prose, and dynamic character interactions, making it ideal for creative tasks.

Model Details

Model Description

Epos-8B is an 8 billion parameter language model fine-tuned for storytelling and narrative tasks.

Developed by: P0x0
Funded by: P0x0
Shared by: P0x0
Model type: Transformer-based Language Model
Language(s) (NLP): Primarily English
License: Apache 2.0
Finetuned from model: meta-llama/Llama-3.1-8B

Model Sources

Repository: Epos-8B on Hugging Face
GGUF: GGUF by mradermache
imatrix GGUF:imatrix quants by mradermacher

Uses

Direct Use

Epos-8B is ideal for:

Storytelling: Generate detailed, immersive, and engaging narratives.
Dialogue Creation: Create realistic and dynamic character interactions for stories or games.

How to Get Started with the Model

To run the quantized version of the model, you can use KoboldCPP, which allows you to run quantized GGUF models locally.

Steps:

Download KoboldCPP.
Follow the setup instructions provided in the repository.
Download the GGUF variant of Epos-8B from Epos-8B-GGUF.
Load the model in KoboldCPP and start generating!

Alternatively, integrate the model directly into your code with the following snippet:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("P0x0/Epos-8B")
model = AutoModelForCausalLM.from_pretrained("P0x0/Epos-8B")

input_text = "Once upon a time in a distant land..."
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))