kaiokendev/SuperCOT-dataset
Viewer • Updated • 58.3k • 30 • 46
How to use lloorree/mythxl-70b with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("text-generation", model="lloorree/mythxl-70b") # Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("lloorree/mythxl-70b")
model = AutoModelForCausalLM.from_pretrained("lloorree/mythxl-70b")How to use lloorree/mythxl-70b with vLLM:
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lloorree/mythxl-70b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "lloorree/mythxl-70b",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'docker model run hf.co/lloorree/mythxl-70b
How to use lloorree/mythxl-70b with SGLang:
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
--model-path "lloorree/mythxl-70b" \
--host 0.0.0.0 \
--port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "lloorree/mythxl-70b",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "lloorree/mythxl-70b" \
--host 0.0.0.0 \
--port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "lloorree/mythxl-70b",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'How to use lloorree/mythxl-70b with Docker Model Runner:
docker model run hf.co/lloorree/mythxl-70b
70B recreation of MythoMax.
Differences:
Known limitation: it strongly prefers novel format in roleplay, and will revert to it over time regardless of context or conversation history.
License is strictly noncommercial, both to match that of its major dependency Chronos 70B and in its own right.
This model primarily uses Alpaca formatting, so for optimal model performance, use:
<System prompt/Character Card>
### Instruction:
Your instruction or question here.
For roleplay purposes, I suggest the following - Write <CHAR NAME>'s next reply in a chat between <YOUR NAME> and <CHAR NAME>. Write a single reply only.
### Response: