Instructions to use mattshumer/ref_70_e3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mattshumer/ref_70_e3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mattshumer/ref_70_e3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mattshumer/ref_70_e3")
model = AutoModelForCausalLM.from_pretrained("mattshumer/ref_70_e3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use mattshumer/ref_70_e3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mattshumer/ref_70_e3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mattshumer/ref_70_e3

SGLang

How to use mattshumer/ref_70_e3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mattshumer/ref_70_e3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mattshumer/ref_70_e3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mattshumer/ref_70_e3 with Docker Model Runner:
```
docker model run hf.co/mattshumer/ref_70_e3
```

🚩 Report: Ethical issue(s)

#142

by Impulse2000 - opened Sep 24, 2024

Discussion

Impulse2000

Sep 24, 2024

The Evals are not reproducible (https://x.com/ArtificialAnlys/status/1832457791010959539/photo/1)
Even the provider HyperBolic and all other providers have stopped hosting the model (Lack of Adoption means its not effective)
Matt schumer did not disclose he is an investor in Glaive (the platform he promoted)
Huge Claims made about the model (being the best model in the world)
HF evals are not very good. it scores a 30.74 on average on the HF 2 leaderboard. the original llama3.1 70b model scores 41.74 on average.

I don't post this report lightly, i waited a while, gathering information on this.

The action i think is appropriate, is getting the model author to correct their model README.md, lowering the claims made, and showing the real benchmarks, which have been pushed to this repo probably a hundred times by now, but Matt Schumer refuses to accept the real benchmarks.

If they fail to do this within a reasonably time period, like 30 days. i would reccomend you remove the model from the platform or put a disclaimer at the top of the model card, saying its claims are proven false.

Thanks for reading,
James Clarke.
Founder of Novora LLC.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment