Instructions to use RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits

SGLang

How to use RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits with Docker Model Runner:
```
docker model run hf.co/RichardErkhov/KnutJaegersberg_-_webMistral-7B-8bits
```

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Quantization made by Richard Erkhov.

Github

Discord

Request more models

webMistral-7B - bnb 8bits

Model creator: https://huggingface.co/KnutJaegersberg/
Original model: https://huggingface.co/KnutJaegersberg/webMistral-7B/

Original model description:

license: cc-by-nc-4.0

Prompt Example:

## Question: What is slowing down our internet speeds? 
### Google Search Results: - Are my other devices slowing down my connection? Gadgets with slower internet technology can slow down speeds for all your other devices. Everyone knows the feeling: you’re hard at work and then suddenly the Internet seems to slow down. Why is that? From background programs to cheap routers, here are a few reasons why your Internet might be a concern. When working outside your home, here’s how to keep your information safe on public Wi-Fi. If your internet slows down only when too many other people are online simultaneously, you’re probably using more bandwidth than your plan allows. Use our internet speed test to see if you’re getting the speed advertised by your ISP. If your results are close to your plan speed, consider upgrading. Generally, your modem or router (or both) will create a speed bottleneck if not working properly—the same goes with wireless gateways. If your equipment is too old, it may not support important internet protocols. Equipment damage, such as bad ports or components, can also cause slowdowns. Is your internet suddenly moving slowly? It could be due to an outdated router or a less-than-ideal router location. Your connection issues may need only an easy fix, like upgrading to a mesh network (which also has to be set up in the right spot) or simply restarting your modem and router. But if you've already attempted many of the tried-and-true methods and your internet speeds are still subpar, the issue might be something your internet service provider is intentionally doing: bandwidth throttling. 
### Response: There are several factors that can slow down internet speeds. These include having gadgets with slower internet technology, running background programs[2], using more bandwidth than your plan allows[3], equipment damage[4], an outdated router or a less-than-ideal router location[5], and bandwidth throttling by the internet service provider[5].

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	47.08
ARC (25-shot)	59.04
HellaSwag (10-shot)	80.89
MMLU (5-shot)	59.0
TruthfulQA (0-shot)	39.71
Winogrande (5-shot)	76.32
GSM8K (5-shot)	8.87
DROP (3-shot)	5.75

Downloads last month: 4

Safetensors

Model size

7B params

Tensor type

F32

F16