Instructions to use nothingiisreal/phi-3-mini-zeta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nothingiisreal/phi-3-mini-zeta with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nothingiisreal/phi-3-mini-zeta", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nothingiisreal/phi-3-mini-zeta", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("nothingiisreal/phi-3-mini-zeta", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nothingiisreal/phi-3-mini-zeta with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nothingiisreal/phi-3-mini-zeta"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nothingiisreal/phi-3-mini-zeta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nothingiisreal/phi-3-mini-zeta

SGLang

How to use nothingiisreal/phi-3-mini-zeta with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nothingiisreal/phi-3-mini-zeta" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nothingiisreal/phi-3-mini-zeta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nothingiisreal/phi-3-mini-zeta" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nothingiisreal/phi-3-mini-zeta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nothingiisreal/phi-3-mini-zeta with Docker Model Runner:
```
docker model run hf.co/nothingiisreal/phi-3-mini-zeta
```

We went ahead and ran abliteration 3 times over (or maybe it was 4)

Process is as follows:

Run abliteration
Save model
Re-run it

Re-running it like this does make the effect stronger.

We didn't run it to decensor it, we were trying to get rid of GPT Slopping and make its writing more natural, which we kinda achieved.

We don't recommend people spend more effort and time in this direction, at least I couldn't get much more success, I think building our own RLHF and ORPO open source structures will be way more rewarding in the long term in combatting draconian content policies.

We used a jupyter notebook you can find in the repo. Make sure to run install_deps.sh first!

Downloads last month: 3

Safetensors

Model size

4B params

Tensor type

F16