Instructions to use GreenBitAI/yi-34b-w4a16g32 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use GreenBitAI/yi-34b-w4a16g32 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="GreenBitAI/yi-34b-w4a16g32")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("GreenBitAI/yi-34b-w4a16g32")
model = AutoModelForCausalLM.from_pretrained("GreenBitAI/yi-34b-w4a16g32")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use GreenBitAI/yi-34b-w4a16g32 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "GreenBitAI/yi-34b-w4a16g32"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreenBitAI/yi-34b-w4a16g32",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/GreenBitAI/yi-34b-w4a16g32

SGLang

How to use GreenBitAI/yi-34b-w4a16g32 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "GreenBitAI/yi-34b-w4a16g32" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreenBitAI/yi-34b-w4a16g32",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "GreenBitAI/yi-34b-w4a16g32" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreenBitAI/yi-34b-w4a16g32",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use GreenBitAI/yi-34b-w4a16g32 with Docker Model Runner:
```
docker model run hf.co/GreenBitAI/yi-34b-w4a16g32
```

New exciting quant method

by Yhyu13 - opened Dec 25, 2023

Discussion

Yhyu13

Dec 25, 2023

@TheBloke @LoneStriker

Hi, checkout out this quant method which have best performance to model size even compare to GPTQ https://github.com/GreenBitAI/low_bit_llama

LoneStriker

Dec 25, 2023

Thank you for the pointer to this new quant method. I had not heard of it before. Have you compared the perplexity to llama.cpp's GGUF and Exllamav2's exl2 models at the same model sizes? It would help adoption of any new quantization method if there were equivalent measurements done of both GreenBitAI's quantizations vs. llama.cpp and Exllamav2.

@TheBloke creates llama.cpp models at sizes ranging from Q2 to Q8, Yi models here:
https://huggingface.co/TheBloke/Yi-34B-GGUF/tree/main

I've generated quants from 3.0 bpw to 8.0 bpw here:
https://huggingface.co/models?p=2&sort=created&search=lonestriker%2Fyi-34b

exllamav2 is also capable of fractional bit-quantization because the stated bitrate is an average over the layers. A popular low-bit quant size for Exllamav2 is 2.4 bpw (especially for 70B models.) At 2.4 bpw, a 70B model fits in a single 3090 or 4090 with full contxt. The model size is only 20 GB at 2.4 bpw and 22 GB at 2.65 bpw. Example models here:
https://huggingface.co/models?sort=modified&search=lonestriker+2.4bpw+70b

And for Mixtral, at 3.0 bpw, the model will run entirely in a single 3090 or 4090 at full 32K context:
https://huggingface.co/models?sort=modified&search=lonestriker+3.0bpw+8x7

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment