Instructions to use justheuristic/test-1bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use justheuristic/test-1bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="justheuristic/test-1bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("justheuristic/test-1bit")
model = AutoModelForCausalLM.from_pretrained("justheuristic/test-1bit")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use justheuristic/test-1bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "justheuristic/test-1bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "justheuristic/test-1bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/justheuristic/test-1bit

SGLang

How to use justheuristic/test-1bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "justheuristic/test-1bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "justheuristic/test-1bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "justheuristic/test-1bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "justheuristic/test-1bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use justheuristic/test-1bit with Docker Model Runner:
```
docker model run hf.co/justheuristic/test-1bit
```

BlackSamorez

justheuristic commited on May 30, 2024

Commit

4c53d47

verified ·

1 Parent(s): 0ab711f

Create model card (#1)

Browse files

- Create model card (dc6871874c2cc635d34d0f61c47a7af2eda2b7a4)
- Update README.md (1e4dd958ec290ac197ff9c8869de42b7d7702bf4)
- Update README.md (8315dcabcaa219d38cbbbec76e5bbec947181186)
- Update README.md (4bd28af77e457999364dcce722348a84443737c6)

Co-authored-by: Yozh <justheuristic@users.noreply.huggingface.co>

Files changed (1) hide show

README.md +29 -0

README.md ADDED Viewed

	@@ -0,0 +1,29 @@

+---
+library_name: transformers
+tags:
+- llama
+- facebook
+- meta
+- llama-2
+- conversational
+- text-generation-inference
+---
+An official quantization of [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b) using [PV-Tuning](https://arxiv.org/abs/2405.14852) on top of [AQLM](https://arxiv.org/abs/2401.06118).
+For this quantization, we used 1 codebook of 16 bits for groups of 8 weights.
+| Model      | AQLM scheme | WikiText 2 PPL | Model size, Gb | Hub link                                                                 |
+|------------|-------------|----------------|----------------|--------------------------------------------------------------------------|
+| Llama-2-7b (this) | 1x16        | 5.68          | 2.4            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-7b-AQLM-PV-2Bit-1x16-hf) |
+| Llama-2-7b | 2x8         | 5.90          | 2.2            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-7b-AQLM-PV-2Bit-2x8-hf)  |
+| Llama-2-13b| 1x16        | 5.05           | 4.1            | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-13b-AQLM-PV-2Bit-1x16-hf)|
+| Llama-2-70b| 1x16        | 3.78           | 18.8           | [Link](https://huggingface.co/ISTA-DASLab/Llama-2-70b-AQLM-PV-2Bit-1x16-hf)|
+The 1x16g16 (1-bit) models are on the way, as soon as we update the inference lib with their respective kernels.
+To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the [official GitHub repo](https://github.com/Vahe1994/AQLM).
+The original code for PV-Tuning can be found in the [AQLM@pv-tuning](https://github.com/Vahe1994/AQLM/tree/pv-tuning) branch.