Instructions to use GreenBitAI/yi-6b-w4a16g32 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use GreenBitAI/yi-6b-w4a16g32 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="GreenBitAI/yi-6b-w4a16g32")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("GreenBitAI/yi-6b-w4a16g32")
model = AutoModelForCausalLM.from_pretrained("GreenBitAI/yi-6b-w4a16g32")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use GreenBitAI/yi-6b-w4a16g32 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "GreenBitAI/yi-6b-w4a16g32"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreenBitAI/yi-6b-w4a16g32",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/GreenBitAI/yi-6b-w4a16g32

SGLang

How to use GreenBitAI/yi-6b-w4a16g32 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "GreenBitAI/yi-6b-w4a16g32" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreenBitAI/yi-6b-w4a16g32",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "GreenBitAI/yi-6b-w4a16g32" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreenBitAI/yi-6b-w4a16g32",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use GreenBitAI/yi-6b-w4a16g32 with Docker Model Runner:
```
docker model run hf.co/GreenBitAI/yi-6b-w4a16g32
```

yi-6b-w4a16g32

Commit History

Delete special_tokens_map.json

787f7f6

NicoNico commited on Jan 8, 2024

Delete tokenization_yi.py

c798c15

NicoNico commited on Jan 8, 2024

Delete modeling_yi.py

63037df

NicoNico commited on Jan 8, 2024

Delete configuration_yi.py

bcbe6d6

NicoNico commited on Jan 8, 2024

Upload 4 files

f853e60

NicoNico commited on Jan 8, 2024

Update README.md

ba54290

NicoNico commited on Dec 25, 2023

Update README.md

78e3d8e

yanghaojin commited on Dec 20, 2023

Update README.md

615ae37

NicoNico commited on Dec 15, 2023

Update README.md

0c30ce4

NicoNico commited on Dec 14, 2023

Update README.md

5151017

NicoNico commited on Dec 14, 2023

Update README.md

032a9a8

NicoNico commited on Dec 1, 2023

Update README.md

23396d3

NicoNico commited on Nov 17, 2023

Update README.md

e6251a4

NicoNico commited on Nov 17, 2023

Update README.md

6fa29ed

NicoNico commited on Nov 17, 2023

Update README.md

218660c

NicoNico commited on Nov 17, 2023

update

b26a2b0

NicoNico6 commited on Nov 16, 2023

update Yi-6B w4a16g32

f4d8732

NicoNico6 commited on Nov 16, 2023

initial commit

e1f2d10

NicoNico commited on Nov 16, 2023

Commit History

Delete special_tokens_map.json 787f7f6

Delete tokenization_yi.py c798c15

Delete modeling_yi.py 63037df

Delete configuration_yi.py bcbe6d6

Upload 4 files f853e60

Update README.md ba54290

Update README.md 78e3d8e

Update README.md 615ae37

Update README.md 0c30ce4

Update README.md 5151017

Update README.md 032a9a8

Update README.md 23396d3

Update README.md e6251a4

Update README.md 6fa29ed

Update README.md 218660c

update b26a2b0

update Yi-6B w4a16g32 f4d8732

initial commit e1f2d10

Delete special_tokens_map.json

787f7f6

Delete tokenization_yi.py

c798c15

Delete modeling_yi.py

63037df

Delete configuration_yi.py

bcbe6d6

Upload 4 files

f853e60

Update README.md

ba54290

Update README.md

78e3d8e

Update README.md

615ae37

Update README.md

0c30ce4

Update README.md

5151017

Update README.md

032a9a8

Update README.md

23396d3

Update README.md

e6251a4

Update README.md

6fa29ed

Update README.md

218660c

update

b26a2b0

update Yi-6B w4a16g32

f4d8732

initial commit

e1f2d10