Instructions to use princeton-nlp/Sheared-LLaMA-1.3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use princeton-nlp/Sheared-LLaMA-1.3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="princeton-nlp/Sheared-LLaMA-1.3B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("princeton-nlp/Sheared-LLaMA-1.3B")
model = AutoModelForCausalLM.from_pretrained("princeton-nlp/Sheared-LLaMA-1.3B", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use princeton-nlp/Sheared-LLaMA-1.3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "princeton-nlp/Sheared-LLaMA-1.3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "princeton-nlp/Sheared-LLaMA-1.3B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/princeton-nlp/Sheared-LLaMA-1.3B

SGLang

How to use princeton-nlp/Sheared-LLaMA-1.3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "princeton-nlp/Sheared-LLaMA-1.3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "princeton-nlp/Sheared-LLaMA-1.3B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "princeton-nlp/Sheared-LLaMA-1.3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "princeton-nlp/Sheared-LLaMA-1.3B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use princeton-nlp/Sheared-LLaMA-1.3B with Docker Model Runner:
```
docker model run hf.co/princeton-nlp/Sheared-LLaMA-1.3B
```

Sheared-LLaMA-1.3B

Commit History

Update README.md

a4b7693
verified

princeton-nlp commited on Jan 23, 2024

Update README.md

dbdd43e

princeton-nlp commited on Dec 4, 2023

Update README.md

d0fc0b8

princeton-nlp commited on Nov 22, 2023

Update config.json

14922a5

princeton-nlp commited on Nov 20, 2023

Adding Evaluation Results (#6)

25deffa

princeton-nlp

leaderboard-pr-bot commited on Nov 20, 2023

Update README.md

940769b

princeton-nlp commited on Nov 1, 2023

Update README.md

96ca077

princeton-nlp commited on Nov 1, 2023

Update README.md

6f63d66

princeton-nlp commited on Oct 26, 2023

Update README.md

ea72ad6

princeton-nlp commited on Oct 11, 2023

Update README.md

8f5cd82

princeton-nlp commited on Oct 11, 2023

Update README.md

3bc5cf3

princeton-nlp commited on Oct 11, 2023

Update README.md

83be984

princeton-nlp commited on Oct 11, 2023

Update README.md

59aab8d

princeton-nlp commited on Oct 11, 2023

Update README.md

e19f71e

princeton-nlp commited on Oct 11, 2023

Update README.md

b1c3f74

princeton-nlp commited on Oct 10, 2023

add model

c3e41d2

xiamengzhou commited on Oct 10, 2023

initial commit

a868c60

princeton-nlp commited on Oct 10, 2023

Commit History

Update README.md a4b7693 verified

Update README.md dbdd43e

Update README.md d0fc0b8

Update config.json 14922a5

Adding Evaluation Results (#6) 25deffa

Update README.md 940769b

Update README.md 96ca077

Update README.md 6f63d66

Update README.md ea72ad6

Update README.md 8f5cd82

Update README.md 3bc5cf3

Update README.md 83be984

Update README.md 59aab8d

Update README.md e19f71e

Update README.md b1c3f74

add model c3e41d2

initial commit a868c60

Update README.md

a4b7693
verified

Update README.md

dbdd43e

Update README.md

d0fc0b8

Update config.json

14922a5

Adding Evaluation Results (#6)

25deffa

Update README.md

940769b

Update README.md

96ca077

Update README.md

6f63d66

Update README.md

ea72ad6

Update README.md

8f5cd82

Update README.md

3bc5cf3

Update README.md

83be984

Update README.md

59aab8d

Update README.md

e19f71e

Update README.md

b1c3f74

add model

c3e41d2

initial commit

a868c60