Hugging Face's logo

Nafnlaus
/
Wide-Sheared-LLaMA-796M

GGUF
Model card Files
xet
 Community
How to use from
llama.cpp
Install from brew 
brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
# Run inference directly in the terminal:
llama-cli -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
Install from WinGet (Windows) 
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
# Run inference directly in the terminal:
llama-cli -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
Use pre-built binary 
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
# Run inference directly in the terminal:
./llama-cli -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
Build from source code 
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Nafnlaus/Wide-Sheared-LLaMA-796M:
Use Docker 
docker model run hf.co/Nafnlaus/Wide-Sheared-LLaMA-796M:
Quick Links

This is a GGUF conversion of https://huggingface.co/minghaoyan/Wide-Sheared-LLaMA-796M, based on the paper "Decoding Speculative Decoding" by Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman.

https://arxiv.org/pdf/2402.01528

For those not familiar with speculative decoding, it is a technique to accelerate inference of larger models based on the premise that some tokens are much easier to predict than others. A large model is paired with a smaller draft model. The draft model is used to rapidly generate a possible predictive token sequence, which the large model then simultaneously verifies. Wherever the drafted token sequence would differ from what the large model would have generated, the large model's token is used instead (the large model will always correct or add one token), and the small model then drafts new tokens from that point forward, with the process repeating. As a result, the same sequence is generated, but at a significantly accelerated rate.

The wide sheared LLaMA models by minghaoyan are optimized for use as speculative decoding draft models. To use these with llama.cpp, use the "-md " option, and consider tuning the --draft parameter.

Downloads last month
141
GGUF
Model size
1B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for Nafnlaus/Wide-Sheared-LLaMA-796M