Text Generation
Transformers
Safetensors
English
llama
ternary1b
quantization
ternary
smollm2
bitnet
text-generation-inference
Instructions to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET") model = AutoModelForCausalLM.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET
- SGLang
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with Docker Model Runner:
docker model run hf.co/AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET
Download eval/bitnet_quick.json from AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET: direct link, hf CLI and curl.
- Browser
- Download file 631 Bytes
-
https://huggingface.co/AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET/resolve/main/eval/bitnet_quick.json
- Command line
-
hf download hf://AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET/eval/bitnet_quick.json
-
curl -L -o bitnet_quick.json https://huggingface.co/AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET/resolve/main/eval/bitnet_quick.json
631 Bytes
| { | |
| "tag": "bitnet", | |
| "mode": "quick", | |
| "wikitext_ppl": 33.405132697338566, | |
| "tasks": { | |
| "arc_easy": 0.455 | |
| }, | |
| "size": { | |
| "packed_bytes": 523689912, | |
| "packed_mb": 499.4296188354492, | |
| "bf16_bytes": 3422777848, | |
| "compression": 6.5358865419580585, | |
| "bits_per_quantized_weight": 1.6000033378601082, | |
| "quantized_params": 1610612736, | |
| "format": "ternary 1.58 bit" | |
| }, | |
| "samples": [], | |
| "summary": { | |
| "wikitext_ppl": 33.405132697338566, | |
| "task_mean": 0.455, | |
| "size_mb": 499.4296188354492, | |
| "bits": 1.6000033378601082, | |
| "compression": 6.5358865419580585 | |
| }, | |
| "minutes": 1.5600075046221415 | |
| } |