Text Generation
Transformers
Safetensors
English
gpt_oss
Mixture of Experts
mixture-of-experts
causal-lm
tinystories
tiny-model
validation
debug-model
mxfp4
e2m1
e8m0
Instructions to use shibatch/tinygptossmoe3m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shibatch/tinygptossmoe3m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="shibatch/tinygptossmoe3m")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("shibatch/tinygptossmoe3m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use shibatch/tinygptossmoe3m with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "shibatch/tinygptossmoe3m" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygptossmoe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/shibatch/tinygptossmoe3m
- SGLang
How to use shibatch/tinygptossmoe3m with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "shibatch/tinygptossmoe3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygptossmoe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "shibatch/tinygptossmoe3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygptossmoe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use shibatch/tinygptossmoe3m with Docker Model Runner:
docker model run hf.co/shibatch/tinygptossmoe3m
File size: 902 Bytes
2129f51 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | {
"description": "Canonical low-nibble-first E2M1 and E8M0 packing vector",
"quantizer": "triton_kernels.numerics_details.mxfp.downcast_to_mxfp_torch",
"triton_revision": "9e1e203f64752cf99abf0e44286231c5d5df7e76",
"rounding_mode": "DequantScaleRoundingMode.ROUND_DOWN",
"block_size": 32,
"input_float32": [
0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0,
-0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0,
0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0,
-0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0
],
"packed_uint8_decimal": [
16, 50, 84, 118, 152, 186, 220, 254,
16, 50, 84, 118, 152, 186, 220, 254
],
"packed_uint8_hex": [
"10", "32", "54", "76", "98", "ba", "dc", "fe",
"10", "32", "54", "76", "98", "ba", "dc", "fe"
],
"scale_uint8_decimal": [127],
"scale_exponent_after_bias": [0],
"nibble_rule": "input[2*i] is bits 0..3; input[2*i+1] is bits 4..7"
}
|