Text Generation
Transformers
Safetensors
English
gpt_oss
Mixture of Experts
mixture-of-experts
causal-lm
tinystories
tiny-model
validation
debug-model
mxfp4
e2m1
e8m0
Instructions to use shibatch/tinygptossmoe3m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shibatch/tinygptossmoe3m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="shibatch/tinygptossmoe3m")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("shibatch/tinygptossmoe3m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use shibatch/tinygptossmoe3m with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "shibatch/tinygptossmoe3m" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygptossmoe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/shibatch/tinygptossmoe3m
- SGLang
How to use shibatch/tinygptossmoe3m with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "shibatch/tinygptossmoe3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygptossmoe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "shibatch/tinygptossmoe3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygptossmoe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use shibatch/tinygptossmoe3m with Docker Model Runner:
docker model run hf.co/shibatch/tinygptossmoe3m
| { | |
| "description": "Canonical low-nibble-first E2M1 and E8M0 packing vector", | |
| "quantizer": "triton_kernels.numerics_details.mxfp.downcast_to_mxfp_torch", | |
| "triton_revision": "9e1e203f64752cf99abf0e44286231c5d5df7e76", | |
| "rounding_mode": "DequantScaleRoundingMode.ROUND_DOWN", | |
| "block_size": 32, | |
| "input_float32": [ | |
| 0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, | |
| -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0, | |
| 0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, | |
| -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0 | |
| ], | |
| "packed_uint8_decimal": [ | |
| 16, 50, 84, 118, 152, 186, 220, 254, | |
| 16, 50, 84, 118, 152, 186, 220, 254 | |
| ], | |
| "packed_uint8_hex": [ | |
| "10", "32", "54", "76", "98", "ba", "dc", "fe", | |
| "10", "32", "54", "76", "98", "ba", "dc", "fe" | |
| ], | |
| "scale_uint8_decimal": [127], | |
| "scale_exponent_after_bias": [0], | |
| "nibble_rule": "input[2*i] is bits 0..3; input[2*i+1] is bits 4..7" | |
| } | |