Text Generation
Transformers
Safetensors
English
gemma4
gemma4-text
gemma4-moe
Mixture of Experts
mixture-of-experts
causal-lm
tinystories
tiny-model
validation
debug-model
legacy-tokenizer
Instructions to use shibatch/tinygemma4moe3m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shibatch/tinygemma4moe3m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="shibatch/tinygemma4moe3m")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("shibatch/tinygemma4moe3m", dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use shibatch/tinygemma4moe3m with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "shibatch/tinygemma4moe3m" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygemma4moe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/shibatch/tinygemma4moe3m
- SGLang
How to use shibatch/tinygemma4moe3m with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "shibatch/tinygemma4moe3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygemma4moe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "shibatch/tinygemma4moe3m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shibatch/tinygemma4moe3m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use shibatch/tinygemma4moe3m with Docker Model Runner:
docker model run hf.co/shibatch/tinygemma4moe3m
| { | |
| "Once upon": "Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red apple on the ground. She picked it up and showed it to her mom.\n\n\"Mommy, look what I found!\" Lily said.\n\n\"That's a big apple, Lily. It's a special apple. It's very special,\" her mom replied.\n\n", | |
| "There was a little": "There was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red apple on the ground. She picked it up and showed it to her mom.\n\n\"Mommy, look at the apple!\" Lily said.\n\n\"That's a nice apple, Lily. It's very pretty,\" her mom replied.\n\nLily was happy to have a new apple and wanted to", | |
| "One day": "One day, a little girl named Lily went to the park with her mom. She saw a big slide and wanted to try it. But her mom said, \"No, Lily. You have to wait. It's not safe.\"\n\nLily was sad. She wanted to go on the slide. She asked her mom, \"Can I go on the slide?\" Her mom said, \"No, Lily. You have to wait until the slide is safe.\"\n\n" | |
| } |