Text Generation
Transformers
Safetensors
gemma3_text
gemma3
unsloth
gemma
google
conversational
text-generation-inference
Instructions to use unsloth/gemma-3-270m-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use unsloth/gemma-3-270m-it with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="unsloth/gemma-3-270m-it") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("unsloth/gemma-3-270m-it") model = AutoModelForCausalLM.from_pretrained("unsloth/gemma-3-270m-it", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use unsloth/gemma-3-270m-it with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "unsloth/gemma-3-270m-it" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/gemma-3-270m-it", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/unsloth/gemma-3-270m-it
- SGLang
How to use unsloth/gemma-3-270m-it with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "unsloth/gemma-3-270m-it" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/gemma-3-270m-it", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "unsloth/gemma-3-270m-it" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "unsloth/gemma-3-270m-it", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Unsloth Studio
How to use unsloth/gemma-3-270m-it with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for unsloth/gemma-3-270m-it to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for unsloth/gemma-3-270m-it to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for unsloth/gemma-3-270m-it to start chatting
Load model with FastModel
pip install unsloth from unsloth import FastModel model, tokenizer = FastModel.from_pretrained( model_name="unsloth/gemma-3-270m-it", max_seq_length=2048, ) - Docker Model Runner
How to use unsloth/gemma-3-270m-it with Docker Model Runner:
docker model run hf.co/unsloth/gemma-3-270m-it
Upload folder using huggingface_hub
Browse files- README.md +16 -26
- chat_template.jinja +2 -6
- config.json +1 -1
- generation_config.json +1 -1
- tokenizer_config.json +1 -1
README.md
CHANGED
|
@@ -1,21 +1,24 @@
|
|
| 1 |
---
|
| 2 |
-
base_model:
|
| 3 |
-
|
| 4 |
-
- en
|
| 5 |
-
library_name: transformers
|
| 6 |
license: gemma
|
| 7 |
tags:
|
| 8 |
-
- unsloth
|
| 9 |
- gemma3
|
|
|
|
| 10 |
- gemma
|
| 11 |
- google
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
<div>
|
| 14 |
-
|
| 15 |
-
<
|
| 16 |
-
</p>
|
| 17 |
-
<p style="margin-bottom: 0;">
|
| 18 |
-
<em><a href="https://docs.unsloth.ai/basics/tutorial-how-to-run-gemma-3-effectively">Read our Guide</a> to see how to Run Gemma 3 correctly.</em>
|
| 19 |
</p>
|
| 20 |
<div style="display: flex; gap: 5px; align-items: center; ">
|
| 21 |
<a href="https://github.com/unslothai/unsloth/">
|
|
@@ -24,25 +27,12 @@ tags:
|
|
| 24 |
<a href="https://discord.gg/unsloth">
|
| 25 |
<img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173">
|
| 26 |
</a>
|
| 27 |
-
<a href="https://docs.unsloth.ai/
|
| 28 |
<img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143">
|
| 29 |
</a>
|
| 30 |
</div>
|
| 31 |
-
<h1 style="margin-top: 0rem;">✨ Fine-tune Gemma 3 with Unsloth!</h1>
|
| 32 |
</div>
|
| 33 |
|
| 34 |
-
- Fine-tune Gemma 3 (270M) for free using our Google [Colab notebook here](https://docs.unsloth.ai/get-started/unsloth-notebooks)!
|
| 35 |
-
- Read our Blog about Gemma 3 support: [unsloth.ai/blog/gemma3](https://unsloth.ai/blog/gemma3)
|
| 36 |
-
- View the rest of our notebooks in our [docs here](https://docs.unsloth.ai/get-started/unsloth-notebooks).
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
| Unsloth supports | Free Notebooks | Performance | Memory use |
|
| 40 |
-
|-----------------|--------------------------------------------------------------------------------------------------------------------------|-------------|----------|
|
| 41 |
-
| **Gemma 3 (4B)** | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb) | 2x faster | 80% less |
|
| 42 |
-
| **Gemma-3n-E4B** | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) | 2x faster | 60% less |
|
| 43 |
-
| **Gemma-3n-E4B (Audio)** | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Audio.ipynb) | 2x faster | 60% less |
|
| 44 |
-
| **GRPO with Gemma 3 (1B)** | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(1B)-GRPO.ipynb) | 2x faster | 80% less |
|
| 45 |
-
| **Gemma 3 (4B) Vision** | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) | 2x faster | 60% less |
|
| 46 |
|
| 47 |
# Gemma 3 model card
|
| 48 |
|
|
@@ -83,7 +73,7 @@ for everyone.
|
|
| 83 |
- **Input:**
|
| 84 |
- Text string, such as a question, a prompt, or a document to be summarized
|
| 85 |
- Images, normalized to 896 x 896 resolution and encoded to 256 tokens
|
| 86 |
-
each
|
| 87 |
- Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and
|
| 88 |
32K tokens for the 1B and 270M sizes.
|
| 89 |
|
|
@@ -556,4 +546,4 @@ alternatives.
|
|
| 556 |
[jax]: https://github.com/jax-ml/jax
|
| 557 |
[ml-pathways]: https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
|
| 558 |
[sustainability]: https://sustainability.google/operating-sustainably/
|
| 559 |
-
[gemini-2-paper]: https://arxiv.org/abs/2312.11805
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model:
|
| 3 |
+
- gg-hf-gm/gemma-3-270m-it
|
|
|
|
|
|
|
| 4 |
license: gemma
|
| 5 |
tags:
|
|
|
|
| 6 |
- gemma3
|
| 7 |
+
- unsloth
|
| 8 |
- gemma
|
| 9 |
- google
|
| 10 |
+
pipeline_tag: text-generation
|
| 11 |
+
library_name: transformers
|
| 12 |
+
extra_gated_heading: Access Gemma on Hugging Face
|
| 13 |
+
extra_gated_prompt: >-
|
| 14 |
+
To access Gemma on Hugging Face, you’re required to review and agree to
|
| 15 |
+
Google’s usage license. To do this, please ensure you’re logged in to Hugging
|
| 16 |
+
Face and click below. Requests are processed immediately.
|
| 17 |
+
extra_gated_button_content: Acknowledge license
|
| 18 |
---
|
| 19 |
<div>
|
| 20 |
+
<p style="margin-top: 0;margin-bottom: 0;">
|
| 21 |
+
<em><a href="https://docs.unsloth.ai/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0</a> achieves superior accuracy & outperforms other leading quants.</em>
|
|
|
|
|
|
|
|
|
|
| 22 |
</p>
|
| 23 |
<div style="display: flex; gap: 5px; align-items: center; ">
|
| 24 |
<a href="https://github.com/unslothai/unsloth/">
|
|
|
|
| 27 |
<a href="https://discord.gg/unsloth">
|
| 28 |
<img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173">
|
| 29 |
</a>
|
| 30 |
+
<a href="https://docs.unsloth.ai/">
|
| 31 |
<img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143">
|
| 32 |
</a>
|
| 33 |
</div>
|
|
|
|
| 34 |
</div>
|
| 35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
# Gemma 3 model card
|
| 38 |
|
|
|
|
| 73 |
- **Input:**
|
| 74 |
- Text string, such as a question, a prompt, or a document to be summarized
|
| 75 |
- Images, normalized to 896 x 896 resolution and encoded to 256 tokens
|
| 76 |
+
each, for the 4B, 12B, and 27B sizes.
|
| 77 |
- Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and
|
| 78 |
32K tokens for the 1B and 270M sizes.
|
| 79 |
|
|
|
|
| 546 |
[jax]: https://github.com/jax-ml/jax
|
| 547 |
[ml-pathways]: https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
|
| 548 |
[sustainability]: https://sustainability.google/operating-sustainably/
|
| 549 |
+
[gemini-2-paper]: https://arxiv.org/abs/2312.11805
|
chat_template.jinja
CHANGED
|
@@ -1,5 +1,4 @@
|
|
| 1 |
{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}
|
| 2 |
-
{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}
|
| 3 |
{{ bos_token }}
|
| 4 |
{%- if messages[0]['role'] == 'system' -%}
|
| 5 |
{%- if messages[0]['content'] is string -%}
|
|
@@ -37,10 +36,8 @@
|
|
| 37 |
{{ item['text'] | trim }}
|
| 38 |
{%- endif -%}
|
| 39 |
{%- endfor -%}
|
| 40 |
-
{%-
|
| 41 |
-
{
|
| 42 |
-
{{ raise_exception("Invalid content type") }}
|
| 43 |
-
{%- endif -%}
|
| 44 |
{%- endif -%}
|
| 45 |
{{ '<end_of_turn>
|
| 46 |
' }}
|
|
@@ -50,5 +47,4 @@
|
|
| 50 |
'}}
|
| 51 |
{%- endif -%}
|
| 52 |
|
| 53 |
-
{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}
|
| 54 |
{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}
|
|
|
|
| 1 |
{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}
|
|
|
|
| 2 |
{{ bos_token }}
|
| 3 |
{%- if messages[0]['role'] == 'system' -%}
|
| 4 |
{%- if messages[0]['content'] is string -%}
|
|
|
|
| 36 |
{{ item['text'] | trim }}
|
| 37 |
{%- endif -%}
|
| 38 |
{%- endfor -%}
|
| 39 |
+
{%- elif message['content'] is defined -%}
|
| 40 |
+
{{ raise_exception("Invalid content type") }}
|
|
|
|
|
|
|
| 41 |
{%- endif -%}
|
| 42 |
{{ '<end_of_turn>
|
| 43 |
' }}
|
|
|
|
| 47 |
'}}
|
| 48 |
{%- endif -%}
|
| 49 |
|
|
|
|
| 50 |
{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}
|
config.json
CHANGED
|
@@ -47,7 +47,7 @@
|
|
| 47 |
"rope_theta": 1000000.0,
|
| 48 |
"sliding_window": 512,
|
| 49 |
"torch_dtype": "bfloat16",
|
| 50 |
-
"transformers_version": "4.55.
|
| 51 |
"unsloth_fixed": true,
|
| 52 |
"use_bidirectional_attention": false,
|
| 53 |
"use_cache": true,
|
|
|
|
| 47 |
"rope_theta": 1000000.0,
|
| 48 |
"sliding_window": 512,
|
| 49 |
"torch_dtype": "bfloat16",
|
| 50 |
+
"transformers_version": "4.55.2",
|
| 51 |
"unsloth_fixed": true,
|
| 52 |
"use_bidirectional_attention": false,
|
| 53 |
"use_cache": true,
|
generation_config.json
CHANGED
|
@@ -10,5 +10,5 @@
|
|
| 10 |
"pad_token_id": 0,
|
| 11 |
"top_k": 64,
|
| 12 |
"top_p": 0.95,
|
| 13 |
-
"transformers_version": "4.55.
|
| 14 |
}
|
|
|
|
| 10 |
"pad_token_id": 0,
|
| 11 |
"top_k": 64,
|
| 12 |
"top_p": 0.95,
|
| 13 |
+
"transformers_version": "4.55.2"
|
| 14 |
}
|
tokenizer_config.json
CHANGED
|
@@ -51342,5 +51342,5 @@
|
|
| 51342 |
"tokenizer_class": "GemmaTokenizer",
|
| 51343 |
"unk_token": "<unk>",
|
| 51344 |
"use_default_system_prompt": false,
|
| 51345 |
-
"chat_template": "{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}\n{
|
| 51346 |
}
|
|
|
|
| 51342 |
"tokenizer_class": "GemmaTokenizer",
|
| 51343 |
"unk_token": "<unk>",
|
| 51344 |
"use_default_system_prompt": false,
|
| 51345 |
+
"chat_template": "{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}\n{{ bos_token }}\n{%- if messages[0]['role'] == 'system' -%}\n {%- if messages[0]['content'] is string -%}\n {%- set first_user_prefix = messages[0]['content'] + '\n\n' -%}\n {%- else -%}\n {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n\n' -%}\n {%- endif -%}\n {%- set loop_messages = messages[1:] -%}\n{%- else -%}\n {%- set first_user_prefix = \"\" -%}\n {%- set loop_messages = messages -%}\n{%- endif -%}\n{%- for message in loop_messages -%}\n {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}\n {{ raise_exception(\"Conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif -%}\n {%- if (message['role'] == 'assistant') -%}\n {%- set role = \"model\" -%}\n {%- else -%}\n {%- set role = message['role'] -%}\n {%- endif -%}\n {{ '<start_of_turn>' + role + '\n' + (first_user_prefix if loop.first else \"\") }}\n {%- if message['content'] is string -%}\n {{ message['content'] | trim }}\n {%- elif message['content'] is iterable -%}\n {%- for item in message['content'] -%}\n {%- if item['type'] == 'image' -%}\n {{ '<start_of_image>' }}\n {%- elif item['type'] == 'text' -%}\n {{ item['text'] | trim }}\n {%- endif -%}\n {%- endfor -%}\n {%- elif message['content'] is defined -%}\n {{ raise_exception(\"Invalid content type\") }}\n {%- endif -%}\n {{ '<end_of_turn>\n' }}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n {{'<start_of_turn>model\n'}}\n{%- endif -%}\n\n{# Copyright 2025-present Unsloth. Apache 2.0 License. Unsloth Chat template fixes #}"
|
| 51346 |
}
|