Instructions to use OptGear/Opt.Gear-270M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OptGear/Opt.Gear-270M with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OptGear/Opt.Gear-270M", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("OptGear/Opt.Gear-270M", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OptGear/Opt.Gear-270M with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OptGear/Opt.Gear-270M"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OptGear/Opt.Gear-270M",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OptGear/Opt.Gear-270M

SGLang

How to use OptGear/Opt.Gear-270M with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OptGear/Opt.Gear-270M" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OptGear/Opt.Gear-270M",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OptGear/Opt.Gear-270M" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OptGear/Opt.Gear-270M",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use OptGear/Opt.Gear-270M with Docker Model Runner:
```
docker model run hf.co/OptGear/Opt.Gear-270M
```

Opt.Gear-270M / README.md

Opt-AI

Update README.md

02d1007 verified 3 days ago

preview code

Raw

History Blame Contribute Delete

14.5 kB

	---
	license: cc-by-nc-sa-4.0
	library_name: transformers
	pipeline_tag: text-generation
	language:
	- en
	- ko
	base_model:
	- OptAI/Opt.Gear-270M
	---

	# Opt.Gear-270M

	<img width="1000px" src="./OptAIxHuggingFace02.png">

	[![OptAI](https://img.shields.io/badge/%F0%9F%8F%A0%20OptAI-1a1a2e)](https://opt-ai.kr)
	[![Tech Report](https://img.shields.io/badge/%F0%9F%93%84%20Tech%20Report-5b4bcf)](https://huggingface.co/OptGear)

	> [!Note]
	> This repository contains model weights and configuration files for the post-trained (instruction-tuned) model in the Hugging Face Transformers format.
	>
	> These artifacts are compatible with Hugging Face Transformers, llama.cpp, ExecuTorch, and vLLM. Executable binaries optimized for NPU inference (Qualcomm Hexagon NPU, Apple ANE) are also provided.
	>
	> Opt.Gear-270M targets lower-latency deployment on more constrained mobile and edge devices. For a stronger quality-efficiency trade-off at the same context length, see [Opt.Gear-1B](https://huggingface.co/OptGear/Opt.Gear-1B). <!-- TODO: 링크 확인 -->

	Opt.Gear is the first generation of OptAI Foundation Models, designed for efficient on-device deployment, real-time inference, and strong task capability. Rather than simply shrinking a server-scale LLM, Gear jointly optimizes downstream quality, device-side latency, memory efficiency, long-context capability, and practical deployability under hardware constraints such as limited SRAM, memory bandwidth, and power.

	Opt.Gear-270M shares the same hybrid design and 64K context support as Opt.Gear-1B, while targeting lower-latency deployment on smaller mobile and edge environments.

	## Opt.Gear Highlights

	- Data-efficient training: Trained on a curated 0.5T-token subset selected from a 2T-token candidate corpus, without knowledge distillation from a teacher model. Against the distilled Gemma3-270M (6T tokens), Opt.Gear-270M achieves competitive or superior scores with 12× fewer training tokens.

	- Hybrid architecture with ConvKV-Gated Mixer: A small set of global GQA layers handles long-range routing, while ConvKV-Gated Mixer blocks replace many local attention layers with softmax-free, matrix-multiplication-free local mixing. The persistent local state scales with the convolution kernel (L_conv = 3) instead of the sliding window (W = 512), substantially reducing live decoding state and memory bandwidth.

	- NPU-friendly by design: Dynamic QKV matrix multiplications and softmax normalization are replaced with static linear, convolution, and element-wise operations, making the model easy to map onto CPUs, GPUs, and NPUs.

	- Korean-English bilingual: Built on the KORMo tokenizer (125,184 vocab) with Korean-English pretraining, leading same-scale baselines on Korean benchmarks (KMMLU, KoBEST, HAERAE).

	- 64K context on device: Hybrid attention layout with separate global/local RoPE frequencies supports long-context modeling up to 65,536 tokens while keeping the local cache footprint small.

	For more details, please refer to our tech report and blog post. <!-- TODO: 링크 연결 -->

	## Model Overview

	- Type: Causal Language Model (hybrid attention + convolutional mixer)
	- Training Stage: Pre-training (0.5T tokens) → Long-context extension (4K → 32K → 64K) → Two-stage SFT (general instruction → reasoning-oriented)
	- Architecture
	- Number of Parameters: 270M
	- Hidden Dimension: 640
	- Number of Layers: 18
	- Hidden Layout: hybrid of Global GQA, Local (sliding-window) Attention, and ConvKV-Gated Mixer
	- Grouped-Query Attention:
	- Number of Attention Heads: 4 for Q and 1 for KV
	- Head Dimension: 256
	- Sliding-Window Size (local attention): 512
	- QK-Normalization: QK-LN
	- ConvKV-Gated Mixer:
	- Causal depthwise 1D convolution on key/value streams
	- Convolution Kernel Size: 3 (fixed-size persistent state, independent of context length)
	- Feed-Forward Network:
	- Type: GeGLU (gated dense MLP)
	- Intermediate Dimension: 2,048
	- Rotary Position Embedding: global theta 1,000,000 / local theta 10,000
	- Tokenizer: KORMo (byte-level BPE), vocabulary 125,184 (shared with Opt.Gear-1B)
	- Word Embedding: untied (separate input embedding and LM head)
	- Context Length: 65,536 natively

	> [!Important]
	> Base and Instruction models share the same tokenizer, but use different end-of-generation tokens: the Base model emits `<EOS>` while the Instruction model emits `<EOT>`. When fine-tuning, make sure training examples terminate with the same convention used by the target runtime — malformed termination wastes decoding steps and increases latency on device.

	## Benchmark Results

	All results are obtained using the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) and may differ from scores reported elsewhere. Perplexity-based evaluation is adopted for HellaSwag, PIQA, WinoGrande, MMLU, GPQA, ARC, KMMLU, and KoBEST.

	<div style="max-width:1000px;margin:0 auto;padding:16px 0">
	<table style="border-collapse:collapse;font-size:15px">
	<thead><tr>
	<th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #5b4bcf;color:#5b4bcf"></th>
	<th style="padding:10px 7px;text-align:center;font-weight:600;border-bottom:2px solid #5b4bcf;color:#5b4bcf">Opt.Gear-270M</th>
	<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">SmolLM2-135M</th>
	<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">Gemma3-270M</th>
	<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">LFM2.5-350M</th>
	<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #5b4bcf;color:#5b4bcf">Qwen3-0.6B</th>
	</tr></thead>
	<tbody>
	<tr>
	<td style="padding:7px;border-bottom:1px solid rgba(128,128,128,0.15)"># Trained Tokens</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">0.5T</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">2T</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">6T</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">28T</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">36T</td>
	</tr>
	<tr>
	<td style="padding:7px;border-bottom:1px solid rgba(128,128,128,0.15)">Distilled</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">✗</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">✗</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">✓</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">✓</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">✓</td>
	</tr>
	<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#5b4bcf;border-bottom:1px solid rgba(91,75,207,0.2);background:rgba(91,75,207,0.08)">English</td></tr>
	<tr>
	<td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">MMLU</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">25.9</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">25.3</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">26.5</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">41.0</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">47.3</td>
	</tr>
	<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#5b4bcf;border-bottom:1px solid rgba(91,75,207,0.2);background:rgba(91,75,207,0.08)">Korean</td></tr>
	<tr>
	<td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">KMMLU</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">30.0</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">29.1</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">28.0</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">31.4</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.0</td>
	</tr>
	<tr>
	<td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">KoBEST</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">51.9</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.7</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.0</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.1</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">54.4</td>
	</tr>
	<tr>
	<td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">CLIcK</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">27.2</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">22.5</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">27.3</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.0</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">36.7</td>
	</tr>
	<tr>
	<td style="padding:7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)">HAERAE</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">21.5</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">18.9</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">20.4</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.8</td>
	<td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">37.0</td>
	</tr>
	</tbody>
	</table>
	<p style="margin-top:12px;font-size:11px;opacity:0.7">
	* All results are obtained using LM Evaluation Harness and may differ from other reported scores.<br>
	* While baselines are trained on 2T–36T tokens and frequently leverage knowledge distillation, Opt.Gear-270M is trained on only 0.5T tokens without any distillation. Against the similarly-sized distilled Gemma3-270M, Opt.Gear-270M leads on reasoning (PIQA 67.3 vs 66.9, ARC-Easy 59.1 vs 56.1) and Korean benchmarks (KMMLU 30.0 vs 28.0, KoBEST 51.9 vs 50.0, HAERAE 21.5 vs 20.4).
	</p>
	</div>

	### On-Device Inference

	Opt.Gear-270M shares the same NPU-friendly hybrid architecture as Opt.Gear-1B, which achieves up to ×4.9 faster prefill and decoding on NPUs compared to models of similar scale (measured with Qualcomm AI Runtime and CoreML — see the [Opt.Gear-1B card](https://huggingface.co/OptGear/Opt.Gear-1B) and the tech report for detailed device benchmarks). <!-- TODO: 270M 자체 디바이스 측정치가 있으면 표로 교체 -->

	## Quickstart

	### Hugging Face Transformers

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model_id = "OptGear/Opt.Gear-270M"

	tokenizer = AutoTokenizer.from_pretrained(model_id)
	model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

	messages = [
	{"role": "user", "content": "온디바이스 AI가 왜 중요한지 설명해줘."},
	]
	inputs = tokenizer.apply_chat_template(
	messages, add_generation_prompt=True, return_tensors="pt"
	).to(model.device)

	outputs = model.generate(inputs, max_new_tokens=512)
	print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
	```
	<!--
	### llama.cpp

	```shell
	# TODO: GGUF 파일명 확정 후 교체
	llama-cli -m opt-gear-270m-q4.gguf -p "안녕하세요" -n 128 -->
	<!-- ``` -->

	### On-Device Deployment

	Executable binaries optimized for NPU inference are provided for:

	- Qualcomm Hexagon NPU — via Qualcomm AI Engine Direct (QAIRT); recommended for Snapdragon devices
	- Apple ANE — via CoreML Runtime
	- ExecuTorch — for mobile/edge PyTorch deployment

	<!-- TODO: 디바이스별 배포 가이드 링크 추가 -->

	## Best Practices

	1. Termination tokens: The Instruction model terminates generation with `<EOT>`, while the Base model uses `<EOS>`. When fine-tuning either variant, ensure training examples terminate with the matching token.

	2. Runtime selection: Snapdragon NPU → QAIRT; Apple devices → CoreML; CPU/GPU → llama.cpp. Avoid the llama.cpp NPU backend for models with sliding-window attention.

	3. Languages: The model is trained primarily on English (~92%) and Korean (~6%) with mathematical text (~2%). It is best suited for Korean-English bilingual understanding, summarization, rewriting, and instruction following on latency-sensitive devices.

	4. Not intended for code generation: The pre-training corpus deliberately excludes code data — models at the 270M–1B scale are not typically used for code generation tasks.

	5. Long context: The model natively supports 65,536 tokens with a fixed-size local cache, making long-context decoding memory-efficient even at this scale.

	## Limitations

	The constrained 0.5T-token training budget limits capabilities in complex reasoning and mathematical tasks that typically benefit from larger training corpora. At the 270M scale, knowledge-heavy benchmarks (e.g., MMLU) remain challenging; the model is best used for well-scoped generation and understanding tasks rather than open-domain knowledge QA. See the tech report for a detailed discussion.

	## Citation

	If you find our work helpful, feel free to give us a cite.

	```bibtex
	@misc{optgear2026,
	title = {{Opt-Gear} Technical Report},
	author = {{Opt.Gear Team}},
	year = {2026},
	url = {https://huggingface.co/OptGear}
	}
	```

	---

	Correspondence: [contact@opt-ai.kr](mailto:contact@opt-ai.kr) · Hugging Face: [huggingface.co/OptAI](https://huggingface.co/OptGear)