Instructions to use inference-optimization/Qwen3-0.9B-A0.8B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Qwen3-0.9B-A0.8B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/Qwen3-0.9B-A0.8B-small")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-0.9B-A0.8B-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3-0.9B-A0.8B-small", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Qwen3-0.9B-A0.8B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Qwen3-0.9B-A0.8B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-0.9B-A0.8B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Qwen3-0.9B-A0.8B-small

SGLang

How to use inference-optimization/Qwen3-0.9B-A0.8B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Qwen3-0.9B-A0.8B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-0.9B-A0.8B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Qwen3-0.9B-A0.8B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-0.9B-A0.8B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Qwen3-0.9B-A0.8B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Qwen3-0.9B-A0.8B-small
```

Qwen3-0.9B-A0.8B-small

This is a small version of Qwen/Qwen3-30B-A3B created for testing and development. It keeps every hyperparameter of the base model identical (hidden sizes, head counts, MoE topology, vocabulary, untied embeddings) and only reduces the number of decoder layers and the number of experts per MoE layer.

Model Details

Base Model: Qwen/Qwen3-30B-A3B
Architecture: Qwen3MoeForCausalLM (qwen3_moe)
Total Parameters: 0.89B
Activated Parameters: 0.85B (topk 8 of 10 routed experts per layer)

Configuration Changes

Only the allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
`num_hidden_layers`	48	4
`num_local_experts`	128	10

All layers are MoE (decoder_sparse_step 1, mlp_only_layers []) with full attention, as in the base. 10 experts = topk (8) + 2 so that routing is actually exercised (8 = topk would make every expert always active). num_experts_per_tok (topk 8) is unchanged. All other values are kept from the base, including hidden_size 2048, moe_intermediate_size 768, 32 attention heads / 4 KV heads (head_dim 128, q/k norms), vocab_size 151936, untied embeddings, rope_theta 1e6.

Checkpoint Structure

Single-file bf16 model.safetensors (159 tensors) using the original base checkpoint layout: per-expert tensors model.layers.N.mlp.experts.{i}.{gate,up,down}_proj.weight (as in the base's sharded checkpoint), plus standard attention/norm/embedding/lm_head names. Tensor-name patterns match the base model.safetensors.index.json exactly, modulo layer and expert counts (validated programmatically: 15/15 normalized patterns, 0 missing, 0 extra; every small layer has the exact base per-layer key pattern set). The checkpoint loads through AutoModelForCausalLM.from_pretrained with the same weight-conversion path as the base repo, and all conversions round-trip bit-exactly.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3-0.9B-A0.8B-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-0.9B-A0.8B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

Loaded the base AutoConfig and applied only the allowed changes (4 layers, 10 experts).
Built the model with skip_weights_download, re-initialized weights (init_weights() + NaN fixup), cast to bf16, and saved.
Converted the saved checkpoint from transformers stacked-expert layout (mlp.experts.gate_up_proj / down_proj) to the base per-expert layout (reversing the qwen2_moe weight-conversion mapping).
Fine-tuned on the skill's copypasta dataset in bf16 until target perplexity was reached (lr 5e-5, ~350 steps, early-stopped).
Re-converted the fine-tuned checkpoint to the base layout and validated structure + round-trip against the base index.

Validation

Perplexity on the skill's validation text: 1.001 (target <= 10)
Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.

Notes

The base model has untied embeddings (tie_word_embeddings: false, separate lm_head.weight in the checkpoint), so the embedding matrices alone account for 2 x 0.31B parameters. This makes the total 0.89B rather than the ~0.6B a tied-embedding variant would give.
The tokenizer's pad_token was set to <|im_end|> (matching eos_token) for training convenience.
No quantization_config in the base; nothing stripped. bf16 throughout.