Instructions to use lianghsun/Marble-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lianghsun/Marble-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="lianghsun/Marble-3B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("lianghsun/Marble-3B")
model = AutoModelForCausalLM.from_pretrained("lianghsun/Marble-3B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use lianghsun/Marble-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lianghsun/Marble-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lianghsun/Marble-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/lianghsun/Marble-3B

SGLang

How to use lianghsun/Marble-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lianghsun/Marble-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lianghsun/Marble-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lianghsun/Marble-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lianghsun/Marble-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use lianghsun/Marble-3B with Docker Model Runner:
```
docker model run hf.co/lianghsun/Marble-3B
```

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Model Card for Marble-3B

Marble-3B 是以 ibm-granite/granite-3.1-3b-a800m-base（IBM Granite 3.1 3B-A800M MoE）為基底，針對繁體中文與中華民國台灣語境完成持續預訓練（CPT）之 MoE 基底模型，作為 Marble-3B-Instruct 等下游模型的繁中底座。

⚠️ 規格重點：本模型為 3B Mixture-of-Experts（MoE）基底模型、純文本單模態、僅做 CPT、未做指令微調，需自行 SFT 後才有對話能力。

Model Details

IBM Granite 3.1 系列引入 MoE 架構（3B 總參數、800M 活化參數），在推論成本與能力之間提供有趣的折衷點。Marble-3B 把繁中與台灣語境語料注入 Granite 3.1 3B-A800M，使下游任務能在 MoE 架構下取得繁中底層能力，兼顧推論效率與多領域覆蓋。

核心特點 (Key Features)

MoE 架構繁中底座：少量活化參數（800M）卻有 3B 總參數的容量，部署效率較同等級稠密模型佳。
多領域適配：MoE 結構天然適合多領域知識分流，配合繁中 CPT 可作為法律、教育、生活等多領域微調的共同基底。
可下游微調：作為 Instruct、領域應用之 SFT／DPO 起點。

Model Description

Developed by: Liang Hsun Huang
Funded by: APMIC
Base model: ibm-granite/granite-3.1-3b-a800m-base
Model type: GraniteMoeForCausalLM (Transformers)
Language(s) (NLP): Traditional Chinese, English
License: MIT

Model Sources

Repository: lianghsun/Marble-3B

Citation

@misc{marble_3b,
  title        = {Marble-3B: A Traditional Chinese Continued-Pretrained Granite 3B-A800M MoE Model for Taiwan},
  author       = {Huang, Liang Hsun},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/lianghsun/Marble-3B}}
}