Instructions to use AlexHung29629/MyModelCleanFreezeRouter with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/MyModelCleanFreezeRouter with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlexHung29629/MyModelCleanFreezeRouter")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AlexHung29629/MyModelCleanFreezeRouter")
model = AutoModelForMultimodalLM.from_pretrained("AlexHung29629/MyModelCleanFreezeRouter", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/MyModelCleanFreezeRouter with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/MyModelCleanFreezeRouter"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanFreezeRouter",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/MyModelCleanFreezeRouter

SGLang

How to use AlexHung29629/MyModelCleanFreezeRouter with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/MyModelCleanFreezeRouter" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanFreezeRouter",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/MyModelCleanFreezeRouter" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanFreezeRouter",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/MyModelCleanFreezeRouter with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/MyModelCleanFreezeRouter
```

MyModelCleanFreezeRouter / README.md

AlexHung29629

Upload folder using huggingface_hub

771d2e3 verified 4 days ago

preview code

Raw History Blame Contribute Delete

6.39 kB

---
library_name: transformers
base_model: google/gemma-4-26B-A4B-it
tags:
- generated_from_trainer
datasets:
- data/ultradata-sft-clean.jsonl
- data/ultradata-sft-agent-clean.jsonl
- data/tw-sft-clean.jsonl
model-index:
- name: outputs/gemma4-26b-fft-clean-freeze-router
  results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.19.0.dev0`
```yaml
# 4 節點 × 8 張 H100，全參數 FSDP2。clean 版資料，凍結 MoE router。
# 從 train-clean.yml 改來。由 train-clean-freeze-router.sbatch 送出。不要改 train-clean.yml。
# 不要在登入節點跑 preprocess 或 train。
# global batch = 32 * micro_batch_size * gradient_accumulation_steps。
# output 跟 train-clean.yml 分開，避免蓋到 gemma4-26b-fft-clean。
# 資料、長度、packing 跟 clean 65536 一樣，preprocess 可沿用那份 cache。
base_model: google/gemma-4-26B-A4B-it
# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
# 這次是文字 SFT，凍結 vision / audio tower。
freeze_mm_modules: true
# Axolotl 沒有「只凍結這些名字」的鍵。unfrozen_parameters 會先凍結全部，
# 再把對得上的參數設回 requires_grad。點會被當成字面量，萬用字元是 .+，不是 .*。
# 對照 google/gemma-4-26B-A4B-it 的 index：每個 decoder layer 的 router 只有
# proj.weight、scale、per_expert_scale，共 90 個。這三個不在下面的名單裡。
# expert、attention、MLP、layer_scalar、embed、norm、lm_head 照練。
# vision / audio / embed_vision 也不在名單裡，跟 freeze_mm_modules 一樣保持凍結。
unfrozen_parameters:
  - model.language_model.layers.\d+.(self_attn|mlp|experts|input_layernorm|post_attention_layernorm|pre_feedforward_layernorm|post_feedforward_layernorm|post_feedforward_layernorm_1|post_feedforward_layernorm_2|pre_feedforward_layernorm_2).+
  - model.language_model.layers.\d+.layer_scalar$
  - model.language_model.(embed_tokens|norm).+
  - lm_head.+
plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
# multimodal Gemma 4 的 liger 沒有 fused linear CE。
liger_fused_linear_cross_entropy: false
# EAFT 自帶 fused lm_head+loss，不能跟 CCE 同時開。
cut_cross_entropy: false
# 容器內建的 EAFT kernel。這四個鍵要一起開。
use_eaft: true
eaft_alpha: 0.333
eaft_k: 20
eaft_normalize: true
# 全參數。不要加 adapter、4bit、quantize_moe_experts。
load_in_8bit: false
load_in_4bit: false
datasets:
  - path: data/ultradata-sft-clean.jsonl
    type: input_output
  - path: data/ultradata-sft-agent-clean.jsonl
    type: input_output
  - path: data/tw-sft-clean.jsonl
    type: input_output
dataset_prepared_path: ./prepared/gemma4-26b-fft-clean-65536
output_dir: ./outputs/gemma4-26b-fft-clean-freeze-router
dataset_num_proc: 64
train_on_inputs: false
val_set_size: 0
seed: 42
# 跟 train-clean.yml 相同：65536、gradient_accumulation_steps 1。
# 超過這個長度的樣本，Axolotl preprocess 會丟掉。
sequence_len: 65536
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
use_tensorboard: true
# 32 張 H100。global batch = 32 * 1 * 1 * 65536 = 2,097,152 tokens。
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 2
optimizer: adamw_torch_fused
learning_rate: 5.0e-6
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
max_grad_norm: 1.0
bf16: auto
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
activation_offloading: false
# FSDP2 把參數、梯度、optimizer state 切到 32 張卡。
# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true
# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。
attn_implementation: flex_attention
flex_attn_kernel_options:
  num_stages: 1
  bwd_num_stages: 1
logging_steps: 1
saves_per_epoch: 1
save_total_limit: 2
# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
save_only_model: true
special_tokens:
  bos_token: "<bos>"
  eos_token: "<eos>"
  pad_token: "<pad>"
  unk_token: "<unk>"
# preprocess --debug 若把要學的 <turn|> 標成 -100，拿掉這段。
eot_tokens:
  - "<turn|>"
```

</details><br>

# outputs/gemma4-26b-fft-clean-freeze-router

This model is a fine-tuned version of [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl and the data/tw-sft-clean.jsonl datasets.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 32
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 21
- training_steps: 712

### Training results



### Framework versions

- Transformers 5.14.1
- Pytorch 2.11.0+cu130
- Datasets 4.8.4
- Tokenizers 0.22.2

	---
	library_name: transformers
	base_model: google/gemma-4-26B-A4B-it
	tags:
	- generated_from_trainer
	datasets:
	- data/ultradata-sft-clean.jsonl
	- data/ultradata-sft-agent-clean.jsonl
	- data/tw-sft-clean.jsonl
	model-index:
	- name: outputs/gemma4-26b-fft-clean-freeze-router
	results: []
	---

	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
	<details><summary>See axolotl config</summary>

	axolotl version: `0.19.0.dev0`
	```yaml
	# 4 節點 × 8 張 H100，全參數 FSDP2。clean 版資料，凍結 MoE router。
	# 從 train-clean.yml 改來。由 train-clean-freeze-router.sbatch 送出。不要改 train-clean.yml。
	# 不要在登入節點跑 preprocess 或 train。
	# global batch = 32 * micro_batch_size * gradient_accumulation_steps。
	# output 跟 train-clean.yml 分開，避免蓋到 gemma4-26b-fft-clean。
	# 資料、長度、packing 跟 clean 65536 一樣，preprocess 可沿用那份 cache。
	base_model: google/gemma-4-26B-A4B-it

	# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
	# 這次是文字 SFT，凍結 vision / audio tower。
	freeze_mm_modules: true

	# Axolotl 沒有「只凍結這些名字」的鍵。unfrozen_parameters 會先凍結全部，
	# 再把對得上的參數設回 requires_grad。點會被當成字面量，萬用字元是 .+，不是 .*。
	# 對照 google/gemma-4-26B-A4B-it 的 index：每個 decoder layer 的 router 只有
	# proj.weight、scale、per_expert_scale，共 90 個。這三個不在下面的名單裡。
	# expert、attention、MLP、layer_scalar、embed、norm、lm_head 照練。
	# vision / audio / embed_vision 也不在名單裡，跟 freeze_mm_modules 一樣保持凍結。
	unfrozen_parameters:
	- model.language_model.layers.\d+.(self_attn\|mlp\|experts\|input_layernorm\|post_attention_layernorm\|pre_feedforward_layernorm\|post_feedforward_layernorm\|post_feedforward_layernorm_1\|post_feedforward_layernorm_2\|pre_feedforward_layernorm_2).+
	- model.language_model.layers.\d+.layer_scalar$
	- model.language_model.(embed_tokens\|norm).+
	- lm_head.+

	plugins:
	- axolotl.integrations.liger.LigerPlugin

	liger_rope: true
	liger_rms_norm: true
	liger_glu_activation: true
	liger_layer_norm: true
	# multimodal Gemma 4 的 liger 沒有 fused linear CE。
	liger_fused_linear_cross_entropy: false
	# EAFT 自帶 fused lm_head+loss，不能跟 CCE 同時開。
	cut_cross_entropy: false

	# 容器內建的 EAFT kernel。這四個鍵要一起開。
	use_eaft: true
	eaft_alpha: 0.333
	eaft_k: 20
	eaft_normalize: true

	# 全參數。不要加 adapter、4bit、quantize_moe_experts。
	load_in_8bit: false
	load_in_4bit: false

	datasets:
	- path: data/ultradata-sft-clean.jsonl
	type: input_output
	- path: data/ultradata-sft-agent-clean.jsonl
	type: input_output
	- path: data/tw-sft-clean.jsonl
	type: input_output

	dataset_prepared_path: ./prepared/gemma4-26b-fft-clean-65536
	output_dir: ./outputs/gemma4-26b-fft-clean-freeze-router
	dataset_num_proc: 64
	train_on_inputs: false
	val_set_size: 0
	seed: 42

	# 跟 train-clean.yml 相同：65536、gradient_accumulation_steps 1。
	# 超過這個長度的樣本，Axolotl preprocess 會丟掉。
	sequence_len: 65536
	sample_packing: true
	eval_sample_packing: false
	pad_to_sequence_len: true

	wandb_project:
	wandb_entity:
	wandb_watch:
	wandb_name:
	wandb_log_model:

	use_tensorboard: true

	# 32 張 H100。global batch = 32 * 1 * 1 * 65536 = 2,097,152 tokens。
	gradient_accumulation_steps: 1
	micro_batch_size: 1
	num_epochs: 2

	optimizer: adamw_torch_fused
	learning_rate: 5.0e-6
	lr_scheduler: cosine
	warmup_ratio: 0.03
	weight_decay: 0.01
	max_grad_norm: 1.0

	bf16: auto
	tf32: true

	gradient_checkpointing: true
	gradient_checkpointing_kwargs:
	use_reentrant: false
	# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
	activation_offloading: false

	# FSDP2 把參數、梯度、optimizer state 切到 32 張卡。
	# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
	# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
	fsdp_version: 2
	fsdp_config:
	offload_params: false
	cpu_ram_efficient_loading: true
	auto_wrap_policy: TRANSFORMER_BASED_WRAP
	transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
	state_dict_type: FULL_STATE_DICT
	reshard_after_forward: true

	# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
	# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。
	attn_implementation: flex_attention
	flex_attn_kernel_options:
	num_stages: 1
	bwd_num_stages: 1

	logging_steps: 1
	saves_per_epoch: 1
	save_total_limit: 2
	# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
	save_only_model: true

	special_tokens:
	bos_token: "<bos>"
	eos_token: "<eos>"
	pad_token: "<pad>"
	unk_token: "<unk>"

	# preprocess --debug 若把要學的 <turn\|> 標成 -100，拿掉這段。
	eot_tokens:
	- "<turn\|>"

	```

	</details><br>

	# outputs/gemma4-26b-fft-clean-freeze-router

	This model is a fine-tuned version of [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl and the data/tw-sft-clean.jsonl datasets.

	## Model description

	More information needed

	## Intended uses & limitations

	More information needed

	## Training and evaluation data

	More information needed

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 5e-06
	- train_batch_size: 1
	- eval_batch_size: 1
	- seed: 42
	- distributed_type: multi-GPU
	- num_devices: 32
	- total_train_batch_size: 32
	- total_eval_batch_size: 32
	- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_steps: 21
	- training_steps: 712

	### Training results



	### Framework versions

	- Transformers 5.14.1
	- Pytorch 2.11.0+cu130
	- Datasets 4.8.4
	- Tokenizers 0.22.2