Instructions to use AlexHung29629/MyModelCleanCP2SFT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/MyModelCleanCP2SFT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlexHung29629/MyModelCleanCP2SFT")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AlexHung29629/MyModelCleanCP2SFT")
model = AutoModelForMultimodalLM.from_pretrained("AlexHung29629/MyModelCleanCP2SFT", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/MyModelCleanCP2SFT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/MyModelCleanCP2SFT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanCP2SFT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/MyModelCleanCP2SFT

SGLang

How to use AlexHung29629/MyModelCleanCP2SFT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/MyModelCleanCP2SFT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanCP2SFT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/MyModelCleanCP2SFT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanCP2SFT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/MyModelCleanCP2SFT with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/MyModelCleanCP2SFT
```

MyModelCleanCP2SFT / README.md

AlexHung29629

Upload folder using huggingface_hub

bb4fe9d verified 2 days ago

preview code

Raw History Blame Contribute Delete

6.93 kB

---
library_name: transformers
base_model: google/gemma-4-26B-A4B-it
tags:
- generated_from_trainer
datasets:
- data/ultradata-sft-clean.jsonl
- data/ultradata-sft-agent-clean.jsonl
- data/tw-sft-clean.jsonl
model-index:
- name: outputs/gemma4-26b-fft-clean-noeaft-cp
  results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.21.1`
```yaml
# 4 節點 × 8 張 H100，全參數 FSDP2。clean 版資料，純 SFT，序列 131072。
# 從 train-clean-noeaft.yml 改來，套 train-clean-cp.yml 的 Ulysses CP 2。
# 由 train-clean-noeaft-cp.sbatch 送出。容器是 ../axolotl-cp.sqsh。
# train-clean-noeaft.yml 仍是 65536、沒有 CP。train-clean-cp.yml 仍開 EAFT。兩份都不要改。
# job 183855 用 train-clean-cp.yml，axolotl-cp 拒絕 use_eaft 配 context_parallel_size > 1。
# 不要在登入節點跑 preprocess 或 train。
# 同一個 CP group 的 2 張卡看同一條序列。
# global batch = (32 / context_parallel.size) * micro_batch_size * gradient_accumulation_steps。
# prepared 跟 train-clean-cp.yml 共用 131072 cache。65536 的 cache 已經丟掉超過 65536 的樣本。
# output 跟 train-clean-noeaft.yml、train-clean-cp.yml 分開。
base_model: google/gemma-4-26B-A4B-it
# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
# 這次是文字 SFT，凍結 vision / audio tower。
freeze_mm_modules: true
plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
# multimodal Gemma 4 的 liger 沒有 fused linear CE。
liger_fused_linear_cross_entropy: false
# 純交叉熵。CCE 不物化整段 logits，並套用 final_logit_softcapping=30。
# CP 2 時每張卡看 65536 token。這段 bf16 logits 約 32GiB，放不進 H100。
# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
cut_cross_entropy: true
use_eaft: false
# 全參數。不要加 adapter、4bit、quantize_moe_experts。
load_in_8bit: false
load_in_4bit: false
datasets:
  - path: data/ultradata-sft-clean.jsonl
    type: input_output
  - path: data/ultradata-sft-agent-clean.jsonl
    type: input_output
  - path: data/tw-sft-clean.jsonl
    type: input_output
dataset_prepared_path: ./prepared/gemma4-26b-fft-clean-131072
output_dir: ./outputs/gemma4-26b-fft-clean-noeaft-cp
dataset_num_proc: 64
train_on_inputs: false
val_set_size: 0
seed: 42
# 128k。超過這個長度的樣本，Axolotl preprocess 會丟掉。
# clean agent 有 1445 筆超過 65536，其中 1379 筆落在 131072 以內，66 筆更長。
sequence_len: 131072
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
use_tensorboard: true
# 32 張 H100，CP 2。每個 step 16 條序列。
# global batch = 32 / 2 * 1 * 1 * 131072 = 2,097,152 tokens。
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 2
optimizer: adamw_torch_fused
learning_rate: 5.0e-6
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
max_grad_norm: 1.0
bf16: auto
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
activation_offloading: false
# FSDP2 把參數、梯度、optimizer state 切到 32 張卡（dp_shard × cp）。
# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
# 185902：Accelerate 只從環境變數組 mesh。context_parallel 會設 CP_SIZE=2，
# dp_shard 預設 1，total_size=2，對不上 32 個 process。
# 明寫 dp_shard_size，axolotl 才會設 PARALLELISM_CONFIG_DP_SHARD_SIZE。
# 32 卡 / CP 2 = 16。改節點數或 CP 時要一起改。
dp_shard_size: 16
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true
# Ulysses 把序列切成 2 段。auto 搭 flex 會被擋，backend 一定要明寫 ulysses。
# 不要加 gemma4_hybrid_attn_impl，它會繞過 CP wrapper。
# 185909 的 mesh 已是 (16, 2)。接著 CP 讀 num_key_value_heads，
# Gemma 4 sliding=8、full attention=2，Transformers 拒絕讀全域值。
# 185910：Ulysses create_block_mask 會實體化整段 Q×KV，131072 的 int64 是 128GiB。
# 這兩件事都由 cp_compat/sitecustomize.py 處理，train_worker 用 PYTHONPATH 掛上。
context_parallel:
  size: 2
  backend: ulysses
# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。Ulysses 的 flex 也吃這組。
attn_implementation: flex_attention
flex_attn_kernel_options:
  num_stages: 1
  bwd_num_stages: 1
logging_steps: 1
saves_per_epoch: 1
save_total_limit: 2
# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
save_only_model: true
special_tokens:
  bos_token: "<bos>"
  eos_token: "<eos>"
  pad_token: "<pad>"
  unk_token: "<unk>"
# preprocess --debug 若把要學的 <turn|> 標成 -100，拿掉這段。
eot_tokens:
  - "<turn|>"
```

</details><br>

# outputs/gemma4-26b-fft-clean-noeaft-cp

This model is a fine-tuned version of [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl and the data/tw-sft-clean.jsonl datasets.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 32
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 24
- training_steps: 824

### Training results



### Framework versions

- Transformers 5.20.1
- Pytorch 2.13.0+cu130
- Datasets 5.0.1
- Tokenizers 0.23.3

	---
	library_name: transformers
	base_model: google/gemma-4-26B-A4B-it
	tags:
	- generated_from_trainer
	datasets:
	- data/ultradata-sft-clean.jsonl
	- data/ultradata-sft-agent-clean.jsonl
	- data/tw-sft-clean.jsonl
	model-index:
	- name: outputs/gemma4-26b-fft-clean-noeaft-cp
	results: []
	---

	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
	<details><summary>See axolotl config</summary>

	axolotl version: `0.21.1`
	```yaml
	# 4 節點 × 8 張 H100，全參數 FSDP2。clean 版資料，純 SFT，序列 131072。
	# 從 train-clean-noeaft.yml 改來，套 train-clean-cp.yml 的 Ulysses CP 2。
	# 由 train-clean-noeaft-cp.sbatch 送出。容器是 ../axolotl-cp.sqsh。
	# train-clean-noeaft.yml 仍是 65536、沒有 CP。train-clean-cp.yml 仍開 EAFT。兩份都不要改。
	# job 183855 用 train-clean-cp.yml，axolotl-cp 拒絕 use_eaft 配 context_parallel_size > 1。
	# 不要在登入節點跑 preprocess 或 train。
	# 同一個 CP group 的 2 張卡看同一條序列。
	# global batch = (32 / context_parallel.size) * micro_batch_size * gradient_accumulation_steps。
	# prepared 跟 train-clean-cp.yml 共用 131072 cache。65536 的 cache 已經丟掉超過 65536 的樣本。
	# output 跟 train-clean-noeaft.yml、train-clean-cp.yml 分開。
	base_model: google/gemma-4-26B-A4B-it

	# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
	# 這次是文字 SFT，凍結 vision / audio tower。
	freeze_mm_modules: true

	plugins:
	- axolotl.integrations.liger.LigerPlugin
	- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

	liger_rope: true
	liger_rms_norm: true
	liger_glu_activation: true
	liger_layer_norm: true
	# multimodal Gemma 4 的 liger 沒有 fused linear CE。
	liger_fused_linear_cross_entropy: false
	# 純交叉熵。CCE 不物化整段 logits，並套用 final_logit_softcapping=30。
	# CP 2 時每張卡看 65536 token。這段 bf16 logits 約 32GiB，放不進 H100。
	# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
	cut_cross_entropy: true
	use_eaft: false

	# 全參數。不要加 adapter、4bit、quantize_moe_experts。
	load_in_8bit: false
	load_in_4bit: false

	datasets:
	- path: data/ultradata-sft-clean.jsonl
	type: input_output
	- path: data/ultradata-sft-agent-clean.jsonl
	type: input_output
	- path: data/tw-sft-clean.jsonl
	type: input_output

	dataset_prepared_path: ./prepared/gemma4-26b-fft-clean-131072
	output_dir: ./outputs/gemma4-26b-fft-clean-noeaft-cp
	dataset_num_proc: 64
	train_on_inputs: false
	val_set_size: 0
	seed: 42

	# 128k。超過這個長度的樣本，Axolotl preprocess 會丟掉。
	# clean agent 有 1445 筆超過 65536，其中 1379 筆落在 131072 以內，66 筆更長。
	sequence_len: 131072
	sample_packing: true
	eval_sample_packing: false
	pad_to_sequence_len: true

	wandb_project:
	wandb_entity:
	wandb_watch:
	wandb_name:
	wandb_log_model:

	use_tensorboard: true

	# 32 張 H100，CP 2。每個 step 16 條序列。
	# global batch = 32 / 2 * 1 * 1 * 131072 = 2,097,152 tokens。
	gradient_accumulation_steps: 1
	micro_batch_size: 1
	num_epochs: 2

	optimizer: adamw_torch_fused
	learning_rate: 5.0e-6
	lr_scheduler: cosine
	warmup_ratio: 0.03
	weight_decay: 0.01
	max_grad_norm: 1.0

	bf16: auto
	tf32: true

	gradient_checkpointing: true
	gradient_checkpointing_kwargs:
	use_reentrant: false
	# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
	activation_offloading: false

	# FSDP2 把參數、梯度、optimizer state 切到 32 張卡（dp_shard × cp）。
	# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
	# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
	# 185902：Accelerate 只從環境變數組 mesh。context_parallel 會設 CP_SIZE=2，
	# dp_shard 預設 1，total_size=2，對不上 32 個 process。
	# 明寫 dp_shard_size，axolotl 才會設 PARALLELISM_CONFIG_DP_SHARD_SIZE。
	# 32 卡 / CP 2 = 16。改節點數或 CP 時要一起改。
	dp_shard_size: 16
	fsdp_version: 2
	fsdp_config:
	offload_params: false
	cpu_ram_efficient_loading: true
	auto_wrap_policy: TRANSFORMER_BASED_WRAP
	transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
	state_dict_type: FULL_STATE_DICT
	reshard_after_forward: true

	# Ulysses 把序列切成 2 段。auto 搭 flex 會被擋，backend 一定要明寫 ulysses。
	# 不要加 gemma4_hybrid_attn_impl，它會繞過 CP wrapper。
	# 185909 的 mesh 已是 (16, 2)。接著 CP 讀 num_key_value_heads，
	# Gemma 4 sliding=8、full attention=2，Transformers 拒絕讀全域值。
	# 185910：Ulysses create_block_mask 會實體化整段 Q×KV，131072 的 int64 是 128GiB。
	# 這兩件事都由 cp_compat/sitecustomize.py 處理，train_worker 用 PYTHONPATH 掛上。
	context_parallel:
	size: 2
	backend: ulysses

	# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
	# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。Ulysses 的 flex 也吃這組。
	attn_implementation: flex_attention
	flex_attn_kernel_options:
	num_stages: 1
	bwd_num_stages: 1

	logging_steps: 1
	saves_per_epoch: 1
	save_total_limit: 2
	# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
	save_only_model: true

	special_tokens:
	bos_token: "<bos>"
	eos_token: "<eos>"
	pad_token: "<pad>"
	unk_token: "<unk>"

	# preprocess --debug 若把要學的 <turn\|> 標成 -100，拿掉這段。
	eot_tokens:
	- "<turn\|>"

	```

	</details><br>

	# outputs/gemma4-26b-fft-clean-noeaft-cp

	This model is a fine-tuned version of [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl and the data/tw-sft-clean.jsonl datasets.

	## Model description

	More information needed

	## Intended uses & limitations

	More information needed

	## Training and evaluation data

	More information needed

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 5e-06
	- train_batch_size: 1
	- eval_batch_size: 1
	- seed: 42
	- distributed_type: multi-GPU
	- num_devices: 32
	- total_train_batch_size: 32
	- total_eval_batch_size: 32
	- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_steps: 24
	- training_steps: 824

	### Training results



	### Framework versions

	- Transformers 5.20.1
	- Pytorch 2.13.0+cu130
	- Datasets 5.0.1
	- Tokenizers 0.23.3