Instructions to use AlexHung29629/MyModelCleanCP2SFT_SEARCH with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/MyModelCleanCP2SFT_SEARCH with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlexHung29629/MyModelCleanCP2SFT_SEARCH")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AlexHung29629/MyModelCleanCP2SFT_SEARCH")
model = AutoModelForMultimodalLM.from_pretrained("AlexHung29629/MyModelCleanCP2SFT_SEARCH", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/MyModelCleanCP2SFT_SEARCH with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/MyModelCleanCP2SFT_SEARCH"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanCP2SFT_SEARCH",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/MyModelCleanCP2SFT_SEARCH

SGLang

How to use AlexHung29629/MyModelCleanCP2SFT_SEARCH with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/MyModelCleanCP2SFT_SEARCH" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanCP2SFT_SEARCH",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/MyModelCleanCP2SFT_SEARCH" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanCP2SFT_SEARCH",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/MyModelCleanCP2SFT_SEARCH with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/MyModelCleanCP2SFT_SEARCH
```

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

See axolotl config

axolotl version: 0.21.1

# 4 節點 × 8 張 H100，全參數 FSDP2。接續訓練，序列 131072，Ulysses CP 2。
# 從 train-clean-noeaft-cp.yml 改來。超參、CP、CCE、FSDP 維持原樣。
# 權重從 outputs/gemma4-26b-fft-clean-noeaft-cp 的最終模型開始。
# 那次 save_only_model，沒有 optimizer state，不能 resume。這裡是新的 run：
# 新的 output_dir、新的 cosine、warmup 重新算。不要把 output_dir 指回舊目錄。
# 資料只有 search-sft-clean。prompt/completion 已切好，轉成 input_output。
# 由 convert_search.sbatch 先轉換和 preprocess，再由 train-search-noeaft-cp.sbatch 送出。
# 容器是 ../axolotl-cp.sqsh。不要改 train-clean-noeaft-cp.yml。
# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
# 不要在登入節點跑 preprocess 或 train。
# 同一個 CP group 的 2 張卡看同一條序列。
# global batch = (32 / context_parallel.size) * micro_batch_size * gradient_accumulation_steps。
base_model: /mnt/shared/p01/alex/SFT/outputs/gemma4-26b-fft-clean-noeaft-cp

# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
# 這次是文字 SFT，凍結 vision / audio tower。
freeze_mm_modules: true

plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
# multimodal Gemma 4 的 liger 沒有 fused linear CE。
liger_fused_linear_cross_entropy: false
# 純交叉熵。CCE 不物化整段 logits，並套用 final_logit_softcapping=30。
# CP 2 時每張卡看 65536 token。這段 bf16 logits 約 32GiB，放不進 H100。
# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
cut_cross_entropy: true
use_eaft: false

# 全參數。不要加 adapter、4bit、quantize_moe_experts。
load_in_8bit: false
load_in_4bit: false

datasets:
  - path: data/search-sft-clean.jsonl
    type: input_output

dataset_prepared_path: ./prepared/gemma4-26b-fft-search-131072
output_dir: ./outputs/gemma4-26b-fft-clean-noeaft-cp-search
dataset_num_proc: 64
train_on_inputs: false
val_set_size: 0
seed: 42

# 128k。超過這個長度的樣本，Axolotl preprocess 會丟掉。
sequence_len: 131072
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

use_tensorboard: true

# 32 張 H100，CP 2。每個 step 16 條序列。
# global batch = 32 / 2 * 1 * 1 * 131072 = 2,097,152 tokens。
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 2

optimizer: adamw_torch_fused
learning_rate: 5.0e-6
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
max_grad_norm: 1.0

bf16: auto
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
activation_offloading: false

# FSDP2 把參數、梯度、optimizer state 切到 32 張卡（dp_shard × cp）。
# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
# 185902：Accelerate 只從環境變數組 mesh。context_parallel 會設 CP_SIZE=2，
# dp_shard 預設 1，total_size=2，對不上 32 個 process。
# 明寫 dp_shard_size，axolotl 才會設 PARALLELISM_CONFIG_DP_SHARD_SIZE。
# 32 卡 / CP 2 = 16。改節點數或 CP 時要一起改。
dp_shard_size: 16
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true

# Ulysses 把序列切成 2 段。auto 搭 flex 會被擋，backend 一定要明寫 ulysses。
# 不要加 gemma4_hybrid_attn_impl，它會繞過 CP wrapper。
# 185909 的 mesh 已是 (16, 2)。接著 CP 讀 num_key_value_heads，
# Gemma 4 sliding=8、full attention=2，Transformers 拒絕讀全域值。
# 185910：Ulysses create_block_mask 會實體化整段 Q×KV，131072 的 int64 是 128GiB。
# 這兩件事都由 cp_compat/sitecustomize.py 處理，train_worker 用 PYTHONPATH 掛上。
context_parallel:
  size: 2
  backend: ulysses

# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。Ulysses 的 flex 也吃這組。
attn_implementation: flex_attention
flex_attn_kernel_options:
  num_stages: 1
  bwd_num_stages: 1

logging_steps: 1
saves_per_epoch: 1
save_total_limit: 2
# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
save_only_model: true

special_tokens:
  bos_token: "<bos>"
  eos_token: "<eos>"
  pad_token: "<pad>"
  unk_token: "<unk>"

# preprocess --debug 若把要學的 <turn|> 標成 -100，拿掉這段。
eot_tokens:
  - "<turn|>"

outputs/gemma4-26b-fft-clean-noeaft-cp-search

This model was trained from scratch on the data/search-sft-clean.jsonl dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 32
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 8
training_steps: 298

Training results

Framework versions

Transformers 5.20.1
Pytorch 2.13.0+cu130
Datasets 5.0.1
Tokenizers 0.23.3

Downloads last month: 3

Safetensors

Model size

26B params

Tensor type

BF16