Instructions to use AlexHung29629/MyEBFTModelSFTCP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/MyEBFTModelSFTCP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlexHung29629/MyEBFTModelSFTCP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AlexHung29629/MyEBFTModelSFTCP")
model = AutoModelForMultimodalLM.from_pretrained("AlexHung29629/MyEBFTModelSFTCP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/MyEBFTModelSFTCP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/MyEBFTModelSFTCP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyEBFTModelSFTCP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/MyEBFTModelSFTCP

SGLang

How to use AlexHung29629/MyEBFTModelSFTCP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/MyEBFTModelSFTCP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyEBFTModelSFTCP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/MyEBFTModelSFTCP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyEBFTModelSFTCP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/MyEBFTModelSFTCP with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/MyEBFTModelSFTCP
```

MyEBFTModelSFTCP / README.md

AlexHung29629

Upload folder using huggingface_hub

aaba2d3 verified 1 day ago

preview code

Raw History Blame Contribute Delete

7.22 kB

---
library_name: transformers
base_model: AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged
tags:
- generated_from_trainer
datasets:
- data/ultradata-sft-clean.jsonl
- data/ultradata-sft-agent-clean.jsonl
- data/tw-sft-clean.jsonl
- data/search-sft-clean.jsonl
model-index:
- name: outputs/gemma4-26b-ebft-mix-noeaft-cp
  results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.21.1`
```yaml
# 4 節點 × 8 張 H100，全參數 FSDP2。序列 131072，Ulysses CP 2。
# 資料是 train-clean-noeaft-cp.yml 的三份 clean jsonl，加上 train-search-noeaft-cp.yml 的 search。
# 超參、CP、CCE、FSDP 跟那兩份相同。不要改那兩份，也不要覆寫它們的 output_dir。
# 權重起點是 AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged。
# snapshot 已在 HF_HOME。worker 設 HF_HUB_OFFLINE=1，不要改去 google/gemma-4-26B-A4B-it。
# 這是新的 run：新的 output_dir、新的 prepared cache、新的 cosine。不能從舊 optimizer 續訓。
# 由 prep-mix-ebft-noeaft-cp.sbatch 先 preprocess，再由 train-mix-ebft-noeaft-cp.sbatch 送出。
# 容器是 ../axolotl-cp.sqsh。
# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
# 不要在登入節點跑 preprocess 或 train。
# 同一個 CP group 的 2 張卡看同一條序列。
# global batch = (32 / context_parallel.size) * micro_batch_size * gradient_accumulation_steps。
base_model: AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged
# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
# 這次是文字 SFT，凍結 vision / audio tower。
freeze_mm_modules: true
plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
# multimodal Gemma 4 的 liger 沒有 fused linear CE。
liger_fused_linear_cross_entropy: false
# 純交叉熵。CCE 不物化整段 logits，並套用 final_logit_softcapping=30。
# CP 2 時每張卡看 65536 token。這段 bf16 logits 約 32GiB，放不進 H100。
# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
cut_cross_entropy: true
use_eaft: false
# 全參數。不要加 adapter、4bit、quantize_moe_experts。
load_in_8bit: false
load_in_4bit: false
datasets:
  - path: data/ultradata-sft-clean.jsonl
    type: input_output
  - path: data/ultradata-sft-agent-clean.jsonl
    type: input_output
  - path: data/tw-sft-clean.jsonl
    type: input_output
  - path: data/search-sft-clean.jsonl
    type: input_output
dataset_prepared_path: ./prepared/gemma4-26b-ebft-mix-131072
output_dir: ./outputs/gemma4-26b-ebft-mix-noeaft-cp
dataset_num_proc: 64
train_on_inputs: false
val_set_size: 0
seed: 42
# 128k。超過這個長度的樣本，Axolotl preprocess 會丟掉。
sequence_len: 131072
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
use_tensorboard: true
# 32 張 H100，CP 2。每個 step 16 條序列。
# global batch = 32 / 2 * 1 * 1 * 131072 = 2,097,152 tokens。
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 2
optimizer: adamw_torch_fused
learning_rate: 5.0e-6
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
max_grad_norm: 1.0
bf16: auto
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
activation_offloading: false
# FSDP2 把參數、梯度、optimizer state 切到 32 張卡（dp_shard × cp）。
# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
# 185902：Accelerate 只從環境變數組 mesh。context_parallel 會設 CP_SIZE=2，
# dp_shard 預設 1，total_size=2，對不上 32 個 process。
# 明寫 dp_shard_size，axolotl 才會設 PARALLELISM_CONFIG_DP_SHARD_SIZE。
# 32 卡 / CP 2 = 16。改節點數或 CP 時要一起改。
dp_shard_size: 16
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true
# Ulysses 把序列切成 2 段。auto 搭 flex 會被擋，backend 一定要明寫 ulysses。
# 不要加 gemma4_hybrid_attn_impl，它會繞過 CP wrapper。
# 185909 的 mesh 已是 (16, 2)。接著 CP 讀 num_key_value_heads，
# Gemma 4 sliding=8、full attention=2，Transformers 拒絕讀全域值。
# 185910：Ulysses create_block_mask 會實體化整段 Q×KV，131072 的 int64 是 128GiB。
# 這兩件事都由 cp_compat/sitecustomize.py 處理，train_worker 用 PYTHONPATH 掛上。
context_parallel:
  size: 2
  backend: ulysses
# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。Ulysses 的 flex 也吃這組。
attn_implementation: flex_attention
flex_attn_kernel_options:
  num_stages: 1
  bwd_num_stages: 1
logging_steps: 1
saves_per_epoch: 1
save_total_limit: 2
# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
save_only_model: true
special_tokens:
  bos_token: "<bos>"
  eos_token: "<eos>"
  pad_token: "<pad>"
  unk_token: "<unk>"
# preprocess --debug 若把要學的 <turn|> 標成 -100，拿掉這段。
eot_tokens:
  - "<turn|>"
```

</details><br>

# outputs/gemma4-26b-ebft-mix-noeaft-cp

This model is a fine-tuned version of [AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged](https://huggingface.co/AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged) on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl, the data/tw-sft-clean.jsonl and the data/search-sft-clean.jsonl datasets.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 32
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 33
- training_steps: 1122

### Training results



### Framework versions

- Transformers 5.20.1
- Pytorch 2.13.0+cu130
- Datasets 5.0.1
- Tokenizers 0.23.3

	---
	library_name: transformers
	base_model: AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged
	tags:
	- generated_from_trainer
	datasets:
	- data/ultradata-sft-clean.jsonl
	- data/ultradata-sft-agent-clean.jsonl
	- data/tw-sft-clean.jsonl
	- data/search-sft-clean.jsonl
	model-index:
	- name: outputs/gemma4-26b-ebft-mix-noeaft-cp
	results: []
	---

	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
	<details><summary>See axolotl config</summary>

	axolotl version: `0.21.1`
	```yaml
	# 4 節點 × 8 張 H100，全參數 FSDP2。序列 131072，Ulysses CP 2。
	# 資料是 train-clean-noeaft-cp.yml 的三份 clean jsonl，加上 train-search-noeaft-cp.yml 的 search。
	# 超參、CP、CCE、FSDP 跟那兩份相同。不要改那兩份，也不要覆寫它們的 output_dir。
	# 權重起點是 AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged。
	# snapshot 已在 HF_HOME。worker 設 HF_HUB_OFFLINE=1，不要改去 google/gemma-4-26B-A4B-it。
	# 這是新的 run：新的 output_dir、新的 prepared cache、新的 cosine。不能從舊 optimizer 續訓。
	# 由 prep-mix-ebft-noeaft-cp.sbatch 先 preprocess，再由 train-mix-ebft-noeaft-cp.sbatch 送出。
	# 容器是 ../axolotl-cp.sqsh。
	# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
	# 不要在登入節點跑 preprocess 或 train。
	# 同一個 CP group 的 2 張卡看同一條序列。
	# global batch = (32 / context_parallel.size) * micro_batch_size * gradient_accumulation_steps。
	base_model: AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged

	# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
	# 這次是文字 SFT，凍結 vision / audio tower。
	freeze_mm_modules: true

	plugins:
	- axolotl.integrations.liger.LigerPlugin
	- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

	liger_rope: true
	liger_rms_norm: true
	liger_glu_activation: true
	liger_layer_norm: true
	# multimodal Gemma 4 的 liger 沒有 fused linear CE。
	liger_fused_linear_cross_entropy: false
	# 純交叉熵。CCE 不物化整段 logits，並套用 final_logit_softcapping=30。
	# CP 2 時每張卡看 65536 token。這段 bf16 logits 約 32GiB，放不進 H100。
	# 不要開 use_eaft。183855 已確認 EAFT 不能跟 context parallel 同時開。
	cut_cross_entropy: true
	use_eaft: false

	# 全參數。不要加 adapter、4bit、quantize_moe_experts。
	load_in_8bit: false
	load_in_4bit: false

	datasets:
	- path: data/ultradata-sft-clean.jsonl
	type: input_output
	- path: data/ultradata-sft-agent-clean.jsonl
	type: input_output
	- path: data/tw-sft-clean.jsonl
	type: input_output
	- path: data/search-sft-clean.jsonl
	type: input_output

	dataset_prepared_path: ./prepared/gemma4-26b-ebft-mix-131072
	output_dir: ./outputs/gemma4-26b-ebft-mix-noeaft-cp
	dataset_num_proc: 64
	train_on_inputs: false
	val_set_size: 0
	seed: 42

	# 128k。超過這個長度的樣本，Axolotl preprocess 會丟掉。
	sequence_len: 131072
	sample_packing: true
	eval_sample_packing: false
	pad_to_sequence_len: true

	wandb_project:
	wandb_entity:
	wandb_watch:
	wandb_name:
	wandb_log_model:

	use_tensorboard: true

	# 32 張 H100，CP 2。每個 step 16 條序列。
	# global batch = 32 / 2 * 1 * 1 * 131072 = 2,097,152 tokens。
	gradient_accumulation_steps: 1
	micro_batch_size: 1
	num_epochs: 2

	optimizer: adamw_torch_fused
	learning_rate: 5.0e-6
	lr_scheduler: cosine
	warmup_ratio: 0.03
	weight_decay: 0.01
	max_grad_norm: 1.0

	bf16: auto
	tf32: true

	gradient_checkpointing: true
	gradient_checkpointing_kwargs:
	use_reentrant: false
	# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
	activation_offloading: false

	# FSDP2 把參數、梯度、optimizer state 切到 32 張卡（dp_shard × cp）。
	# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
	# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
	# 185902：Accelerate 只從環境變數組 mesh。context_parallel 會設 CP_SIZE=2，
	# dp_shard 預設 1，total_size=2，對不上 32 個 process。
	# 明寫 dp_shard_size，axolotl 才會設 PARALLELISM_CONFIG_DP_SHARD_SIZE。
	# 32 卡 / CP 2 = 16。改節點數或 CP 時要一起改。
	dp_shard_size: 16
	fsdp_version: 2
	fsdp_config:
	offload_params: false
	cpu_ram_efficient_loading: true
	auto_wrap_policy: TRANSFORMER_BASED_WRAP
	transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
	state_dict_type: FULL_STATE_DICT
	reshard_after_forward: true

	# Ulysses 把序列切成 2 段。auto 搭 flex 會被擋，backend 一定要明寫 ulysses。
	# 不要加 gemma4_hybrid_attn_impl，它會繞過 CP wrapper。
	# 185909 的 mesh 已是 (16, 2)。接著 CP 讀 num_key_value_heads，
	# Gemma 4 sliding=8、full attention=2，Transformers 拒絕讀全域值。
	# 185910：Ulysses create_block_mask 會實體化整段 Q×KV，131072 的 int64 是 128GiB。
	# 這兩件事都由 cp_compat/sitecustomize.py 處理，train_worker 用 PYTHONPATH 掛上。
	context_parallel:
	size: 2
	backend: ulysses

	# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
	# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。Ulysses 的 flex 也吃這組。
	attn_implementation: flex_attention
	flex_attn_kernel_options:
	num_stages: 1
	bwd_num_stages: 1

	logging_steps: 1
	saves_per_epoch: 1
	save_total_limit: 2
	# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
	save_only_model: true

	special_tokens:
	bos_token: "<bos>"
	eos_token: "<eos>"
	pad_token: "<pad>"
	unk_token: "<unk>"

	# preprocess --debug 若把要學的 <turn\|> 標成 -100，拿掉這段。
	eot_tokens:
	- "<turn\|>"

	```

	</details><br>

	# outputs/gemma4-26b-ebft-mix-noeaft-cp

	This model is a fine-tuned version of [AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged](https://huggingface.co/AlexHung29629/ebft_gemma4_26b_lora_lr5e6_merged) on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl, the data/tw-sft-clean.jsonl and the data/search-sft-clean.jsonl datasets.

	## Model description

	More information needed

	## Intended uses & limitations

	More information needed

	## Training and evaluation data

	More information needed

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 5e-06
	- train_batch_size: 1
	- eval_batch_size: 1
	- seed: 42
	- distributed_type: multi-GPU
	- num_devices: 32
	- total_train_batch_size: 32
	- total_eval_batch_size: 32
	- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_steps: 33
	- training_steps: 1122

	### Training results



	### Framework versions

	- Transformers 5.20.1
	- Pytorch 2.13.0+cu130
	- Datasets 5.0.1
	- Tokenizers 0.23.3