ToastyPigeon/qwen14-creative-epoch2

See axolotl config

axolotl version: 0.8.0.dev0

# === Start-up Commands ===
# curl -LsSf https://astral.sh/uv/install.sh | sh
# export PATH="$HOME/.local/bin:$PATH"
# git clone https://github.com/axolotl-ai-cloud/axolotl
# cd axolotl
# git checkout d8b4027200de0fe60f4ae0a71272c1a8cb2888f7
# uv venv
# source .venv/bin/activate
# uv pip install packaging ninja setuptools huggingface_hub[cli,hf_transfer]
# uv pip install "cut-cross-entropy[transformers] @ git+https://github.com/apple/ml-cross-entropy.git"
# uv pip install apollo-torch
# uv pip install --no-build-isolation -e .[flash-attn,deepspeed]
# uv pip install git+https://github.com/huggingface/transformers.git
# export HF_HUB_ENABLE_HF_TRANSFER=1
# huggingface-cli login --token $hf_key && wandb login $wandb_key
# axolotl preprocess qwen14-creative-v2-2.yml
# axolotl train qwen14-creative-v2-2.yml

# curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" && git clone https://github.com/axolotl-ai-cloud/axolotl && cd axolotl && uv venv && source .venv/bin/activate && uv pip install packaging ninja setuptools huggingface_hub[cli,hf_transfer] && uv pip install apollo-torch && uv pip install --no-build-isolation -e .[flash-attn,deepspeed] && uv pip install git+https://github.com/huggingface/transformers.git && export HF_HUB_ENABLE_HF_TRANSFER=1 && cd .. && huggingface-cli login --token $hf_key && wandb login $wandb_key

# === Model Configuration ===
base_model: ToastyPigeon/qwen2.5-14b-1m-unalign-v2  
load_in_8bit: false
load_in_4bit: false

# === Training Setup ===
num_epochs: 2
micro_batch_size: 2
gradient_accumulation_steps: 1
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

# === Evaluation ===
val_set_size: 50
evals_per_epoch: 10
eval_table_size:
eval_max_new_tokens: 256
eval_sample_packing: true

# === Hyperparameter Configuration ===
optimizer: apollo_adamw_layerwise
# Apollo-mini configuration:
optim_args: "proj=random,rank=1,scale=128.0,scale_type=tensor,update_proj_gap=200"
# Regular Apollo configuration:
# optim_args: 
optim_target_modules: all_linear
learning_rate: 1e-5
lr_scheduler: cosine
weight_decay: 0.01
warmup_ratio: 0.05

# === Data Configuration ===
datasets:
  - path: Columbidae/merge-glue
    data_files: conversation-glue.json
    type: chat_template
    split: train[:50%]
    field_messages: conversations
    message_field_role: from
    message_field_content: value
  - path: Columbidae/merge-glue
    data_files: completion-glue.json
    type: completion
    split: train[:50%]
    field: text

dataset_prepared_path: last_run_prepared
chat_template: tokenizer_default
# Example custom template:
# chat_template: jinja
# chat_template_jinja: |
#   {{- bos_token }}{%- for message in messages %}
#   {%- if message['role'] == 'system' %}
#   {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}
#   {%- elif message['role'] == 'user' %}
#   {{- '[INST]' + message['content'] + '[/INST]' }}
#   {%- elif message['role'] == 'assistant' %}
#   {{- message['content'] + eos_token }}
#   {%- endif %}
#   {%- endfor %}

# === Plugins ===
plugins:
  - axolotl.integrations.liger.LigerPlugin
#  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

# === Hardware Optimization ===
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true
#unsloth_cross_entropy_loss: true
#cut_cross_entropy: true
# Only if using multiple GPUs:
#deepspeed: axolotl/deepspeed_configs/zero3_bf16.json

# === Wandb Tracking ===
wandb_project: Qwen-14b-Revisit
# wandb_entity: [WANDB_ENTITY]
# wandb_name: [WANDB_RUN_NAME]

# === MLflow Tracking ===
#mlflow_tracking_uri: https://public-tracking.mlflow-e00zzfjq11ky6jcgtv.backbone-#e00bgn6e63256prmhq.msp.eu-north1.nebius.cloud
#mlflow_experiment_name: [EXPERIMENT NAME]  # e.g. "ms-24b-rp-inkmixv2-apollo"
#hf_mlflow_log_artifacts: true

# === Checkpointing ===
saves_per_epoch: 2
save_total_limit: 2

# === Advanced Settings ===
output_dir: ./ckpts
bf16: auto
flash_attention: true
train_on_inputs: false
group_by_length: false
save_safetensors: true
logging_steps: 1
gc_steps: 10
seed: 69

ckpts

This model is a fine-tuned version of ToastyPigeon/qwen2.5-14b-1m-unalign-v2 on the Columbidae/merge-glue and the Columbidae/merge-glue datasets. It achieves the following results on the evaluation set:

• Loss: 2.1558

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 69
• optimizer: Use OptimizerNames.APOLLO_ADAMW_LAYERWISE with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=proj=random,rank=1,scale=128.0,scale_type=tensor,update_proj_gap=200
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 116
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.3282	0.0009	1	2.2377
2.1584	0.1	116	2.1839
2.2901	0.2	232	2.1747
2.0968	0.3	348	2.1847
1.9414	0.4	464	2.1670
2.0265	0.5	580	2.1619
2.1412	0.6	696	2.1608
1.7849	0.7	812	2.1596
1.8671	0.8	928	2.1594
2.0826	0.9	1044	2.1577
2.2851	1.0	1160	2.1572
2.121	1.1	1276	2.1565
2.6014	1.2	1392	2.1565
2.4435	1.3	1508	2.1565
2.3827	1.4	1624	2.1560
2.2584	1.5	1740	2.1558
2.1037	1.6	1856	2.1559
2.197	1.7	1972	2.1559
2.1096	1.8	2088	2.1559
2.0501	1.9	2204	2.1558
2.1046	2.0	2320	2.1558

Framework versions

• Transformers 4.50.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0