See axolotl config

axolotl version: 0.13.0.dev0

# !pip install transformers==4.55.4
# !pip install --no-deps trl==0.22.2
# !pip install --no-build-isolation mamba_ssm==2.2.5
# !pip install --no-build-isolation causal_conv1d==1.5.2
# === Model Configuration ===
base_model: LatitudeGames/Muse-12B
load_in_8bit: true
load_in_4bit: false

# === HF Configuration === 
hub_model_id: ToastyPigeon/muse-marvin-lora
hub_strategy: "every_save"
output_dir: ckpts

# === Training Setup ===
num_epochs: 1
micro_batch_size: 6
gradient_accumulation_steps: 1
sequence_len: 16384
#sequence_parallel_degree: 2
#heads_k_stride: 1
sample_packing: true
pad_to_sequence_len: true
#temperature: 0.7
#max_steps: 10
# === Evaluation ===
val_set_size: 0.025
evals_per_epoch: 10
#eval_steps: 20
#max_steps: 60
#eval_table_size:
eval_max_new_tokens: 128
#eval_sample_packing: true
#eval_strategy: "no"

# === LoRA Configuration ===
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 32
lora_dropout: 0.1
lora_target_linear: true
lora_target_modules:
lora_fan_in_fan_out:
lora_target_modules:
peft_use_rslora: false
#lora_modules_to_save:
#  - embed_tokens
#  - lm_head
#fix_untrained_tokens: true
#lora_mlp_kernel: true
#lora_qkv_kernel: true
#lora_o_kernel: true

# === Hyperparameter Configuration ===
#optimizer: apollo_adamw_layerwise
#warmup_steps: 0
warmup_ratio: 0.025
#optimizer: adamw_torch_fused
optimizer: paged_adamw_8bit
#optim_args:
#  enable_stochastic_rounding: true
#  enable_cautious: true
#  enable_8bit: true
# Apollo-mini configuration:
#optim_args: "proj=random,rank=128,scale=128.0,scale_type=tensor,update_proj_gap=100"
# Regular Apollo configuration:
# optim_args: 
#optim_target_modules: all_linear
learning_rate: 1e-5
lr_scheduler: cosine
#cosine_min_lr_ratio: 0.2
#lr_scheduler: cosine_with_min_lr
#lr_scheduler_kwargs:
#  cosine_min_lr: 1e-6
weight_decay: 0.01
max_grad_norm: 1.0
#warmup_steps: 0
#warmup_ratio: 0.025


# === Data Configuration ===
#
#chat_template: jinja
#chat_template: chatml
special_tokens:
#  eos_token: "<|im_end|>"
#  eos_token: "</s>"
#tokenizer_use_mistral_common: true
shuffle_merged_datasets: true
datasets:
  - path: grimulkan/LimaRP-augmented
    type: chat_template
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value
#  - path: allenai/tulu-3-sft-personas-instruction-following
#    type: chat_template
#    split: train[:10%]
#  - path: ToastyPigeon/mixed-medical-reasoning-formatted
#    type: chat_template
#    data_files: mixed-medical-thinking.json
#    split: train[:10%]
  - path: ToastyPigeon/steve-and-marvin
    type: completion
    data_files: marvin.json
  - path: ToastyPigeon/kimi-stories-completion
    type: completion
#  - path: ToastyPigeon/new-story-dataset
 #   type: customcompletion-regex
#    type: completion
#    data_files: new-story-dataset-v2.json
#  - path: allura-org/fujin-instruct-v2
#    type: customchatml-regex
#    type: chat_template
#    field_messages: conversations
#    message_property_mappings:
#      role: from
#      content: value
#  - path: ToastyPigeon/some-rp-extended
 #   type: customchatml-regex
#    type: chat_template
#    field_messages: conversations
#    message_property_mappings:
#      role: from
#      content: value
#    roles_to_train: ["user","assistant"]
#  - path: ToastyPigeon/gutenberg-sft
#    type: customchatml-regex
#    type: chat_template
#    field_messages: conversations
#    message_property_mappings:
#      role: from
#      content: value
#  - path: ToastyPigeon/SpringDragon
#    type: customcompletion-regex
#    type: completion
#    split: train
#  - path: ToastyPigeon/some-erotica
#    type: customcompletion-regex
#    type: completion
#    split: train[:10%]

dataset_prepared_path: last_run_prepared


# === Plugins ===
plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

# === Hardware Optimization ===
gradient_checkpointing: true
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
#liger_fused_linear_cross_entropy: true
cut_cross_entropy: true

#deepspeed: deepspeed_configs/zero3_bf16.json

# === FSDP Config === 
#fsdp:
#  - full_shard
#  - auto_wrap
#fsdp_config:
#  fsdp_limit_all_gathers: true
#  fsdp_sync_module_states: true
#  fsdp_offload_params: true
#  fsdp_activation_checkpointing: true
#  fsdp_use_orig_params: false
#  fsdp_cpu_ram_efficient_loading: true
#  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
#  fsdp_transformer_layer_cls_to_wrap: MistralDecoderLayer
#  fsdp_state_dict_type: FULL_STATE_DICT
#  fsdp_sharding_strategy: FULL_SHARD
#  fsdp_version: 2
# === Wandb Tracking ===
wandb_project: MuseMarvin
# wandb_entity: [WANDB_ENTITY]
# wandb_name: [WANDB_RUN_NAME]

# === Checkpointing ===
#save_steps: 10
saves_per_epoch: 10
save_total_limit: 1

# === Advanced Settings ===
bf16: auto
flash_attention: true
train_on_inputs: false
group_by_length: false
save_safetensors: true
logging_steps: 1
gc_steps: 10
seed: 69

muse-marvin-lora

This model is a fine-tuned version of LatitudeGames/Muse-12B on the grimulkan/LimaRP-augmented, the ToastyPigeon/steve-and-marvin and the ToastyPigeon/kimi-stories-completion datasets. It achieves the following results on the evaluation set:

• Loss: 2.3857
• Memory/max Active (gib): 31.3
• Memory/max Allocated (gib): 31.3
• Memory/device Reserved (gib): 32.18

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 6
• eval_batch_size: 6
• seed: 69
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 7
• training_steps: 309

Training results

Training Loss	Epoch	Step	Validation Loss	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	2.4839	31.29	31.29	32.17
2.5492	0.1	31	2.4235	31.31	31.31	32.18
2.3906	0.2	62	2.4048	31.32	31.32	32.18
2.2984	0.3	93	2.3961	31.31	31.31	32.18
2.4423	0.4	124	2.3916	31.31	31.31	32.18
2.4106	0.5	155	2.3889	31.3	31.3	32.18
2.526	0.6	186	2.3875	31.3	31.3	32.18
2.3574	0.7	217	2.3863	31.3	31.3	32.18
2.4005	0.8	248	2.3858	31.3	31.3	32.18
2.4227	0.9	279	2.3857	31.3	31.3	32.18

Framework versions

• PEFT 0.17.1
• Transformers 4.56.1
• Pytorch 2.8.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.1