See axolotl config

axolotl version: 0.13.0.dev0

# 1. Base Model & Tokenizer
base_model: google/gemma-2-2b-it
model_type: AutoModelForCausalLM # Corrected from 'type_of_model' for axolotl
tokenizer_type: AutoTokenizer
hub_model_id: AiAF/rp-2b # New model ID for this finetune
hub_strategy: checkpoint

# 2. LoRA / QLoRA Configuration
load_in_4bit: true
adapter: qlora
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_linear: true

# 3. Dataset Configuration (TRAIN = streamed)
streaming: true
streaming_multipack_buffer_size: 10000
sample_packing: true
datasets:
  - path: AiAF/conversations
    data_files: conversations_V3.jsonl
   # revision:
    type: chat_template
    split: train
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value
    chat_template: jinja
    chat_template_jinja: |
      {{ bos_token }}
      {% for m in messages %}
        {% set role = 'model' if m['role']=='assistant' else 'user' %}
        {{ '<start_of_turn>' + role + '\n' + m['content'] | trim + '<end_of_turn>\n' }}
      {% endfor %}
      {% if add_generation_prompt %}
      {{ '<start_of_turn>model\n' }}
      {% endif %}

#    chat_template_jinja: |
#      {{ bos_token }}
#      {% set last = None %}
#      {% for m in messages %}
#        {% set raw_role = 'model' if m['role']=='assistant' else m['role'] %}
#        {% set role = 'user' if raw_role=='system' else raw_role %}
#        {% if role == last and role == 'user' %}
#          {{ m['content'] | trim }}
#        {% else %}
#          {{ '<start_of_turn>' + role + '\n' + m['content'] | trim + '<end_of_turn>\n' }}
#        {% endif %}
#        {% set last = role %}
#      {% endfor %}
#      {% if add_generation_prompt %}
#      {{ '<start_of_turn>model\n' }}
#      {% endif %}
    roles_to_train: ["assistant"]
    train_on_eos: "turn"
# Use a fixed (non-streamed) eval file with the same schema/Jinja
test_datasets:
  - path: .
    name: json
    type: chat_template
    data_files: eval-datasets/shuf-1000_conversations_V2.jsonl        # small, representative eval slice
    split: train
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value
    chat_template: jinja
    chat_template_jinja: |
      {{ bos_token }}
      {% for m in messages %}
        {% set role = 'model' if m['role']=='assistant' else 'user' %}
        {{ '<start_of_turn>' + role + '\n' + m['content'] | trim + '<end_of_turn>\n' }}
      {% endfor %}
      {% if add_generation_prompt %}
      {{ '<start_of_turn>model\n' }}
      {% endif %}
#    chat_template_jinja: |
#      {{ bos_token }}
#      {% set last = None %}
#      {% for m in messages %}
#        {% set raw_role = 'model' if m['role']=='assistant' else m['role'] %}
#        {% set role = 'user' if raw_role=='system' else raw_role %}
#        {% if role == last and role == 'user' %}
#          {{ m['content'] | trim }}
#        {% else %}
#          {{ '<start_of_turn>' + role + '\n' + m['content'] | trim + '<end_of_turn>\n' }}
#        {% endif %}
#        {% set last = role %}
#      {% endfor %}
#      {% if add_generation_prompt %}
#      {{ '<start_of_turn>model\n' }}
#      {% endif %}
    roles_to_train: ["assistant"]

# 4. Training Parameters
sequence_len: 2048
sample_packing: true
eval_sample_packing: true
# val_set_size: 0.05            #  remove for streaming
# num_epochs: 10                 #  replace epochs with max_steps
max_steps: 1000                 #  set your target steps
dataset_prepared_path: last_run_prepared

# 5. Saving and Evaluation Strategy (use steps with streaming)
evaluation_strategy: steps
save_strategy: steps
eval_steps: 50
save_steps: 50
save_total_limit: 100

resume_from_checkpoint:

# 6. Output & Logging
output_dir: ./outputs/sft/gemma-2-2b-it-rp-sft-qlora

wandb_project: "rp-sft"
wandb_name: "gemma-2-2b-it-rp-sft-qlora"
wandb_log_model: "false"
wandb_run_id: "gemma-2-2b-it-rp-sft-qlora"

# 7. Batching & Optimizer
gradient_accumulation_steps: 4
micro_batch_size: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
weight_decay: 0.0

# 8. Hardware & Performance
bf16: true
#fp16: true
tf32: true

flash_attention: true
gradient_checkpointing: true
logging_steps: 1

# 9. Special Tokens
eot_tokens: ["<end_of_turn>"]
special_tokens:
  bos_token: "<bos>"
  eos_token: "<eos>"
  pad_token: "<pad>"

rp-2b

This model is a fine-tuned version of google/gemma-2-2b-it on the AiAF/conversations dataset. It achieves the following results on the evaluation set:

• Loss: 2.2455
• Memory/max Active (gib): 7.78
• Memory/max Allocated (gib): 7.78
• Memory/device Reserved (gib): 17.79

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 30
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	3.1654	7.61	7.61	8.66
2.7377	0.05	50	2.5978	7.78	7.78	17.75
2.3997	0.1	100	2.5592	7.78	7.78	17.79
2.6275	0.15	150	2.5410	7.78	7.78	17.79
2.8182	0.2	200	2.5224	7.78	7.78	17.79
2.4428	0.25	250	2.4962	7.78	7.78	17.79
2.6206	0.3	300	2.4672	7.78	7.78	17.79
2.4492	0.35	350	2.4435	7.78	7.78	17.79
2.2787	0.4	400	2.4185	7.78	7.78	17.79
2.541	0.45	450	2.3998	7.78	7.78	17.79
2.5542	0.5	500	2.3640	7.78	7.78	17.79
2.6825	0.55	550	2.3484	7.78	7.78	17.79
2.6304	0.6	600	2.3278	7.78	7.78	17.79
2.4854	0.65	650	2.3104	7.78	7.78	17.79
2.3788	0.7	700	2.2877	7.78	7.78	17.79
2.2126	0.75	750	2.2748	7.78	7.78	17.79
2.4695	0.8	800	2.2662	7.78	7.78	17.79
2.5086	0.85	850	2.2553	7.78	7.78	17.79
2.404	0.9	900	2.2489	7.78	7.78	17.79
2.4012	0.95	950	2.2460	7.78	7.78	17.79
2.2586	1.0	1000	2.2455	7.78	7.78	17.79

Framework versions

• PEFT 0.17.1
• Transformers 4.57.0
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1