--- library_name: peft license: gemma base_model: google/gemma-2-2b-it tags: - axolotl - base_model:adapter:google/gemma-2-2b-it - lora - transformers datasets: - AiAF/conversations pipeline_tag: text-generation model-index: - name: rp-2b results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.13.0.dev0` ```yaml # 1. Base Model & Tokenizer base_model: google/gemma-2-2b-it model_type: AutoModelForCausalLM # Corrected from 'type_of_model' for axolotl tokenizer_type: AutoTokenizer hub_model_id: AiAF/rp-2b # New model ID for this finetune hub_strategy: checkpoint # 2. LoRA / QLoRA Configuration load_in_4bit: true adapter: qlora lora_r: 64 lora_alpha: 128 lora_dropout: 0.05 lora_target_linear: true # 3. Dataset Configuration (TRAIN = streamed) streaming: true streaming_multipack_buffer_size: 10000 sample_packing: true datasets: - path: AiAF/conversations data_files: conversations_V3.jsonl # revision: type: chat_template split: train field_messages: conversations message_property_mappings: role: from content: value chat_template: jinja chat_template_jinja: | {{ bos_token }} {% for m in messages %} {% set role = 'model' if m['role']=='assistant' else 'user' %} {{ '' + role + '\n' + m['content'] | trim + '\n' }} {% endfor %} {% if add_generation_prompt %} {{ 'model\n' }} {% endif %} # chat_template_jinja: | # {{ bos_token }} # {% set last = None %} # {% for m in messages %} # {% set raw_role = 'model' if m['role']=='assistant' else m['role'] %} # {% set role = 'user' if raw_role=='system' else raw_role %} # {% if role == last and role == 'user' %} # {{ m['content'] | trim }} # {% else %} # {{ '' + role + '\n' + m['content'] | trim + '\n' }} # {% endif %} # {% set last = role %} # {% endfor %} # {% if add_generation_prompt %} # {{ 'model\n' }} # {% endif %} roles_to_train: ["assistant"] train_on_eos: "turn" # Use a fixed (non-streamed) eval file with the same schema/Jinja test_datasets: - path: . name: json type: chat_template data_files: eval-datasets/shuf-1000_conversations_V2.jsonl # small, representative eval slice split: train field_messages: conversations message_property_mappings: role: from content: value chat_template: jinja chat_template_jinja: | {{ bos_token }} {% for m in messages %} {% set role = 'model' if m['role']=='assistant' else 'user' %} {{ '' + role + '\n' + m['content'] | trim + '\n' }} {% endfor %} {% if add_generation_prompt %} {{ 'model\n' }} {% endif %} # chat_template_jinja: | # {{ bos_token }} # {% set last = None %} # {% for m in messages %} # {% set raw_role = 'model' if m['role']=='assistant' else m['role'] %} # {% set role = 'user' if raw_role=='system' else raw_role %} # {% if role == last and role == 'user' %} # {{ m['content'] | trim }} # {% else %} # {{ '' + role + '\n' + m['content'] | trim + '\n' }} # {% endif %} # {% set last = role %} # {% endfor %} # {% if add_generation_prompt %} # {{ 'model\n' }} # {% endif %} roles_to_train: ["assistant"] # 4. Training Parameters sequence_len: 2048 sample_packing: true eval_sample_packing: true # val_set_size: 0.05 # remove for streaming # num_epochs: 10 # replace epochs with max_steps max_steps: 1000 # set your target steps dataset_prepared_path: last_run_prepared # 5. Saving and Evaluation Strategy (use steps with streaming) evaluation_strategy: steps save_strategy: steps eval_steps: 50 save_steps: 50 save_total_limit: 100 resume_from_checkpoint: # 6. Output & Logging output_dir: ./outputs/sft/gemma-2-2b-it-rp-sft-qlora wandb_project: "rp-sft" wandb_name: "gemma-2-2b-it-rp-sft-qlora" wandb_log_model: "false" wandb_run_id: "gemma-2-2b-it-rp-sft-qlora" # 7. Batching & Optimizer gradient_accumulation_steps: 4 micro_batch_size: 2 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.0002 weight_decay: 0.0 # 8. Hardware & Performance bf16: true #fp16: true tf32: true flash_attention: true gradient_checkpointing: true logging_steps: 1 # 9. Special Tokens eot_tokens: [""] special_tokens: bos_token: "" eos_token: "" pad_token: "" ```

# rp-2b This model is a fine-tuned version of [google/gemma-2-2b-it](https://huggingface.co/google/gemma-2-2b-it) on the AiAF/conversations dataset. It achieves the following results on the evaluation set: - Loss: 2.2455 - Memory/max Active (gib): 7.78 - Memory/max Allocated (gib): 7.78 - Memory/device Reserved (gib): 17.79 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 1 - eval_batch_size: 2 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 4 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 30 - training_steps: 1000 ### Training results | Training Loss | Epoch | Step | Validation Loss | Active (gib) | Allocated (gib) | Reserved (gib) | |:-------------:|:-----:|:----:|:---------------:|:------------:|:---------------:|:--------------:| | No log | 0 | 0 | 3.1654 | 7.61 | 7.61 | 8.66 | | 2.7377 | 0.05 | 50 | 2.5978 | 7.78 | 7.78 | 17.75 | | 2.3997 | 0.1 | 100 | 2.5592 | 7.78 | 7.78 | 17.79 | | 2.6275 | 0.15 | 150 | 2.5410 | 7.78 | 7.78 | 17.79 | | 2.8182 | 0.2 | 200 | 2.5224 | 7.78 | 7.78 | 17.79 | | 2.4428 | 0.25 | 250 | 2.4962 | 7.78 | 7.78 | 17.79 | | 2.6206 | 0.3 | 300 | 2.4672 | 7.78 | 7.78 | 17.79 | | 2.4492 | 0.35 | 350 | 2.4435 | 7.78 | 7.78 | 17.79 | | 2.2787 | 0.4 | 400 | 2.4185 | 7.78 | 7.78 | 17.79 | | 2.541 | 0.45 | 450 | 2.3998 | 7.78 | 7.78 | 17.79 | | 2.5542 | 0.5 | 500 | 2.3640 | 7.78 | 7.78 | 17.79 | | 2.6825 | 0.55 | 550 | 2.3484 | 7.78 | 7.78 | 17.79 | | 2.6304 | 0.6 | 600 | 2.3278 | 7.78 | 7.78 | 17.79 | | 2.4854 | 0.65 | 650 | 2.3104 | 7.78 | 7.78 | 17.79 | | 2.3788 | 0.7 | 700 | 2.2877 | 7.78 | 7.78 | 17.79 | | 2.2126 | 0.75 | 750 | 2.2748 | 7.78 | 7.78 | 17.79 | | 2.4695 | 0.8 | 800 | 2.2662 | 7.78 | 7.78 | 17.79 | | 2.5086 | 0.85 | 850 | 2.2553 | 7.78 | 7.78 | 17.79 | | 2.404 | 0.9 | 900 | 2.2489 | 7.78 | 7.78 | 17.79 | | 2.4012 | 0.95 | 950 | 2.2460 | 7.78 | 7.78 | 17.79 | | 2.2586 | 1.0 | 1000 | 2.2455 | 7.78 | 7.78 | 17.79 | ### Framework versions - PEFT 0.17.1 - Transformers 4.57.0 - Pytorch 2.7.1+cu126 - Datasets 4.0.0 - Tokenizers 0.22.1