| | --- |
| | base_model: Qwen/Qwen3.5-9B |
| | library_name: peft |
| | model_name: output-fujin |
| | tags: |
| | - base_model:adapter:Qwen/Qwen3.5-9B |
| | - lora |
| | - sft |
| | - transformers |
| | - trl |
| | licence: license |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # output-fujin |
| |
|
| | This model is a fine-tuned version of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). |
| |
|
| | **W&B run:** [https://wandb.ai/cooawoo-personal/huggingface/runs/sr7glk4m](https://wandb.ai/cooawoo-personal/huggingface/runs/sr7glk4m) |
| | ## Training procedure |
| |
|
| | ### Hyperparameters |
| |
|
| | | Parameter | Value | |
| | |-----------|-------| |
| | | Learning rate | `0.0002` | |
| | | LR scheduler | SchedulerType.COSINE | |
| | | Per-device batch size | 1 | |
| | | Gradient accumulation | 8 | |
| | | Effective batch size | 8 | |
| | | Epochs | 1 | |
| | | Max sequence length | 2048 | |
| | | Optimizer | OptimizerNames.PAGED_ADEMAMIX_8BIT | |
| | | Weight decay | 0.01 | |
| | | Warmup ratio | 0.05 | |
| | | Max gradient norm | 1.0 | |
| | | Precision | bf16 | |
| | | Loss type | nll | |
| |
|
| |
|
| | ### LoRA configuration |
| |
|
| | | Parameter | Value | |
| | |-----------|-------| |
| | | Rank (r) | 128 | |
| | | Alpha | 16 | |
| | | Dropout | 0.05 | |
| | | Target modules | attn.proj, down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, linear_fc1, linear_fc2, o_proj, out_proj, q_proj, qkv, up_proj, v_proj | |
| | | Quantization | 4-bit (nf4) | |
| |
|
| |
|
| | ### Dataset statistics |
| |
|
| | | Dataset | Samples | Total tokens | Trainable tokens | |
| | |---------|--------:|-------------:|-----------------:| |
| | | rpDungeon/some-revised-datasets/rosier_inf_strict_text.parquet | 36,438 | 65,084,381 | 65,084,381 | |
| | |
| | |
| | <details> |
| | <summary>Training config</summary> |
| | |
| | ```yaml |
| | model_name_or_path: Qwen/Qwen3.5-9B |
| | bf16: true |
| | gradient_checkpointing: true |
| | gradient_checkpointing_kwargs: |
| | use_reentrant: false |
| | use_liger: true |
| | max_length: 2048 |
| | learning_rate: 0.0002 |
| | warmup_ratio: 0.05 |
| | weight_decay: 0.01 |
| | lr_scheduler_type: cosine |
| | label_smoothing_factor: 0.1 |
| | per_device_train_batch_size: 1 |
| | gradient_accumulation_steps: 8 |
| | optim: paged_ademamix_8bit |
| | max_grad_norm: 1.0 |
| | use_peft: true |
| | load_in_4bit: true |
| | lora_r: 128 |
| | lora_alpha: 16 |
| | lora_dropout: 0.05 |
| | logging_steps: 1 |
| | disable_tqdm: true |
| | save_strategy: steps |
| | save_steps: 500 |
| | save_total_limit: 3 |
| | report_to: wandb |
| | output_dir: output-fujin |
| | data_config: data.yaml |
| | prepared_dataset: prepared |
| | num_train_epochs: 1 |
| | saves_per_epoch: 3 |
| | run_name: qwen35-9b-qlora |
| | ``` |
| | |
| | </details> |
| | |
| | <details> |
| | <summary>Data config</summary> |
| | |
| | ```yaml |
| | datasets: |
| | - path: rpDungeon/some-revised-datasets |
| | data_files: rosier_inf_strict_text.parquet |
| | type: text |
| | truncation_strategy: split |
| | shuffle_datasets: true |
| | shuffle_combined: true |
| | shuffle_seed: 42 |
| | eval_split: 0.0 |
| | split_seed: 42 |
| | assistant_only_loss: false |
| | ``` |
| | |
| | </details> |
| | |
| | ### Framework versions |
| | |
| | - PEFT 0.18.1 |
| | - Loft: 0.1.0 |
| | - Transformers: 5.2.0 |
| | - Pytorch: 2.10.0 |
| | - Datasets: 4.5.0 |
| | - Tokenizers: 0.22.2 |
