amharic_lamma_sum_lora_normalized

README.md

@@ -1,58 +1,68 @@
 ---
+library_name: peft
+license: llama3.1
 base_model: meta-llama/Llama-3.1-8B-Instruct
-library_name: transformers
-model_name: amharic_llama_sum_lora_normalized
 tags:
-- generated_from_trainer
-- sft
 - trl
-licence: license
+- sft
+- generated_from_trainer
+model-index:
+- name: amharic_llama_sum_lora_normalized
+  results: []
 ---
 
-# Model Card for amharic_llama_sum_lora_normalized
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+# amharic_llama_sum_lora_normalized
 
-## Quick start
+This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.3919
 
-```python
-from transformers import pipeline
+## Model description
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="userdavek/amharic_llama_sum_lora_normalized", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
+More information needed
 
-## Training procedure
+## Intended uses & limitations
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/userdavek-wollo-university/llama_for_amh_sum_normalized/runs/ntgrpoo3) 
+More information needed
 
+## Training and evaluation data
 
-This model was trained with SFT.
+More information needed
 
-### Framework versions
+## Training procedure
+
+### Training hyperparameters
 
-- TRL: 0.26.2
-- Transformers: 4.57.3
-- Pytorch: 2.8.0+cu128
-- Datasets: 4.4.2
-- Tokenizers: 0.22.1
+The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 128
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- num_epochs: 3
 
-## Citations
+### Training results
 
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 0.4711        | 0.4913 | 400  | 0.4692          |
+| 0.4225        | 0.9826 | 800  | 0.4294          |
+| 0.4064        | 1.4729 | 1200 | 0.4098          |
+| 0.3946        | 1.9642 | 1600 | 0.3988          |
+| 0.3774        | 2.4544 | 2000 | 0.3934          |
+| 0.3834        | 2.9457 | 2400 | 0.3919          |
 
 
-Cite TRL as:
-    
-```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
-}
-```
+### Framework versions
+
+- PEFT 0.14.1.dev0
+- Transformers 4.57.1
+- Pytorch 2.6.0+cu124
+- Datasets 3.2.0
+- Tokenizers 0.22.1

adapter_config.json

@@ -1,12 +1,9 @@
 {
-  "alora_invocation_tokens": null,
   "alpha_pattern": {},
-  "arrow_config": null,
   "auto_mapping": null,
   "base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
   "bias": "none",
   "corda_config": null,
-  "ensure_weight_tying": false,
   "eva_config": null,
   "exclude_modules": null,
   "fan_in_fan_out": false,
@@ -23,24 +20,18 @@
   "megatron_core": "megatron.core",
   "modules_to_save": null,
   "peft_type": "LORA",
-  "peft_version": "0.18.0",
-  "qalora_group_size": 16,
   "r": 16,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "up_proj",
-    "o_proj",
     "v_proj",
-    "gate_proj",
     "down_proj",
+    "q_proj",
     "k_proj",
-    "q_proj"
+    "o_projgate_proj",
+    "up_proj"
   ],
-  "target_parameters": null,
   "task_type": "CAUSAL_LM",
-  "trainable_token_indices": null,
   "use_dora": false,
-  "use_qalora": false,
   "use_rslora": false
 }

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:061f4fd4b3ccec5ec43291093160bf3cb29cfe1adbd1382a11b86e7f6b0769d9
-size 167832240
+oid sha256:ded677b016d4f49d413d04251c94abb34f908f41de8ba43f705edf6f32e923bd
+size 113288920

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:92a37e28694c92cfe2b1c43ab2c8489887a68cfe39d4821e0b84770744cf9c85
-size 6353
+oid sha256:880161907ec0419b1625b1e1c5007cfaa29e8f5db7c9727fd3b44df9c7c0b518
+size 5688