| --- |
| base_model: meta-llama/Llama-2-7b-hf |
| library_name: peft |
| license: llama2 |
| datasets: |
| - timdettmers/openassistant-guanaco |
| language: |
| - en |
| - th |
| - zh |
| metrics: |
| - accuracy |
| pipeline_tag: question-answering |
| --- |
| |
|
|
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| <!-- Provide a longer summary of what this model is. --> |
|
|
|
|
| - **Developed by:** [Jixin Yang @ HKUST] |
| - **Model type:** [PEFT (LoRA) fine-tuned LLaMA-2 7B for backward text generation] |
| - **Finetuned from model [optional]:** [meta-llama/Llama-2-7b-hf] |
|
|
|
|
|
|
| ## Uses |
| This model is designed for backward text generation - given an output text, it generates the corresponding input. |
|
|
|
|
|
|
| ## How to Get Started with the Model |
|
|
| Use the code below to get started with the model. |
|
|
| from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
| model_name = "jasperyeoh2/llama2-7b-backward-model" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto") |
|
|
| input_text = "Output text to reverse" |
| inputs = tokenizer(input_text, return_tensors="pt").to("cuda") |
| outputs = model.generate(**inputs, max_new_tokens=50) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
| ## Training Details |
| |
| ### Training Data |
| |
| <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
| |
| - Dataset: [OpenAssistant-Guanaco](https://huggingface.co/datasets/timdettmers/openassistant-guanaco) |
| - Number of examples used: ~3,200 |
| - Task: Instruction Backtranslation (Answer → Prompt) |
| |
| ### Training Procedure |
| |
| <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
| |
| #### Preprocessing [optional] |
| |
| - Method: PEFT with LoRA (Low-Rank Adaptation) |
| - Quantization: 4-bit (NF4) |
| - LoRA config: |
| - `r`: 8 |
| - `alpha`: 16 |
| - `target_modules`: ["q_proj", "v_proj"] |
| - `dropout`: 0.05 |
| - Max sequence length: 512 tokens |
| - Epochs: 10 |
| - Batch size: 2 |
| - Gradient accumulation steps: 8 |
| - Effective batch size: 16 |
| - Learning rate: 2e-5 |
| - Scheduler: linear with warmup |
| - Optimizer: AdamW |
| - Early stopping: enabled (patience=2) |
| |
| |
| #### Metrics |
| |
| <!-- These are the evaluation metrics being used, ideally with a description of why. --> |
| |
| [wandb: https://wandb.ai/jyang577-hong-kong-university-of-science-and-technology/huggingface?nw=nwuserjyang577] |
| |
| ### Results |
| |
| [- Final eval loss: ~1.436 |
| - Final train loss: ~1.4 |
| - Training completed in ~8 epochs] |
| |
| |
| |
| ### Compute Infrastructure |
| |
| - GPU: 1× NVIDIA A800 (80GB) |
| - CUDA Version: 12.1 |
| |
| #### Software |
| |
| - OS: Ubuntu 20.04 |
| - Python: 3.10 |
| - Transformers: 4.38.2 |
| - PEFT: 0.15.1 |
| - Accelerate: 0.28.0 |
| - BitsAndBytes: 0.41.2] |
| |
| #### Hardware |
| |
| NVIDIA A800 GPU |
| |
| |
| ### Framework versions |
| |
| - PEFT 0.15.1 |
