File size: 1,898 Bytes
84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 84ff570 afa8436 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | ---
library_name: peft
license: apache-2.0
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
tags:
- base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0
- dora
- qlora
- transformers
- text-generation
pipeline_tag: text-generation
model-index:
- name: tinyllama-dora-model
results:
- task:
type: text-generation
dataset:
name: mlabonne/guanaco-llama2-1k
type: instruction-tuning
metrics:
- type: loss
value: 1.5644
name: validation_loss
---
# tinyllama-dora-model
## Model Description
This model is a parameter-efficient fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 using DoRA combined with 4-bit quantization.
---
## Key Features
* Base Model: TinyLlama-1.1B-Chat
* Fine-tuning Method: DoRA
* Quantization: 4-bit
* Framework: Transformers + PEFT
---
## Intended Use
* Instruction-based text generation
* Conversational AI
* Research and experimentation
---
## Limitations
* Small dataset (1k samples)
* May produce incorrect outputs
---
## Dataset
mlabonne/guanaco-llama2-1k
---
## Training Details
* Learning Rate: 5e-5
* Batch Size: 2
* Epochs: 1
---
## Results
Validation Loss: 1.5644
Perplexity = exp(loss)
---
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base_model = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
adapter_model = "Sujith2121/tinyllama-dora-model"
tokenizer = AutoTokenizer.from_pretrained(adapter_model)
model = AutoModelForCausalLM.from_pretrained(base_model, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_model)
prompt = "Explain Docker simply"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
---
## License
Apache 2.0
|