|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# Model Card for FalconAlpaca |
|
|
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
FalconAlpaca is Falcon-7B trained on the [Stanford Alpaca Dataset](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) |
|
|
|
|
|
## Model Details |
|
|
|
|
|
This model was an attempt to influence the learned outputs of Falcon-7B to adapt the outputs to become more information-rich and focused. |
|
|
Trained using [Lit GPT](https://github.com/Lightning-AI/lit-gpt), the model took 2 hours to train on 1 4xA6000 node. |
|
|
|
|
|
|
|
|
### Model Description |
|
|
|
|
|
- **License:** [Apache 2.0] |
|
|
- **Finetuned from model :** [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
[Stanford Alpaca Dataset](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) |
|
|
|
|
|
### Out-of-Scope Use |
|
|
|
|
|
This model is not intended for anything but testing purposes. There have been no attempts to control/remove bias, toxicity, or any other form of |
|
|
potentially dangerous or harmful messages. |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
|
|
No effort was made to remove any wrong or harmful information from Falcon-7B or the Alpaca dataset. Any risks and limitations from either of |
|
|
those datasets/models carry over to this project as well. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
Download and install libraries for [Lit GPT](https://github.com/Lightning-AI/lit-gpt) |
|
|
|
|
|
```sh |
|
|
python generate/adapter_v2.py \ |
|
|
--adapter_path path/to/model/lit_model_adapter_finetuned.pth \ |
|
|
--checkpoint_dir path/to/model \ |
|
|
--prompt "What temperature should I cook pork at to ensure it is safe?" |
|
|
``` |
|
|
|
|
|
This uses around 14GB of VRAM. If you need to use less VRAM, you can add the parameters |
|
|
``` |
|
|
--quantize llm.int8 |
|
|
``` |
|
|
or |
|
|
``` |
|
|
--quantize gptq.int4 |
|
|
``` |
|
|
|
|
|
### Training Data |
|
|
|
|
|
[Stanford Alpaca Dataset](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) |
|
|
|
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
The defaults were as follows |
|
|
``` |
|
|
learning_rate = 9e-3 |
|
|
batch_size = 32 |
|
|
micro_batch_size = 2 |
|
|
gradient_accumulation_iters = 16 |
|
|
epoch_size = 50000 |
|
|
num_epochs = 5 |
|
|
max_iters = 125000 |
|
|
weight_decay = 0.02 |
|
|
warmup_iters = 50000 |
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
## More Information |
|
|
|
|
|
[HeitechSoft](https://heitechsoft.com/blog/heitechsoft-s-falcon-7b-fine-tuned-model-paves-the-way-for-advanced-ai-chatbots) |
|
|
|
|
|
|
