File size: 3,844 Bytes
74c44fb 3904d77 74c44fb 1a7ef42 74c44fb 3904d77 dc85974 3904d77 afb9450 3904d77 afb9450 1769902 69413fe afb9450 82b46f1 afb9450 69413fe 1769902 afb9450 69413fe 3904d77 9e25f86 3904d77 1a7ef42 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
---
language:
- en
library_name: peft
pipeline_tag: text-generation
license: llama2
datasets:
- emrgnt-cmplxty/sciphi-textbooks-are-all-you-need
---
# ML1 Previews
This repository contains the previews for the ML1 model - [Reddit Post](https://www.reddit.com/r/LocalLLaMA/comments/16ul4sw/ml1_34b70b_phi_115_reproduction_on_llama2/)
Watch training live here: [https://api.wandb.ai/links/nickmitchko/t5d47kzr](https://api.wandb.ai/links/nickmitchko/t5d47kzr)
<div>
<iframe src="https://wandb.ai/nickmitchko/ml1-0.1/reports/ML1-Phi-Recreation---Vmlldzo1NTM3NjQw" style="border:none;height:1024px;width:100%">
</iframe>
</div>
## Checkpoints
| Model | 1 Epoch Pct | Link |
|---------------|--------|-------|
| ML1-34b | 15% | [Directory](https://huggingface.co/nmitchko/ML1-34b-previews/tree/main/checkpoint-1) |
| ML1-34b | 50% | ~ |
| ML1-34b | 100% | ~ |
| ML1-mistral-7b| 50% | ~ |
| ML1-mistral-7b| 100%|~|
| ML1-70b | 15% | ~ |
| ML1-70b | 50% | ~ |
| ML1-70b | 100% | ~ |
<!--  -->
## Model Description
The goal is to develop a series of models that can express superior performance given high quality data. To achieve this, I plan to experiment with the lovely dataset produced by [/u/docsoc1](https://www.reddit.com/user/docsoc1). Huge shout out to him/her! If you'd like to view that dataset, the link is below.
Dataset: [emrgnt-cmplxty/sciphi-textbooks-are-all-you-need](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
## Prompt Format
The model is trained using the alpaca format. Please see [here](https://github.com/tatsu-lab/stanford_alpaca#data-release) or below for that format:
```text
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
```
### Architecture
`nmitchko/ML1-34b-previews` is a large language model repository of LoRA checkpoints specifically fine-tuned to add text-book synthesized data in the style of Phi 1/1.5.
It is based on [`codellama-34b-hf`](https://huggingface.co/codellama/CodeLlama-34b-hf) at 34 billion parameters.
The primary goal of this model is to test various fine tuning methods around high quality data.
It was trained using [LoRA](https://arxiv.org/abs/2106.09685), specifically [QLora Multi GPU](https://github.com/ChrisHayduk/qlora-multi-gpu), to reduce memory footprint.
See Training Parameters for more info This Lora supports 4-bit and 8-bit modes.
### Requirements
```
bitsandbytes>=0.41.0
peft@main
transformers@main
```
Steps to load this model:
1. Load base model (codellama-34b-hf) using transformers
2. Download a checkpoint folder (checkpoint-1)
3. Apply LoRA using peft
## Training Parameters
The model is currently training on [emrgnt-cmplxty/sciphi-textbooks-are-all-you-need](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
`emrgnt-cmplxty/sciphi-textbooks-are-all-you-need` contains textbook synthesized data.
| Item | Amount | Units |
|---------------|--------|-------|
| LoRA Rank | 64 | ~ |
| LoRA Alpha | 16 | ~ |
| Learning Rate | 1e-4 | SI |
| Dropout | 5 | % |
## Training procedure
The following `bitsandbytes` quantization config was used during training:
- quant_method: QuantizationMethod.BITS_AND_BYTES
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
### Framework versions
- PEFT 0.6.0.dev0 |