Update README.md

README.md

@@ -1,3 +1,94 @@
 ---
+language:
+- en
+library_name: peft
+pipeline_tag: text-generation
 license: llama2
 ---
+
+# ML1 Previews- 34b
+
+This repository contains the previews for the ML1 model - [Reddit Post](https://www.reddit.com/r/LocalLLaMA/comments/16ul4sw/ml1_34b70b_phi_115_reproduction_on_llama2/)
+
+## Checkpoints
+1. `15%` - Complete
+    * [Link]()
+3. `30%` - IP
+4. ...
+
+<!-- ![Screenshot](https://huggingface.co/nmitchko/i2b2-querybuilder-codellama-34b/resolve/main/Example%20Query.png) -->
+
+## Model Description
+
+The goal is to develop a series of models that can express superior performance given high quality data. To achieve this, I plan to experiment with the lovely dataset produced by [/u/docsoc1](https://www.reddit.com/user/docsoc1). Huge shout out to him/her! If you'd like to view that dataset, the link is below.
+
+Dataset: [emrgnt-cmplxty/sciphi-textbooks-are-all-you-need](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
+
+## Prompt Format
+
+The model is trained using the alpaca format. Please see [here](https://github.com/tatsu-lab/stanford_alpaca#data-release) or below for that format:
+
+```text
+Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{instruction}
+
+### Response:
+```
+
+### Architecture
+`nmitchko/ML1-34b-previews` is a large language model repository of LoRA checkpoints specifically fine-tuned to add text-book synthesized data in the style of Phi 1/1.5.
+It is based on [`codellama-34b-hf`](https://huggingface.co/codellama/CodeLlama-34b-hf) at 34 billion parameters.
+
+The primary goal of this model is to improve research accuracy with the i2b2 tool.
+It was trained using [LoRA](https://arxiv.org/abs/2106.09685), specifically [QLora Multi GPU](https://github.com/ChrisHayduk/qlora-multi-gpu), to reduce memory footprint. 
+
+See Training Parameters for more info  This Lora supports 4-bit and 8-bit modes.
+
+### Requirements
+
+```
+bitsandbytes>=0.41.0
+peft@main
+transformers@main
+```
+
+Steps to load this model:
+1. Load base model (codellama-34b-hf) using transformers
+2. Download a checkpoint folder (checkpoint-1)
+3. Apply LoRA using peft
+
+## Training Parameters 
+
+The model is currently training on [emrgnt-cmplxty/sciphi-textbooks-are-all-you-need](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
+
+
+`emrgnt-cmplxty/sciphi-textbooks-are-all-you-need` contains textbook synthesized data.
+
+
+| Item          | Amount | Units |
+|---------------|--------|-------|
+| LoRA Rank     | 64    | ~     |
+| LoRA Alpha    | 16    | ~     |
+| Learning Rate | 1e-4   | SI    |
+| Dropout       | 5      | %     |
+
+## Training procedure
+
+
+The following `bitsandbytes` quantization config was used during training:
+- quant_method: QuantizationMethod.BITS_AND_BYTES
+- load_in_8bit: False
+- load_in_4bit: True
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: nf4
+- bnb_4bit_use_double_quant: True
+- bnb_4bit_compute_dtype: bfloat16
+
+### Framework versions
+
+- PEFT 0.6.0.dev0