Update README.md

README.md

@@ -1,3 +1,57 @@
 ---
 license: apache-2.0
+datasets:
+- HuggingFaceTB/cosmopedia
+- databricks/databricks-dolly-15k
+- Open-Orca/OpenOrca
+language:
+- en
+metrics:
+- accuracy
+library_name: transformers
+pipeline_tag: text-generation
 ---
+
+# WikiChat-v0.2
+Training in progress model to have conversations.
+
+The GGUFs uploaded are full FP32 precision.
+
+Using OpenOrca GPT-4 data + cosmopedia for some extra data + dolly15k for instruct
+
+## Model Details:
+- 40M parameters
+- 8 attention heads
+- 32 layers
+- 384 embeddings size
+- 2048/8192 context (please use 4x RoPE scaling)
+
+## Prompt Format (Alpaca):
+```
+Instruction: {system}
+Input: {prompt}
+Response: {response}
+```
+
+Please structure your prompts in an instruct format for maximum performance.
+
+## Training Details:
+- 1x RTX 3070 8GB (Infrencing speed: 80tok/s, full GPU offload)
+- 1x Ryzen 3 3700x
+- 96gb RAM
+- 10 iterations
+- Loss Target = 2.5 to 3.0
+- Approx 30 samples (>0.0001 epoches)
+- Training data = Refer to OpenOrca page
+
+## Notes:
+
+The model isn't ready yet; this is to test tokenization of OpenOrca and a balance between training speed and model size
+
+## Example output:
+```
+User: What is the square root of 4?
+```
+```
+Assistant: The square root of 4 is 2.
+```