Update README.md

README.md

@@ -1,3 +1,100 @@
----
-license: agpl-3.0
----
+---
+license: agpl-3.0
+datasets:
+- JeanKaddour/minipile
+language:
+- en
+pipeline_tag: text2text-generation
+---
+# Model Card for Model ID
+
+A new way to create embeddings
+
+## Model Details
+
+### Model Description
+
+Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector 
+back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.
+
+- **Developed by:** nochinator
+- **Model type:** embeddings
+- **Language(s) (NLP):** English
+- **License:** AGPL-3.0
+
+### Model Sources 
+
+- **Repository:** https://huggingface.co/nochiantor/ThoughtVectors
+- **Paper:** Comming soon?
+- **Demo:** Comming soon?
+
+## Uses
+
+Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wording, but just capturing the meaning of input, 
+however more percisely then sentence level embeddings.
+
+### Direct Use
+
+Comming soon.
+
+## Bias, Risks, and Limitations
+
+Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger
+then it should end up saving compute.
+
+## How to Get Started with the Model
+
+This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you.
+
+### Training Data
+
+https://huggingface.co/datasets/JeanKaddour/minipile
+
+### Training Procedure
+
+Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing
+with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors.
+
+
+#### Training Hyperparameters
+
+  group_data=data,
+  num_epochs=1000,
+  batch_size=128,
+  val_split=0.8,
+  learning_rate=2e-4,
+  weight_decay=5e-5,
+  length_penalty=0.01,
+  single_vector_prob=0.2,
+  save_path="thought_vectors_prototype.tar",
+  spm_model_prefix="spm",
+  vocab_size=8192,
+  d_model=1024,
+  encoder_nhead=8,
+  decoder_nhead=8,
+  encoder_layers=4,
+  decoder_layers=4,
+  max_thoughts=32,
+  dropout=0.1,
+  max_len=256,
+  termination_threshold=0.9,
+  patience=10
+
+  Data is a generator for MiniPile
+
+## Evaluation
+
+Comming soon
+
+#### Testing Data
+
+https://huggingface.co/datasets/JeanKaddour/minipile
+
+### Model Architecture and Objective
+
+Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String
+
+## Glossary
+
+"Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens.
+A "Thought" is a complete collection of thought vectors representing a full thought