Update README.md

README.md

@@ -1,7 +1,5 @@
 ---
 license: agpl-3.0
-datasets:
-- JeanKaddour/minipile
 language:
 - en
 pipeline_tag: text2text-generation
@@ -17,6 +15,8 @@ A new way to create embeddings
 Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector 
 back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.
 
+As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english.
+
 - **Developed by:** nochinator
 - **Model type:** embeddings
 - **Language(s) (NLP):** English
@@ -30,8 +30,7 @@ back into a (vaugly) similar meaning in text. It doesn't focus on exact wording,
 
 ## Uses
 
-Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wording, but just capturing the meaning of input, 
-however more percisely then sentence level embeddings.
+Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings.
 
 ### Direct Use
 
@@ -58,29 +57,28 @@ with original tokens, then backpropagating the error through both encoder and de
 
 #### Training Hyperparameters
 
-  group_data=data,
+  group_data="train.csv",
+  test_data="val.csv",
   num_epochs=1000,
   batch_size=128,
-  val_split=0.8,
-  learning_rate=2e-4,
-  weight_decay=5e-5,
-  length_penalty=0.01,
-  single_vector_prob=0.2,
+  accum_steps=1,
+  learning_rate=5e-4,
+  weight_decay=2e-5,
+  length_penalty=0.001,
+  single_vector_prob=0.1,
   save_path="thought_vectors_prototype.tar",
   spm_model_prefix="spm",
   vocab_size=8192,
-  d_model=1024,
+  d_model=512,
   encoder_nhead=8,
   decoder_nhead=8,
   encoder_layers=4,
   decoder_layers=4,
-  max_thoughts=32,
+  max_thoughts=16,
   dropout=0.1,
   max_len=256,
-  termination_threshold=0.9,
-  patience=10
-
-  Data is a generator for MiniPile
+  termination_threshold=0.8,
+  patience=5
 
 ## Evaluation
 
@@ -88,7 +86,7 @@ Comming soon
 
 #### Testing Data
 
-https://huggingface.co/datasets/JeanKaddour/minipile
+https://huggingface.co/datasets/sentence-transformers/stsb
 
 ### Model Architecture and Objective