| --- |
| license: agpl-3.0 |
| language: |
| - en |
| pipeline_tag: text2text-generation |
| --- |
| # Model Card for Model ID |
|
|
| A new way to create embeddings |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector |
| back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input. |
|
|
| As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english. |
|
|
| - **Developed by:** nochinator |
| - **Model type:** embeddings |
| - **Language(s) (NLP):** English |
| - **License:** AGPL-3.0 |
|
|
| ### Model Sources |
|
|
| - **Repository:** https://huggingface.co/nochiantor/ThoughtVectors |
| - **Paper:** Comming soon? |
| - **Demo:** Comming soon? |
|
|
| ## Uses |
|
|
| Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings. |
|
|
| ### Direct Use |
|
|
| Comming soon. |
|
|
| ## Bias, Risks, and Limitations |
|
|
| Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger |
| then it should end up saving compute. |
|
|
| ## How to Get Started with the Model |
|
|
| This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you. |
|
|
| ### Training Data |
|
|
| https://huggingface.co/datasets/sentence-transformers/stsb |
| https://huggingface.co/datasets/stanfordnlp/snli/tree/main/plain_text |
| |
| ### Training Procedure |
| |
| Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing |
| with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors. |
| |
| |
| #### Training Hyperparameters |
| |
| group_data="train.csv", |
| test_data="val.csv", |
| num_epochs=1000, |
| batch_size=256, |
| accum_steps=1, |
| learning_rate=1e-4, |
| weight_decay=2e-5, |
| length_penalty=0.001, |
| single_vector_prob=0.1, |
| save_path="thought_vectors_prototype_0.2.tar", |
| spm_model_prefix="spm", |
| vocab_size=8192, |
| d_model=512, |
| encoder_nhead=8, |
| decoder_nhead=8, |
| encoder_layers=4, |
| decoder_layers=4, |
| max_thoughts=16, |
| dropout=0.1, |
| max_len=256, |
| termination_threshold=0.8, |
| patience=10 |
|
|
| Stopped prematurely on stsb, around 5 epochs. |
| Stopped after 1 epoch on snli. |
|
|
| ## Evaluation |
|
|
| Comming soon |
|
|
| #### Testing Data |
|
|
| https://huggingface.co/datasets/sentence-transformers/stsb |
|
|
| ### Model Architecture and Objective |
|
|
| Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String |
|
|
| ## Glossary |
|
|
| "Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens. |
| A "Thought" is a complete collection of thought vectors representing a full thought |
