File size: 3,255 Bytes
4fcd291 e62c1bd 4fcd291 e62c1bd 4fcd291 9b6e46c a832593 4fcd291 e62c1bd 4fcd291 a832593 e62c1bd a832593 e62c1bd a832593 4fcd291 e62c1bd 4fcd291 e62c1bd 4fcd291 e62c1bd a832593 4fcd291 e62c1bd 4fcd291 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | ---
license: agpl-3.0
language:
- en
pipeline_tag: text2text-generation
---
# Model Card for Model ID
A new way to create embeddings
## Model Details
### Model Description
Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector
back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.
As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english.
- **Developed by:** nochinator
- **Model type:** embeddings
- **Language(s) (NLP):** English
- **License:** AGPL-3.0
### Model Sources
- **Repository:** https://huggingface.co/nochiantor/ThoughtVectors
- **Paper:** Comming soon?
- **Demo:** Comming soon?
## Uses
Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings.
### Direct Use
Comming soon.
## Bias, Risks, and Limitations
Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger
then it should end up saving compute.
## How to Get Started with the Model
This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you.
### Training Data
https://huggingface.co/datasets/sentence-transformers/stsb
https://huggingface.co/datasets/stanfordnlp/snli/tree/main/plain_text
### Training Procedure
Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing
with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors.
#### Training Hyperparameters
group_data="train.csv",
test_data="val.csv",
num_epochs=1000,
batch_size=256,
accum_steps=1,
learning_rate=1e-4,
weight_decay=2e-5,
length_penalty=0.001,
single_vector_prob=0.1,
save_path="thought_vectors_prototype_0.2.tar",
spm_model_prefix="spm",
vocab_size=8192,
d_model=512,
encoder_nhead=8,
decoder_nhead=8,
encoder_layers=4,
decoder_layers=4,
max_thoughts=16,
dropout=0.1,
max_len=256,
termination_threshold=0.8,
patience=10
Stopped prematurely on stsb, around 5 epochs.
Stopped after 1 epoch on snli.
## Evaluation
Comming soon
#### Testing Data
https://huggingface.co/datasets/sentence-transformers/stsb
### Model Architecture and Objective
Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String
## Glossary
"Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens.
A "Thought" is a complete collection of thought vectors representing a full thought |