---
license: agpl-3.0
language:
- en
pipeline_tag: text2text-generation
---
# Model Card for Model ID

A new way to create embeddings

## Model Details

### Model Description

Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector 
back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.

As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english.

- **Developed by:** nochinator
- **Model type:** embeddings
- **Language(s) (NLP):** English
- **License:** AGPL-3.0

### Model Sources 

- **Repository:** https://huggingface.co/nochiantor/ThoughtVectors
- **Paper:** Comming soon?
- **Demo:** Comming soon?

## Uses

Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings.

### Direct Use

Comming soon.

## Bias, Risks, and Limitations

Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger
then it should end up saving compute.

## How to Get Started with the Model

This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you.

### Training Data

https://huggingface.co/datasets/sentence-transformers/stsb
https://huggingface.co/datasets/stanfordnlp/snli/tree/main/plain_text

### Training Procedure

Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing
with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors.


#### Training Hyperparameters

  group_data="train.csv",
  test_data="val.csv",
  num_epochs=1000,
  batch_size=256,
  accum_steps=1,
  learning_rate=1e-4,
  weight_decay=2e-5,
  length_penalty=0.001,
  single_vector_prob=0.1,
  save_path="thought_vectors_prototype_0.2.tar",
  spm_model_prefix="spm",
  vocab_size=8192,
  d_model=512,
  encoder_nhead=8,
  decoder_nhead=8,
  encoder_layers=4,
  decoder_layers=4,
  max_thoughts=16,
  dropout=0.1,
  max_len=256,
  termination_threshold=0.8,
  patience=10

Stopped prematurely on stsb, around 5 epochs.
Stopped after 1 epoch on snli.

## Evaluation

Comming soon

#### Testing Data

https://huggingface.co/datasets/sentence-transformers/stsb

### Model Architecture and Objective

Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String

## Glossary

"Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens.
A "Thought" is a complete collection of thought vectors representing a full thought