A new way to create embeddings

Auto-regressivly generates thought vectors (embeddings) for an input. This means fewer elements for the core model to process and thereby less compute to use. Additionally it can decode a thought vector back into a (vaugly) similar meaning in text. It doesn't focus on exact wording, but rather capturing the full meaning of the input.

As of now this is a prototype, not ready for full use. It proves the concept works, runs really fast, and vaugly grasps some concepts in english.

Specifically built for use in chatbots, but the embeddings should apply for any NLP system that doesn't rely on percise wordings (eg. classification) then sentence level embeddings.

Comming soon.

Adds a small amount of compute to the front (and if using decoder, back) of the overall system compared to other embedding mechanisms. However, if the core model is larger then it should end up saving compute.

This "model" is actually a collection of models that don't work without each other (excepting SentencePiece). A library is included in the files to manage it for you.

https://huggingface.co/datasets/sentence-transformers/stsb https://huggingface.co/datasets/stanfordnlp/snli/tree/main/plain_text

Trained by taking a sentence, tokenizing, passing through an encoder to get thought vectors then passing the vectors through the decoder to back tokens and comparing with original tokens, then backpropagating the error through both encoder and decoder. Slightly punishes for longer sets of vectors to encurage fewer vectors.

group_data="train.csv", test_data="val.csv", num_epochs=1000, batch_size=256, accum_steps=1, learning_rate=1e-4, weight_decay=2e-5, length_penalty=0.001, single_vector_prob=0.1, save_path="thought_vectors_prototype_0.2.tar", spm_model_prefix="spm", vocab_size=8192, d_model=512, encoder_nhead=8, decoder_nhead=8, encoder_layers=4, decoder_layers=4, max_thoughts=16, dropout=0.1, max_len=256, termination_threshold=0.8, patience=10

Stopped prematurely on stsb, around 5 epochs. Stopped after 1 epoch on snli.

Comming soon

Strings -> SentencePiece - Tokens -> Encoder -> Thought Vectors -> Thinker (your processing system) -> Thought Vectors (transformed, or just raw output) -> Decoder -> Tokens -> SentencePiece -> String

"Thought Vector" is the name I have given to this type of embedding - a vector that represents thoughts rather than words or tokens. A "Thought" is a complete collection of thought vectors representing a full thought