You can deploy this model and start earning money today!

nixiesearch/
nixie-suggest-small-v1

Name: nixie-suggest-small-v1
Availability: InStock

sentence-transformers

text-embeddings-inference

endpoints_compatible

Views

No views yet

nixie-suggest-small-v1

This is a sentence-transformers model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.

This model is based on E5-small-v2 model, fine-tuned for typical suggester-like workloads:

for a partial and noisy input of the query, it tries to minimize the cosine distance to the correct query
'mil' should be close to 'milk'
model also prone to typical typos like letter drops/swaps/duplications. So 'mikl' is still close to 'milk'.
the model is asymmetrical (as the original E5), so you need to prepend your prefixes with 'query: ' and full queries with 'passage: '

Usage (Sentence-Transformers)

Using this model becomes easy when you have sentence-transformers installed:

pip install -U sentence-transformers

Then you can use the model like this:

python

from sentence_transformers import SentenceTransformer
sentences = ["query: mil", "passage: milk"]

model = SentenceTransformer('nixiesearch/nixie-suggest-small-v1')
embeddings = model.encode(sentences)
print(embeddings)

Training dataset

The training dataset was syntetically generated from the following corpora:

top-100k most frequent English words, from Google N-Gram project: https://github.com/hackerb9/gwordlist
top-1M 2-grams and 3-grams from MultiLex

We did the following permutations to the original 1/2/3-grams:

letter swaps: milk-mikl
letter drops: milk-ilk
qwerty-aware replacements: milk-nilk
duplications: milk-miilk

The original generation code is available on github: https://github.com/nixiesearch/autocomplete-playground

Training

The model was trained with the parameters:

DataLoader:

torch.utils.data.dataloader.DataLoader of length 220359 with parameters:

{'batch_size': 2048, 'sampler': 'torch.utils.data.dataloader._InfiniteConstantSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

Parameters of the fit()-Method:

{
    "epochs": 1,
    "evaluation_steps": 3000,
    "evaluator": "sentence_transformers.evaluation.RerankingEvaluator.RerankingEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
    "optimizer_params": {
        "lr": 2e-05
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": 220358,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
  (2): Normalize()
)

nixiesearch/nixie-suggest-small-v1