This is a Sparse Encoder model trained on the json dataset using the sentence-transformers library. It maps sentences & paragraphs to a 50368-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

Model Type: Sparse Encoder

Maximum Sequence Length: 512 tokens
Output Dimensionality: 50368 dimensions
Similarity Function: Dot Product
Training Dataset:
- json

Model Sources

Documentation: Sentence Transformers Documentation
Documentation: Sparse Encoder Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'ModernBertForEmbeddingsFusedMeanpool'})
  (1): SpladePooling({'pooling_strategy': 'mean', 'activation_function': 'log1p_relu', 'word_embedding_dimension': 50368})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
queries = [
    "This review is for the newcomer to DSLR\u0027s and not the advanced user. So we don\u0027t need to hear the experts bashing on how someone doesn\u0027t know what they are doing and shoudn\u0027t get a camera like this. I am speaking for the person who needs a camera to capture their kids in the school play, soccer game etc. and can\u0027t capture the moment with a point and shoot. You can\u0027t expect awesome pictures if you do not learn how to use all the settings, features, and purpose of an SLR camera. You might even get worse pictures than your point and shoot. Sure an SLR has an auto feature where it will figure what is the best settings to use for a basic picture, but you also end up with a basic picture.  But even all said I would still recommend an SLR and a good zoom lens that you can afford. It is worth spending the money because you can never recapture that moment of your child or whatever your trying to capture.  How many pictures have you taken with a point and shoot that were horrible because they were too dark or blurry? Sure they are fine if the lighting is good and you are within 10 feet or so away. Any further than that well, why bother. The larger zoom type cameras are better but are still limited when lighting is an issue. The pocket type point and shoots usually have a lag time between shots or flash charging up again. So forget about getting off several shots in succesion. Hard to capture a picture if the object is in motion also. The larger zoom types have mimimal lag time and some can even fire off bursts of 3 to 5 in a second with one push of the button and better at capturing something in motion. But they still can\u0027t beat an DSLR. I have gotten shots of my children with an DSLR with low lighting and no flash from many rows of seats away and they were great. The image quality was even good enough where I can crop it and still look good. People using their camera phones crack me up, but that\u0027s another story.  I am not going to get into all the technicalities of this camera. You can read the other reviews for that. I just want to mention that if you are one of those people who need to step up to an SLR because the point and shoot couldn\u0027t do the job but don\u0027t want to become the serious photographer then this inforomation might be what you are looking for.  I find the XSI pictures are soft and not crisp. Some photographers prefer this. The new Nikon DSLR\u0027s take crisper pictures, and they are less expensive.  The main con of the XSI is the focusing points. If you use the camera in auto mode then it will focus on the nearest or most identifiable focus points it can focus on. So it might focus on an object in front of the actual thing your trying to capture that is behind it, or on a purse or print on a shirt for example instead of the persons face or what not. If your trying to get a group picture with the guest of honor in the middle of it, you might not have that person in full focus but instead the preople around because the cameras focus is focusing on the most vivid or closest points. You can set the camera in one of the manual settings and pick one focus point of your choice, like center, top, left, right etc. but you can\u0027t pick all on. In auto it will focus on more than one but like I mentioned before, not excactly on what you want. you can press the shutter button half way and it will try and refocus on different points or you can get closer to your subject and press the shutter button half way and hold it so it keeps that focus point and then you step back and snap the picture. Well you don\u0027t have the luxury of time on your hands for those kid or spontaneous shots.  So this is where I recommend the newer Nikon DSLR\u0027s. Their focus points I find tend to focus on more points and usually on the center also where my subject tends to be. The XSI was here and there. If the area your focusing on is busy, the XSI will pick a few focus points here and there were it thinks are the best. But with the Nikon, I found it to focus on more points which is nice because the whole image will be more sharper instead of certain areas of it when cropping or zooming the image on a computer. Unless you are going for the \"bokeh\" effect in which the subject is in focus and the area around it is blurred.  A camera of this size isn\u0027t always convienient to carry around to all occasions because of it\u0027s size, accessories, worries, or whatever the reason. If you already have a pocket point and shoot or zoom camera, don\u0027t get rid of them. I keep them for when I go on vacation because I don\u0027t want to carry a big DSLR around on vacation. I use the pocket one for when we go out and I don\u0027t want to carry anything bulky and for those you never know shots. I use the zoom one for the amusement parks. It\u0027s small enough where it is still manageable where I can toss in my backpack. I use that one for the amusement parks because I can get pictures of the kids on rides. The zoom and speed is adequate enough where I can get pictures of the kids in the distance or when their zooming down the roller coaster. One day I might get rid of my pocket one and get a waterproof one, that way I have all my bases covered. Ha ha.  But when you learn how to use a DSLR\u0027s features coupled with a decent lens, man the pictures are like night and day.  If you still find that a DSLR is just too much over kill for you then you might want to check out the Nikon S8100. It is a little higher end point and shoot that is a little larger than your average point and shoot but it takes amazing pictures at a distance and in low lighting and has HD video capture with stereo. My nephew has the older S8000 model of this and I was blown away at the pictures and videos he took at my childs play. And we were sitting way back. They made improvements on the S8100 where it counted so I can\u0027t imganine how good it would be now, wow.  I would recommend you try out different cameras and brands in the store and decide which one suits you. Look for optical zoom, optical image stabilization, and good low light capability.",
]
documents = [
    "Mail Boss makes several different models that look identical so pay close attention of the model names. I looked at the Architectural Oasis Jr., Architectural Oasis trilock and the Mail Boss Mail Manager. A person on youtube  demonstrated how easily he was able to manipulate his arm into the mail chute and fish out a package and them demonstrated easily and quickly he jammed a large screwdriver into the edge of the lower door and popped it open shattering the lock. The trilock is the reinforced version of the Jr. It has an upgraded locking system where the Jr. only had a single larch whereas the trilock has a latch and a metal rod on each side of the door that creates three locking points. The Jr.'s door was a simple flat sheet of metal that didn't go all the way to the surrounding edges so it is vulnerable to prying open. The trilock's door sides go all the way to the edge and is flanged for strength and the top edge is tapered to fit under the top door to prevent a screwdriver or something to get behind it.  I mention the above to give insight as to why I chose the Mail Manager. The Jr. was way too easy to pry open so it dropped out of consideration automatically. The trilock seemed just as tamper proof as the Manager as far as prying open is. The ease of fishing mail out of the Jr. and the trillock with your arm is questionable as I was not able to get my arm in like the person on youtube.  The conditions have to be perfect for someone to be able to fish out the mail. The person has to have skinny arms and the mailbox has to be low enough for the person to contort their arm and then reach down. With the Manager the door is smaller thus creating a smaller opening. The Jr, and trilock's doors are larger but there is about a 2 inch plate that hangs on the inside of the box that prevents one from reaching straight into the box and you have to contort around it.  The Jr. and trilock had rubber gaskets around the doors which is nice if you live in an area with heavy rain and wind. The Manager had some mention that their mail gets wet but that is easily addressable by getting thin adhesive gasket strips. The trilock and the Manager in my opinion were about equally secure in tamper resistance and build quality. The Manager and Jr. were about the same price and the trilock was about $20.00 more. The Manager had a little less capacity. The main thing that led me to get the Manager over the trilock was the ability to accept larger packages. It looks like the trilock should be able to accept larger packages because it has a bigger door and the Manager's door is smaller but that is not the case because of the door opening and obstruction I mentioned earlier. All three boxes were only able to accept a package that was about 8 1/2 x 4 3/4 x 3. With the Jr. and trilock the package had to be placed a certain way because of the obstruction on the inside. If you take off the mail clip on the inside of the doors then you can increase the package size to about 10x4 3/4x3. If your getting a locking mailbox then you are worried about mail theft so why would you place outgoing mail clipped to the unlocked upper door anyways. Back to the package capacity. The Jr. and trillock were maxed out in package size unless it is a flat envelope package that you can shove past the overhang with the door closed part way giving you the angle. The manager on the other hand does not have this overhang other than a slight lip so you basically have a clear pass through where you can roughly get a 10 x 15 x 1 1/4 package in. The other notable pluses for the Manager is that it comes with 4 bolts and a specialized drill bit for installation whereas the other two you had to supply your own hardware. And lastly the Jr. and trilock came with two large keys where the Manager came with three small keys. There are bigger ones in the Mail Boss line but they run around a $100.00 more. One negative about the Mail Manager is that it is shallower and slightly narrower than the Jr. And trillock so when then mailman delivers large junk mail like advertisements they can get hung up on the inside lower lip of the door because they are too wide for the box and the box isn't deep enough for gravity to have it fall in completely because the portion of mail on the inside is touching the bottom of the box along with the other end propped up on the door lip and the sides of the junk mail pressing on the sides. If your mailman is good, then all he has to do is give a light nudge and the mail will fall in. The Jr. and trilock offer a slightly wider opening and has a deeper collection area so gravity can do it's work.",
    'This is a really great book and I highly recommend it, both as a compelling autobiography and as a way to learn about Islam and Islamism.  One thing I learned from this book is the tremendous variation within Islam. The distinction that was most important to me was the difference between Religious (or Spiritual) Islam and Political Islam.  The values and social ideals of Religious Islam are virtually identical to Christiaity and Buddhism (the two world religion with which I am most familiar).  The emphasis on care for the poor, the needy, and less fortunate is indistinguishable among these religions.  Further, Islamic mysticism describes Allah in the same sacred language used by mystics of all stripes.  This blends well with our religious heritage and adds new inspiring stories and language that will benefit all with an open mind.  On the other side is Political Islam, it\'s hatred of the West, it\'s lack of sound Islamic scholarship and spirituality, it\'s bigotry and shameful treatment of women, it\'s dedication to an Islamic State, and it\'s willingness to kill in order to attain it.  This form of Islam, and I hesitate even calling it Islam given it\'s shallow scholarship and deviance from fundamental Islamic values and teachings, breeds extremism and terrorism.  This form of Islam poses a concrete threat to America and other Western countries.  Within our country, as well as countries throughout the world, this twisted form of Islam is taught, primarily, by missionaries from Saudia Arabia.  They have combined a rigid, severe, and literalist version of Islam that dates to the mid 1700s with the political goal of transforming the world into a single Islamic state.  They are True Believers of the worst kind and the foundation on which Islamic terrorists stand.  Note:  After reading The Islamist and writing the review above, I read Islam by <PERSON>, an excellent overview of Islam.  One point that he makes is that Islam is not just a religion, but a way of life, and that Islam permeates all aspects of the Muslim,s life, including politics.  Because this applies to all forms of Islam, the distinction made in The Islamist, and repeated in my review, between "Religious Islam" and "Political Islam" is not correct, or at least not precise.  Other labels also have there shortcomings, so I will let the wording used by the author stand.  Nonetheless, The Islamist remains and excellent, powerful, and intensely personal work.',
    "I recently bought a new embroidery machine as an upgrade my old entry level machine.  I thought I'd upgrade my thread as well.  The reviews for this thread seemed very positive.  I recently learned that embroidery thread has a shelf life and it was appealing that the thread was protected so well by the plastic dome and the plug.  I purchased several colors and intended to use this thread exclusively in my new machine.  Unfortunately during my first project with the new machine, this thread was breaking at a discouraging rate.  I thought something was wrong with the machine.  I actually brought it back to the dealer a couple of times but when they tested it, they had no thread breakage.  Of course, they were not using Hemingworth thread.  Nobody thought it was the thread so I would bring the machine home and experience breakage again.  In desperation, I brought the machine back to the dealer and took along my thread, hoop, stabilizer, and one of the designs that had given me grief with a lot of thread breakage.  I spent two hours with the dealer's technician.  The technician examined the machine and could find nothing wrong with it.  We then sewed my design with my Hemingworth red thread.  The thread broke again and again.  We ended up doing a lot of testing with various weight bobbin thread, tension settings, threads, needles, etc..  The conclusion was that it was the Hemingworth thread causing the issues.  The technician who patiently sat with me thought the thread felt heavy and sticky.  He said it was probably old thread from the way it behaved.  What I couldn't argue with was that every time we rethreaded with another brand of thread and sewed the same design under all the same conditions, there was no upper thread breakage.  I'm not happy that I purchased several Hemingworth spools that are now of no use to me but I'm thankful that I didn't purchase any of the large sets I was considering.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 50368] [3, 50368]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[29.5178, 19.0865, 22.2767]])

Evaluation

Metrics

Sparse Information Retrieval

Dataset: sparse-ir-eval
Evaluated with SparseInformationRetrievalEvaluator with these parameters:
json
1{ 2 "max_active_dims": 4096 3}

Metric	Value
dot_accuracy@1	0.0494
dot_accuracy@8	0.1442
dot_accuracy@50	0.3137
dot_accuracy@100	0.4109
dot_precision@1	0.0494
dot_precision@8	0.018
dot_precision@50	0.0063
dot_precision@100	0.0041
dot_recall@1	0.0494
dot_recall@8	0.1442
dot_recall@50	0.3137
dot_recall@100	0.4109
dot_ndcg@10	0.0972
dot_mrr@10	0.0783
dot_map@100	0.0868
query_active_dims	4096.0
query_sparsity_ratio	0.9187
corpus_active_dims	4096.0
corpus_sparsity_ratio	0.9187
avg_flops	2975.3545

Training Details

Training Dataset

json

Dataset: json
Size: 404,856 training samples
Columns: query and positive
Approximate statistics based on the first 1000 samples:
query positive
type string string
details
min: 402 tokens
mean: 496.56 tokens
max: 512 tokens
min: 402 tokens
mean: 496.56 tokens
max: 512 tokens

Samples:

query	positive
This knife, and theJ.A. Henckels Twin Four Star 3-Inch High Carbon Stainless-Steel Paring Knifeare the two best knives in the Henckels four star collection. There is something "just so" about them. They are just right, with all the various design parameters coming together to create a whole that is greater than the sum of the parts. This serrated utility knife works well in a great variety of applications. The five inch serrated blade is nicely thin (but still thick enough for good strength and rigidity) and shallow (i.e. not broad). I find it very useful for cutting pie or cake or brownies, as well as (of course) bread and tomatoes and many other vegetables. I've had this knife now for SIXTEEN YEARS, and it is still going strong, and still one of my favorites. However . . . you must SHARPEN this knife eventually. Like any other knife, it will go dull. NEVER HONE THIS KNIFE OR ANY OTHER SERRATED KNIFE! A sharpening steel is too large a diameter to be used on a serrated knife....	When I moved into my first (and current) house from my apartment, the previous owner had a Whirlpool (Ecodyne) WHER25 reverse osmosis system installed under the kitchen sink. I liked the water the system produced, but the flow control was misfunctioning, causing an annoying dripping sound that was almost constant. The installer (previous owner, not a plumber) had NOT made the common mistake of trimming out the flow control--which was the first thing I suspected. No, the problem, rather, was deformation of the thin rubber membranes (there are two) inside the head of the unit. I flipped them over (they are reversible) and this fixed the problem for a month or so, but it returned. I priced out new membranes/gaskets and flow control insert, with shipping, and decided that I should just start fresh with a whole new unit, since it was on a special sale locally and it would come with all new filters ($80 worth). I replaced just the head and all was well for a while. Then the tank stopp...
When I moved into my first (and current) house from my apartment, the previous owner had a Whirlpool (Ecodyne) WHER25 reverse osmosis system installed under the kitchen sink. I liked the water the system produced, but the flow control was misfunctioning, causing an annoying dripping sound that was almost constant. The installer (previous owner, not a plumber) had NOT made the common mistake of trimming out the flow control--which was the first thing I suspected. No, the problem, rather, was deformation of the thin rubber membranes (there are two) inside the head of the unit. I flipped them over (they are reversible) and this fixed the problem for a month or so, but it returned. I priced out new membranes/gaskets and flow control insert, with shipping, and decided that I should just start fresh with a whole new unit, since it was on a special sale locally and it would come with all new filters ($80 worth). I replaced just the head and all was well for a while. Then the tank stopp...	This knife, and theJ.A. Henckels Twin Four Star 3-Inch High Carbon Stainless-Steel Paring Knifeare the two best knives in the Henckels four star collection. There is something "just so" about them. They are just right, with all the various design parameters coming together to create a whole that is greater than the sum of the parts. This serrated utility knife works well in a great variety of applications. The five inch serrated blade is nicely thin (but still thick enough for good strength and rigidity) and shallow (i.e. not broad). I find it very useful for cutting pie or cake or brownies, as well as (of course) bread and tomatoes and many other vegetables. I've had this knife now for SIXTEEN YEARS, and it is still going strong, and still one of my favorites. However . . . you must SHARPEN this knife eventually. Like any other knife, it will go dull. NEVER HONE THIS KNIFE OR ANY OTHER SERRATED KNIFE! A sharpening steel is too large a diameter to be used on a serrated knife....
The Good: Sawstop customer service is the best I have dealt with in years. When set up correctly it cuts sheet good like a dream. Only a panel saw would seem better. The adjustable stops are stout and easy to use. Great for repeat cuts. Sliding mechanism is very smooth The Bad: No postive stops - in my experience this borders on being a huge problem for two reasons. First, it is not easy to get the fence square to the blade if you want to be very accurate. On the best of days it takes me 5 minutes to get it close enough to make a 48" cut square. Without positive stops I have to square the sliding table fence every time it is bumped or removed. And, I remove it regularly as the sliding table fence sits close enough to the blade that almost all cuts over 48" using the regular saw fence demand the sliding table fence be removed or swung out of the way (if the cut is less than 48" the sliding table can be moved back with fence in place forming a little pocket to work within). The fence th...	The Bad (yes there are a lot of bad things even with 5 stars): One of the worst written non fiction books I have ever owned. I really don't care if one of the authors clients liked a sauce so well that she would eat it over kitty litter. I don't care to read 100s of testaments to how good the recipes are (they are pretty good). I just want to get on with the book. Prove the recipes are not good. Don't spend all those pages trying to convince me. It even backfired. I was sure they were going to be terrible are reading all the testaments. Get ready to cook. A lot. Get ready to do a lot of dishes. Have to plan ahead. Have to make lunches the night before often. The flax seed breakfast takes some work and time. Can't just whip it up. If you run out without having already prepared more you will find yourself without a breakfast. Terribly organized. It is not sequential. You will read something and then find out later you were not suppose to do it when you did unless you read the entire b...

Loss: model.SpladeMixedTopKLoss.SpladeMixedTopKLoss with these parameters:
json
1{ 2 "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)", 3 "document_regularizer_weight": 0.005, 4 "query_regularizer_weight": 0.005, 5 "document_regularizer_threshold": 256, 6 "query_regularizer_threshold": 256 7}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 64
learning_rate: 0.0001
weight_decay: 0.0001
num_train_epochs: 1
lr_scheduler_type: cosine
warmup_ratio: 0.15
save_only_model: True
bf16: True
dataloader_num_workers: 8
gradient_checkpointing: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 64
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 0.0001
weight_decay: 0.0001
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.15
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: True
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 8
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: True
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs